"1. INTRODUCTION. Improving student retention, graduation, and completion rates is a fundamental challenge in improving educational delivery.[?] S3 is intended as a practical end-to-end solution for identifying which students are at risk, understanding why they are at risk, designing interventions to mitigate that risk, and finally closing the feedback loop by assessing the success. Current approaches to building predictive models for identifying at-risk students are stymied by two serious limitations. First, the predictive models are one-off and, therefore, cannot be extended easily from one context to another. We cannot simply assume that a predictive model developed for a particular course at a particular institution is valid for other courses. Can we devise a flexible and scalable methodology for generating predictive models that can accommodate the considerable variability in learning contexts across different courses and different institutions? Secondly, current modeling approaches, even if they generate valid predictions, tend to be black boxes from the standpoint of practitioners. The mere generation of a risk signal (e.g. green, yellow, red) does not convey enough information for the purpose of designing meaningful personalized interventions on behalf of students. As a way of overcoming these limitations we devise a modeling strategy that begins with a generic measure called Success Index which is decomposed initially into five indices: Preparation, Attendance, Participation, Completion, and Social Learning. (As we proceed with the implementation of S3 we anticipate discovering and adding other indices.) Each index is itself a composite expressing a number of relevant activity-tracking variables. These tracking variables are measured on different scales, primarily in terms of the frequency with which a particular action or task is performed or the time spent on-task. These indices or semantic units serve as the foundation for applying an ensemble method for predictive modeling. 2. RELATED WORK. In education predictive models for identifying at-risk students was pioneered by John Campbell and the Course Signals Project at Purdue University.[?] Similar work has been underway at Capella University, Rio Salado College and other institutions. In this section we discuss methodological limitations with current risk modeling approaches in education. In Section 3 we provide a quick overview of S3. In Section 4 we propose how to overcome current methodological limitations in risk modeling. 2.1 Predictive Models in Education. The Course Signals system and recent research studies [?] provide early evidence that student elearning activities are predictive of academic success. Regression modelling such as logistic regression has been applied to build a best-fit coursebased predictive models. Such models incorporate the most significant LMS variables such as total number of discussion messages posted, total number of mail messages sent, and total number of assessments completed. The mathematical aspects of this modeling strategy is briefly described in Sec 4.1. Macfadyen and Shane [?] discuss the limitations of this work in terms of its overall generalizability and interpretation. In particular, the generalizability of such models can be limited by the sample courses used for model fitting, or by focussing on fully online courses within one institution. A core problem in current approaches, as applied in Course Signals-type systems, is that a single hypothesis/model that best fits a collection of course data, is chosen from the space of all possible hypotheses, and then applied to make predictions across different courses in different programs and institutions. There are potential sources of bias in this solution. This methodology is expected to work well when courses on which the model is applied have a relatively consistent instructional model with the courses used to discover the best-fit model, but otherwise lead to a risk of systematic errors in predictions, i.e. relatively high bias. The limitations of this modeling strategy, in terms of generalizability and interpretability, critically hinder the wideranging deployment of discovered models to educational institutions in a meaningful way. Hence, it limits the potential benefits that institutions can draw from their data through the development of predictive analytics capabilities for modeling learner success. In this work, we propose a predictive modeling strategy that aims at closing this gap. We focus on providing a highly-generalizable modeling strategy that is well-suited for supporting wide-ranging needs of educational institutions and for taking full advantage of predictive analytics. We propose an adaptive framework and a stacked-generalization modeling strategy whereby intelligent data analysis can be applied at all levels and graciously combined to express higher-level generalizations. A second key problem is that current predictive modeling systems do not provide diagnostic information. For example, Course Signals generates a prediction that indicates the identified level of risk; however, there is no direct insight into the specific causes, thus making a recommended remediation difficult to specify. Furthermore, the system does not incorporate human insight that can be leveraged via model tuning, if needed. If a system is designed to facilitate interpretability and self-explanation, a by-product is the ability to support a meaningful tuning functionality, thus taking the insight of business domain experts into account. To enable an effective synthesis of machine intelligence and human insight, the proposed S3 provides an interpretable model and data visualizations. In particular, we focus on developing an interpretable modeling strategy, intuitive human experience and powerful interaction with the data and models. Furthermore, for predictive analytics to be successfully applied at an institution, it needs to be deeply integrated into business process, where decision makers can use it in their natural workflow every day. Another issue with a Signals-type model is that it ignores potentially key aspects of learning. One such example is social learning. For example, in [?], social network analysis plays a key role in providing insights into the student learning community and the patterns of peer interactions. In S3, a social network analysis and visualization is incorporated to capture and explain the social learning aspect. Similarly, the treatment of content comprehension is limited to tracking the number of content topics visited. On the other hand, intelligent tutoring systems and related work [?] develop specialized data analysis and domain knowledge representation to model learner behavior and abilities in relation to content usage and knowledge acquisition. In S3, we propose an ensemble strategy whereby a domainspecific decomposition allows for the development and integration of specialized models and algorithms that are best suited for different aspects of learning. In particular, in S3, the proposed decomposition provides an abstraction of learning behavior into semantically meaningful units. Prediction ensembles provide a powerful and flexible paradigm for enhancing the relevance and generalizability of predictive analytics. It can also be viewed as enabling a collaborative platform, whereby institution can plug their own proprietary model as part of the ensemble. Thus, it enables an open, community-driven R&D platform for the application of predictive models to advance learning analytics as well as institutional analytics capabilities. 3. STUDENT SUCCESS SYSTEM. In this section we provide a functional overview of S3. This will serve as background for the modeling strategy described in Section 4. The overview is not intended to be comprehensive. Our aim is to provide enough context for stating the research problem and our proposed solution. 3.1 S3 Functionality. The workflow for S3 is analogous to the workflow associated with the steps in a patient-physician relationship. When a patient sees a physician the basic workflow is: a) understand the problem; b) reach a diagnosis; c) prescribe a course of treatment; d) track the success. S3 follows a similar workflow. First, upon login to S3 an advisor (a possible role in S3) is presented with a pictorial list of her students. Associated with each student is a risk indicator: green indicates not at-risk, yellow indicates possibly at-risk, and red means at-risk. The advisor can immediately click on a particular student or view the screen showing the list of students in a particular category (e.g. high risk). Next, associated with each student is his Student Profile Screen. The Student Profile Screen provides an overview of the student’s profile, including projected risk at both the course and institution level. The screen also serves as a gateway to other screens, including Course Screens which provide views into course-level activity and risks. The Notes Screen provides case notes associated with the student while Referral Screen provides all the relevant referral options available at the institution. 3.2 Data Visualizations. As the user of the S3 navigates through the various success indicators, the underlying models and data are presented in an intuitive and interpretable manner, going from one level of aggregation to another. S3 contains a number of visualizations for diagnostic purposes. These include: Risk Quadrant, Interactive Scatter Plot, Win-Loss Chart, and Sociogram. For illustrative purposes we provide a representation of the Interactive Scatter Plot and the Win-Loss Chart. A user of S3 is able to explore the data that make up the predictive model by selecting the success indicators associated with each domain and visualize patterns such as cluster structures and relations between different indicators and measures of performance. The chart is also dynamic in the sense that data can be animated to visualize paths/trails depicting changes in learner behaviors and performance over time. Figure 1: Visualization - Interactive Scatter Plot. Another example of the charts available in S3 is the WinLoss Chart. As shown below, one can see at glance how the student compares to peers in the overall success indicator and along each of the sub-indicators. Values above, within, or below average are indicated by green, orange and red bars. Option is provided to compare current indicators with the student’s own history. This option help visualize changes in student’s own learning behavior over time. Figure 2: Visualization - Win-Loss Chart. 4. ENSEMBLE MODELING STRATEGY. The idea of prediction ensemble is to enable the selection of a whole collection, or ensemble, of hypotheses from the hypothesis space and combine their predictions appropriately. A key rationale is that various indicators of learning success and risks can be found by analyzing different aspects of the learning and teaching processes, the educational tools and instructional design, the pre-requisite competencies, the dynamics of a particular course, program or institution, as well as the modality of learning being fully online, live, or hybrid. We argue that there is a need for the discovery and blending of multiple models to effectively express and manage complex and diverse patterns of the elearning process. Ensemble methods are designed to boost the predictive generalizability by blending the predictions of multiple models [?, ?, ?]. For example, stacking, also referred to as blending, is a technique in which the predictions of a collection of base models are given to a second-level predictive modeling algorithm, also referred to as a meta-model. The secondlevel algorithm is trained to combine the input predictions optimally into a final set of predictions. Classifier ensembles allow solutions that would be difficult (if not impossible) to reach with only a single model [?]. Stacking, data fusion, adaptive boosting, and related ensemble techniques have successfully been applied in many fields to boost prediction accuracy beyond the level obtained by any single model [?]. S3 represents a particular instance of the ensemble paradigm. It employs aspects of data fusion as explained in Sec 4.1 to build base models for different learning domains. Furthermore, the system utilizes a stacked generalization strategy as explained in 4.2. A best-fit meta-model takes as input predictors the output of the base models and optimally combine them into an aggregated predictor, referred to as a success indicator/index. In this type of stacked generalization, optimization is typically achieve by applying EM (Expectation Maximization) algorithm [?]. 4.1 Base Models. The data fusion model is useful for building individual predictive models that are well suited for sub-domains of an application. In the context the S3, these models correspond to each data-tracking domain and represent different aspects of the learning process. That is, each model is designed for a particular domain of learning behaviour. An initial set of domains are defined as: Attendance, Completion, Participation, and Social Learning. Consider the attendance domain: learner tracking data reflecting online attendance is collected, including for example, number of course visits, total time spent, average time spent per session, in addition to other administrative aspects of the elearning activities such as number of visits to the grade tool, number of visits to the calendar/schedule tool, number of news items/announcements read. A simple logistic regression model, or a generalized additive model, is suitable for this domain. On the other hand, in the case of the social learning domain, social network analysis SNA techniques would need to be applied. The work by [?] demonstrates the key importance for specialized analysis of this aspect of the elearning process. In fact, SNA, in conjunction with text mining on learners discourse, is needed for the extraction of meaningful risk factors and success indicators. In other words, the logistic regression model described above for the attendance domain is considered insufficient for meaninful predictive analysis of the social learning domain. In S3 predictive models for each domain are built independently. Each generate an abstracted success sub-indicator represented as a predicted class and an associated probability estimate (ˆ, p), where p = p(Y = y |X), and X denotes domain-related activities being tracked. 4.2 Combining Model Ouputs. A key design aspect of ensemble systems is the combining process. Combination strategies for ensemble systems are characterized along two dimensions [?]: (1) trainable versus non-trainable rules, and (2) applicability to class labels versus class-specific probabilities. By selecting a trainable rule, the blending weights associated with the prediction of individual models are optimized to obtain a best-fit meta-model. By selecting a non-trainable combination rule, the business user is able to adjust the weight of the base predictions. For example, in a hybrid course where emplasis on discussion and social learning are primarily conducted face-to-face, the instructor can choose to dampen the effect of the social learning model from the overall prediction. The proposed ensemble system takes advantage of the estimated probabilties in combining the base predictions. In S3, there are three risk-levels, and each base model generates as output a vector of three probability values corresponding to estimated probability for each of the levels “At-Risk”, “Potential Risk”, “Success”. Let {g1 , g2 , . . . , gL } denote the learned prediction functions of L predictive models with gi : X i → (Y, p ∈ [0, 1]c )), ∀i, where Y are the risk categories, p is the associated probablity vector, and c is the number of risk categories, i.e. c = 3. For the described instance of S3 we have L = 4 corresponding to each of the data-tracking domains, at the course grouping/template level. The meta-model takes as input a matrix G with c = 3 columns represent the risk categories and L = 4 predictive models, where gij represents the probablity of risk-level j according to predictive model gi . It also takes as input the corresponding true outcomes y in the training dataset. A simple non-trainable combining process would be to average the values gij for each column of G. Normalization to add to 1 over all categories may be applied. Then, the maximum likelihood principle is applied by selecting the risk category with maximum posterior probability as the aggregated success indicator. Alternatively, the outputs of the base models are used as input to find the best-fit secondlevel mapping between the ensemble outputs and the correct outcome (risk level) as given in the training dataset. Typically, to find the best-fit meta-model, an iterative k-fold cross validation process is applied [?]. The training dataset is divided into k = L blocks and each of the first level model is first trained on L − 1 blocks, leaving one block for the second-level model, at each iteration through the L blocks. The process is designed to achieve a reliable model fitting. Linear regression stacking seeks a blended prediction function b represented as b(x) = i wi gi (x), ∀x ∈ X, where a key advantage of this linear model is that it lends itself naturally to intepretation. Furthermore, the computational cost involved in fitting such a model is modest. 5. CONCLUSIONS. We proposed a holistic ensemble-based analytical system S3 for tracking student academic success. From a design perspective, the unique synthesis of using predictive models to identify at-risk students, creating data visualizations to reach diagnostic insights, and incorporating a case-based methodology for managing interventions provides a just-intime mechanism and personalized approach to improving student retention and student success. From a research perspective, an ensemble-based approach to predictive modeling using semantic decomposition overcomes two significant shortcomings in current approaches, namely generalizability and interpretability. Ensemble methods are designed to boost the predictive generalizability by blending the predictions of multiple models. In S3, a stacked generalization strategy is applied to combine the predictions of a collection of base models via a second-level predictive modeling algorithm, a meta-model. The second-level algorithm is trained to combine the input predictions optimally into a more informed set of predictions. To facilitate model interpretability, abstraction of the elearning process into meaningful domains in conjunction with data visualization, interactive and intuitive interface are all part of S3. Furthermore, S3 can be tuned by business experts to best suit their needs. Future work will apply ensemble techniques to real datasets to demonstrate the full power of this methodology."