"1. INTRODUCTION. This paper reports on the first iteration of a design-based research project focused on developing an early warning system (EWS) for a mentoring program that supports undergraduate engineering students. The EWS described in this paper represents an application of learning analytics that is gaining popularity across colleges and universities—the near real-time aggregation and analysis of students' use of campus information technology (IT) systems to support the immediate identification of students in need of academic support (e.g., [1; 2]). The mentoring program for which our EWS was developed, called the M-STEM Academy (Michigan Science, Technology, Engineering, and Mathematics Academy) [3], provides an integrated student development program for first- and second-year undergraduate engineering students. The M-STEM Academy is aimed at increasing academic success and retention of students who, for reasons of socioeconomic status, first generation college status, racial or gender bias, or lack of rigor in their high school preparation, might not be successful at a highly competitive, elite research university. For the first two years of their undergraduate career, students enrolled in the M-STEM Academy participate in a variety of core activities including a summer transition program, a program living community, peer study groups, and monthly meetings with their academic advisor, or ""mentor"". The M-STEM Academy is an academic support program modeled on the Meyerhoff Scholars Program at the University of Maryland-Baltimore County [4] and the Biology Scholars Program (BSP) at the University of California Berkeley [5]. The EWS developed in this project provided Academy mentors with frequent updates on students' academic progress and streamlined the presentation of data to allow immediate identification of students in need of support. Research on ""Data Driven Decision Making"" conducted on in K-12 schools has shown that the access to data does not automatically lead to improvements in organizational functioning or student learning [6]. In our research program, we viewed mentors as critical to increasing students' learning. We provided EWS data to mentors on a weekly basis, allowing mentors to engage students to different degrees or in different ways as a result of having access to the EWS data. Based on more frequent interactions with better informed mentors, students can engage with academic support services in a more timely manner and by doing so may positively affect their course performances. With this model, we provided mentors' with access to data but allowed them to determine the frequency and intensity of connecting students to academic support services provided by the M-STEM Academy and the College of Engineering. 2. BACKGROUND. Our design-based research project draws from previous research on utilizing data from Learning Management Systems (LMSs) for the purposes identifying ""students at risk"" of academic failure. Below, we describe recently deployed solutions that aim to leverage LMS data to improve various stakeholders' decision-making, and ultimately, retention and academic success of students. 2.1 Leveraging LMS Data. The vast majority (over 95%) of postsecondary institutions support, augment, and facilitate classroom instruction using a LMS [7]. Given the penetration of LMSs throughout higher education and the breadth of use on most campuses, these systems offer a ready source of near real-time data that can be analyzed to support a multitude of academic decisions for an increasing number of students. Our research lies at the intersection of learning analytics-based interventions and mining LMS data, where the use of LMS data, in particular, has been demonstrated to provide a useful stream of data to support just-in-time decisionmaking around students' academic performances [9]. Prior efforts to create ""early warning systems"" that alert students and/or faculty to sub-par student performance have generated useful proofs of concept. For example, Morris, Finegan, and Wu [10] found positive correlations between how long students remain on an LMS course site and the degree to which they participate in online discussions with students' likelihood of persisting in an online course. Focusing on online courses, Goggins, Galyen, and Laffey found that students were able to use LMS feedback to identify what their peers were doing, and what they, in turn, might need to accomplish in order to catch up to the ""herd"" [11]. Macfayden and Dawson [9] attempted to identify specific online activities—supported by an LMS—that have unique predictive power; using a variety of statistical techniques, these authors used LMS data to correctly identify 81% of students who failed the course. Two recently deployed systems are examples of scalable learning analytics products in higher education. At Purdue University, Course Signals [1] analyzes students' use of the campus LMS to provide students feedback on how their use of the system corresponds to their possible future grade in a course. At the University of Maryland-Baltimore, the Check My Activity [2] tool supports students' awareness of how their use of the LMS and their current grades compares to that of their peers. The Course Signals project represents the most robust EWS using LMS data; however, this project has yet to pair data visualizations with specific interventions, relying instead on generating notifications to students and instructors without identifying how instructors or students can best use this information. The study presented here addresses this gap between identification of a problem and defining specific interventions. 3. METHODOLOGY. Our research agenda is organized around the principles of design-based research [e.g., 12]. Design-based research involves the iterative production of theories, artifacts, and practices that can potentially impact teaching and learning. A distinguishing feature of design-based research is that the development of learning tools and environments is a collaborative effort among researchers and practitioners [13]. 3.1 Participants. Four cohorts, roughly 50 students each, have been recruited and admitted into the 2-year M-STEM Academy. Data from these four cohorts show that after participating in the Academy, the MSTEM students have GPAs at or above those of their peers in the College of Engineering (CoE), and that they outperform the average CoE student in the three entry-level math courses [3]. Although this represents overall success for the M-STEM Academy, there are still individual students who ""fall between the cracks"" in the mentoring program—especially after the first year—and their failure to perform well is not discovered until final course grades are submitted. The EWS described in this paper tracked approximately 200 M-STEM students across 165 unique courses. 3.2 Procedure. Our work with the M-STEM Academy began in the Winter 2011 term. We provided grade information stored in the LMS to mentors on a weekly basis during the middle weeks of the term. Also during this time, we conducted multiple collaboration sessions with mentors to determine their information requirements and to get feedback on the earliest versions of the EWS. From these sessions, we developed a plan for displaying data, clarified ways in which data visualizations would be helpful to mentors, and identified how mentors used the initial data visualizations before and during mentoring sessions with students. Below, we discuss the specific development of the EWS, and the mentors' reactions to and use of the system. 4. FINDINGS. Data to support the development of the EWS were drawn from the campus LMS. Up-to-date grades from the LMS's Gradebook and Assignments tools were used to track students' performance. Further, we integrated a proxy for student ""effort"" in a course through the use of LMS course website login events. While each tool on the LMS tracks user actions, the number of times that a student logs into a course website provides a parsimonious metric that can be used across all course websites. These data sources were aggregated and translated into a variety of visualizations. Data visualizations included figures displaying students' developing grades and use of the LMS in comparison to their peers, students' performances on specific assignments, and weekto-week classifications of students. These week-to-week classifications were the primary features of the EWS, designed to provide mentors with an easy-to-interpret assessment of students' current academic performances. Based on specific combinations of students' grades and course site login frequency, the EWS displayed whether mentors should take one of three actions: ""encourage"" students to keep doing well, ""explore"" students' progress in more detail, or immediately ""engage"" students to assess possible academic difficulties. To develop the classifications of encourage, explore, and engage, we worked closely with M-STEM Academy faculty and mentors. Classifications of encourage, explore, and engage were generated using three rules: (1) whether a student's percent of points earned was at or above the thresholds of 85%, 75%, or 65%; (2) whether a student was 10% or 5% below the course average in percent of points earned; and (3) whether a student was below the 25th percentile in number of logins. These rules put a premium on students' absolute performances as well as their performances relative to their classmates'. Each student's percentile rank for course site logins was used to classify students close to the borderline on the first two rules. One of the difficulties of using system log data to gauge student performance is the variation in how LMS tools are used course-tocourse. There are two general strategies for overcoming this variability. First, create a composite metric that integrates some combination of tools used on the course site whereby this metric is in units that locate students within an intra-course distribution. Second, identify system events that are shared across all course sites. We chose to work with only the events that are shared across all course sites given the transparency of what these events track; the mentors interpreted this as an acceptable proxy for student effort. Classifications were aligned to a single student enrolled in a single course; for example, a single student enrolled in four courses would receive four classifications. Thus, all comparisons between students were intra-course comparisons and the statistics used, such as percentile ranks, were appropriate for such comparisons. Using a variety of data mining techniques as well as feedback from mentors, we continuously modified the thresholds for each of the above rules and are currently testing the accuracy of the rules across larger samples of courses and students. Data visualizations and classification schemes were compiled and released to mentors on a weekly basis. The EWS was comprised of a summary sheet and multiple, individual student display sheets (see Figure 1). The summary sheet provided a broad overview of all students and the student detailed sheets provided figures depicting an individual student's academic progress. These detailed individual student displays supported mentors in examining students classified as ""explore"" or ""engage"". Figure 1. Screen Shots of Data Displays for Academy Mentors. Summary Display (Left) and Detailed Display (Right) Examples. Because students transition to a less structured participation in the M-STEM Academy during their second year followed by a transition to the general engineering student population in their third year, the mentors became interested in tracking upper division students. Almost immediately upon receiving the first set of performance indicators from the EWS, mentors noticed multiple third- and forth-year students who had fallen behind. They then contacted these students to schedule an advising session where the student received guidance and suggestions for how to improve their grades. Currently, mentors are immediately contacting any student classified as ""engage"", regardless of their year in the program. The mentors report scheduling meetings with these students to discuss their academic performance and to identify the most appropriate course of action. In our future work, we will review the current version of the EWS with academic mentors. We also plan to use this version of the EWS as a template for a web-based version that will allow us to more easily merge various datasets and mathematical functions, as well as integrate data from the university's larger data warehouse. Lastly, we intend to closely monitor the types of services recommended by mentors and the degree to which students follow up on mentors' recommendations to utilize those services. 5. DISCUSSION. Our goal with the M-STEM Academy design-based research project is not simply to design data displays for Academy mentors, but to use this unique group of faculty, staff, and students as a test bed for scaling the EWS to the College of Engineering and, eventually, to the entire university. By beginning the research project with an eye towards broad scale, we hope to avoid the difficulties that can develop when tailoring analytics to a small population that cannot then be implemented with a wider group [e.g., 8]. An important aspect of learning analytics research that surfaced in our investigations of the MSTEM Academy concerns organizational capacity. Though there are multiple definitions (e.g., [14]), here we define organizational capacity as the resources and routines that both enable and constrain access and application of learning analytics tools and related services. Resources include the actual support services available to (all) students, as well as the availability of dedicated mentors or advisors who have time to meet with and assess students' needs. More generally, the culture of data use (e.g., whether data is valued for supporting decision-making as well as that which is considered to be valid data) within an academic unit, such as a school or college, can be an important resource that enables the use of learning analytics. A culture of data use, moreover, can be instantiated in specific policies that incentivize data use or restrict key actors' access to data. Thus, factors such as values and policies are not solely barriers, though often framed as such, but rather possible mechanisms that can be targeted to improve the effectiveness of learning analytics-based interventions. Thus, one of the important resources for supporting student success is ensuring their access to academic mentors and advisors. Faculty members may be too burdened with teaching, research and service to effectively interpret the complex data that is available from the Learning Management System without significant training efforts that have proven difficult at a large research institution like Michigan. Furthermore, while competitive students are typically motivated to continually check on their own progress and standing within their courses (hence the pattern of use seen in Fritz's work [2]), students who are already falling behind are less likely to engage in help-seeking behavior and can therefore become ""stuck"" in a pattern that they do not know how to appropriately confront. Designated academic advisors are uniquely positioned to discuss course requirements with individual faculty and apply that knowledge when engaging with students. Helping these advisors to know which students to contact during the term will allow the students who have the greatest need to get connected with the resources already available and improve their chance of success before the final course grade is ultimately determined. The M-STEM Academy model has already received acclaim and support throughout our university administration and has spawned new initiatives in biology and other STEM disciplines. With that commitment has come an acknowledgement that in order to scale appropriately, the staff and faculty supporting these academies need data about their assigned students in order to have timely interventions with students before they are in serious jeopardy of academic failure. We believe that our model of presenting data to academic mentors about students' performance and effort is not only scalable, but also particularly appropriate given the complexities involved in course planning, locating academic services, and cultivating positive academic behaviors. In our future work, we are exploring several additions to our system that can make the task of identifying students who need assistance easier by leveraging appropriate data sources and technologies. First, we plan to include displays of admissions and registrar data that can help inform students who might be predisposed to greater difficulty in particular courses. We believe that this type of information could be valuable at the start of term as well as when students are planning their course schedules. We plan to test this hypothesis as our design-research project progresses. We will also investigate the potential to integrate additional LMS data (e.g., file downloads) into our existing model. Additionally, we are planning to construct tailored messages to students. These messages will encourage students to be pro-active about seeking advising services and utilizing the support mechanisms available to them that are most appropriate for each course (e.g., the science learning center for physics courses, the writing center for English courses). This type of message will supplement, rather than replace, the role of the academic advisor. Our overall goal is to put the most timely and helpful data in the hands of individuals who can analyze, interpret, and translate the presented data into actionable strategies to support student success. Learning is complex and so too, justifiably, is the task for leveraging appropriate analytics to serve that endeavor. 6. ACKNOWLEDGMENTS. Our thanks to the continued participation and feedback from our partners in the M-STEM Academy including the academic mentors Darryl Koch, Mark Jones, and Debbie Taylor and staff members Cinda-Sue Davis and Dr. Guy Meadows. Also thanks to Gierad Laput for recent efforts for future work in this project and to Dr. Timothy McKay for his feedback about the application of this research program at the University of Michigan."