"1. INTRODUCTION. The first year of college is arguably the most critical with regard to the retention of students into subsequent years of study [2, 3, 8, 9]. Noel and Levitz indicate that retention, or the lack of attrition from college, is a by-product of student success [4]. Tinto has spent much of his career investigating the necessary conditions for student success, and notes that academic support is among the primary pieces necessary to ensure success. In his 1993 book, Leaving College, Tinto proposed three necessary conditions for student persistence. First, an institution needed to put programs into place that placed the welfare of the students higher than that of the university. Second, programs and solutions should be focused on all students at an institution, not just a specific subpopulation. Finally, solutions implemented to enhance student success, and therefore persistence, needed to help integrate a student academically into the institution [6]. Helping a student become academically integrated to the institution is key, as Course Signals helps to promote integration in several ways. First, it allows faculty members to send personalized emails to students that contain information about their current performance in a given course. Second, faculty members can encourage students to visit various help resources on campus or office hours – activities that contribute to a student becoming more fully integrated into the institution. Third, it employs learner analytics to allow for the integration of real-time data on student performance and interaction with the LMS with demographic and past academic history information. This combination creates an intentionally created environment for the students that does “not leave learning to chance,” something Tinto noted was necessary to ensure that a solution would be broadly effective in helping students persist to graduation [7]. The remainder of this paper will describe Course Signals in detail, including its development and outcomes realized as a result of its implementation. In addition, faculty and student perceptions will be shared. 2. COURSE SIGNALS OVERVIEW. 2.1 Description of Course Signals. Course Signals (CS) is a student success system that allows faculty to provide meaningful feedback to student based on predictive models. The premise behind CS is fairly simple: utilize the wealth of data found at an educational institution, including the data collected by instructional tools, to determine in real time which students might be at risk, partially indicated by their effort within a course. Through analytics, large data sets are mined and statistical techniques are applied to predict which students might be falling behind. The goal is to produce “actionable intelligence” —in this case, guiding students to appropriate help resources and explaining how to use them. [1] A predictive student success algorithm (SSA) is run on-demand by instructors. CS works by mining data from multiple university sources and subsequently transforming the data into a generated risk level with supporting information for each student [1]. The algorithm that predicts students’ risk statuses has four components: performance, measured by percentage of points earned in course to date; effort, as defined by interaction with Blackboard Vista, Purdue’s LMS, as compared to students’ peers; prior academic history, including academic preparation, high school GPA, and standardized test scores; and, student characteristics, such as residency, age, or credits attempted. Each component is weighted and pulled into the proprietary algorithm, which then calculates a result for each student. Based on results of the SSA, a red, yellow or green signal is displayed on a student’s course homepage. A red light indicates a high likelihood of being unsuccessful; yellow indicates a potential problem of succeeding; and a green signal demonstrates a high likelihood of succeeding in the course. Instructors then implement an intervention schedule they create, possibly consisting of: - Posting of a traffic signal indicator on a student’s LMS home page; - E-mail messages or reminders; - Text messages; - Referral to academic advisor or academic resource centers; or, - Face to face meetings with the instructor. [1] With Course Signals, students are not placed at risk due to one single factor; risk is determined by a contextualized landscape that varies from student to student based on the data comprising the four components of the SSA. The SSA transforms both static and dynamic data points into a single score, improving the reliability of the prediction. Since a course-specific risk indicator is created for each student based on performance, peer-based behavior, and educational preparation data, instructors can intervene early and give students a realistic opportunity to adapt their behavior to be more specific in a given course. 2.2 History of Course Signals. Facing challenges of under prepared students, budget crises, decreasing retention and longer graduation periods, higher education is working to provide solutions to these challenges while at the same time balancing the demands of providing exceptional student service to foster student success. In an attempt to ease these mounting pressures, CS was developed to help identify students potentially at risk of not reaching their full potential in a course. Once identified, instructors have the ability to deliver meaningful interventions suggesting behaviors a student may wish to change in order to improve her chances of success. [1] In 2007, Purdue University piloted Course Signals. According to Pistilli and Arnold, the system “was built from the ground up using empirical data at every stage to ensure the most predictive student success algorithm” [5]. Course Signals became automated in spring 2009 and partnered with SunGard Higher education in October 2010 in order to help other institutions harness the power of learning analytics. Today, nearly 24,000 students have been impacted by the CS project, and more than 145 instructors have used CS in at least one of their courses. 3. ACADMIC PERFORMANCE AND RETENTION OUTCOMES. 3.1 Impact on Academic Performance. Undeniably, one performance measure of student success is final course grade. Research indicates that courses that implement CS realize a strong increase in satisfactory grades, and a decrease in unsatisfactory grades and withdrawals. Individual courses see variable success with: an increase in As and Bs ranging from 2.23 to 13.84 percentage points; a decrease in Cs ranging from 1.84 to 9.38 percentage points; and a decrease in Ds and Fs ranging from 0.59 to 9.40 percentage. Combining the results of all courses using CS in a given semester, there is a 10.37 percentage point increase in As and Bs awarded between CS users and previous semesters of the same courses not using CS. Along the same lines, there is a 6.41 percentage point decrease in Ds, Fs, and withdrawals awarded to CS users as compared to previous semesters of the same courses not using CS. 3.2 Impact on Student Retention. With increased student success in individual courses comes an expected increase in retention to the University as well, and the data indicate this nicely. Course Signals has been employed at Purdue since 2007, and its use for each beginning cohort at the institution is described in detail below. 3.2.1 Methodology. The fall 2007, 2008, and 2009 beginner cohorts were compared to a master list of all Course Signals participants to determine who from those entering cohorts took courses utilizing Course Signals or not, and, if applicable, the number of times students took courses with Course Signals. From there, students were put through the retention module in the University’s data reporting system. They were analyzed based on the number of times they had a course with CS – a number ranging from zero to five. The beginner cohort for each year consists of the students who are both first-time in college and carrying full-time credit loads. The students who comprise each cohort is determined the second week of each fall semester, and this data set is frozen; once created, students do not leave the cohort for any reason. Each semester, every student in each cohort has some form of retention or leaving behavior – from simply remaining enrolled, to graduating, to being academically dropped or withdrawing voluntarily. The retention rate is calculated by adding those who are still enrolled and who have graduated and dividing that sum by the total number of first-time full-time students in the original cohort. 3.2.2 Results. As indicated below, the students who began at Purdue in fall 2007 (Table 1), 2008 (Table 2), or 2009 (Table 3) and participated in at least one Course Signals course are retained at rates significantly higher than their peers who had no Course Signals classes but who started at Purdue during the same semester. Further, students who have two or more courses with CS are consistently retained at rates higher than those who had only one or no courses with Signals. The analysis detailed in Tables 1, 2, and 3 does not account for when a student had a course with CS, only that at some point during their academic career they did. Tables 4, 5, and 6 examine that aspect. For the 2007, 2008 and 2009 cohorts, instances of CS use across successive semesters was compared to students’ retention behavior for the following semester. This analysis asked if, within a set of semesters, if a student had at least one course with Course Signals. So, for example, for the 2007 cohort, the first row looks at whether or not a student had CS in a course during either the Fall 2007 or Spring 2008 semester, then determines if they were retained into the Fall 2008 semester. The comparison is against students who did not have a course with CS during the same time period. In short, there is a noticeable impact on students having a course with CS early in their academic career; basically, the earlier a student encounters CS the better. Combined with the first analysis, the earlier and the more occurrences, the greater the likelihood students will be retained. Table 1. Retention Rate for the 2007 Entering Cohort. Table 2. Retention Rate for the 2008 Entering Cohort. Table 3. Retention Rate for the 2009 Entering Cohort. Table 4. Analysis of Retention by Semester of Course Signals Use for the 2007 Entering Cohort. Table 5. Analysis of Retention by Semester of Course Signals Use for the 2008 Entering Cohort. Table 6. Analysis of Retention by Semester of Course Signals Use for the 2009 Entering Cohort. In addition to the previous analyses, it should be noted that in every case for the students from the 2007, 2008, and 2009 cohorts, students in courses with CS have a lower average standardized test scores than those in non-CS courses. While this aspect needs to be further investigated, early indications show that lesser-prepared students, with the addition of CS to difficult courses, are faring better with academic success and retention to Purdue than their better-prepared peers in courses not utilizing Course Signals. 4. FEEDBACK FROM STUDENTS AND INSTRUCTORS. While the quantitative data provide evidence that there is an impact on students’ grades and retention behavior, there exist additional data that support the use of CS. The instructors who have employed CS, as well as students who have benefited from the system, have provided information via surveys, focus groups, and interviews that continue to warrant the usage of the system. 4.1 Student Perception and Feedback. One of the major objectives of academic analytics is to identify underperforming students and intervene early enough to allow them the opportunity to change their behavior. For this reason the Course Signals development team has closely tracked the student experience with Signals since the pilot stage. At the end of each semester, a user survey gathers anonymous feedback from students, with more than 1,500 students surveyed across five semesters. In addition, several focus groups have been held. Students report positive experiences with Course Signals overall (89% of respondents stated CS provided a positive experience and 58% said they would like to use CS in every course). Most students perceive the computer-generated e-mails and warnings as personal communication between themselves and their instructor. The e-mails seem to minimize their feelings of “being just a number,” which is particularly common among first-semester students. Students also find the visual indicator of the traffic signal, combined with instructor communication, to be informative (they learn where to go to get help) and motivating (74% said their motivation was positively affected by CS) in changing their behavior. Of the roughly 1,500 student responses, only two wrote of becoming demoralized by the “constant barrage” of negative messages from their instructor. While this perception should not be downplayed, negative feedback from instructors, especially for students who might not be prepared for the rigors of higher education, can be difficult to receive. Aside from these two instances, however, the remainder of the negative feedback concerned faculty use of the tool. For example, many students spoke of over penetration (e-mails, text messages, and LMS messages all delivering the same message), stale traffic signals on their home pages (an intervention was run but not updated, giving a false impression of a student’s status), and a desire for even more specific information. This information notwithstanding, the overwhelming response from the students is that Course Signals is a helpful and important tool that aids in their overall academic success at Purdue. Faculty believe this as well, as indicated in the following section. 4.2 Faculty Perception and Feedback. While the student success algorithm predicts which students might be in jeopardy of not doing as well as they could in a course, it is the faculty and instructors who use the information provided by Course Signals to intervene. It could certainly be argued that these instructors, armed with the data provided by learner analytics, are the most important weapons against student under performance in the classroom. To wit, one instructor asserted that “I want my students to perform well, and knowing which ones need help, and where they need help, benefits me as a teacher.” Faculty have easy access to CS data via the faculty dashboard. Using learner analytics, faculty can provide action-oriented and helpful feedback much earlier in the semester, which students appreciate. This particularly benefits students early in their academic careers, as they often are not fully aware of the behaviors they must exhibit or actions they must take in order to be successful. Faculty also say that students tend to be more proactive as a result of the Course Signals interventions. While students still tended to procrastinate, they began thinking of big projects and assignments earlier. Instructors and TAs also noticed that students posted more questions about assignment requirements well before the due dates. Because of the ability of academic analytics to assess risk early and in real time, the instructors consistently indicate that students are benefiting from knowing how they are really doing in a course and, moreover, understand the importance of completing assignments, and performing well on quizzes and tests. In general, faculty and instructors have a positive response to CS but many approached the system with caution. Before using Course Signals, faculty initially expressed concern about floods of students seeking help; however, few actually reported problems after they began using the system. The most commonly reported issue being an excess of e-mails from concerned students. In addition, faculty reported concerns about creating a dependency in newly arrived students instead of the desired independent learning traits. A final faculty concern was the lack of best practices for using CS, demonstrating that instructors and students share the same concern about the lack of best practices. There is little that can be done to mitigate the first concern, since one of the goals of CS is to have students take a more active role in their success. The second concern is mitigated by the strong retention results discussed in this paper. The final issue was addressed by creating and posting best practice tips at http://www.itap.purdue.edu/learning/tools/signals. 5. CONCLUSION. The use of learner analytics through the application of Course Signals to difficult courses has shown great promise with regard to the success of first and second year students, as well as their overall retention to the University. To date, over 23,000 students across 100 courses have been impacted by Course Signals and over 140 instructors have utilized the system. Plans call for the expansion of CS to include as many as 20,000 students a semester within the next 18 months, and the upper administration the institution is in strong support of this goal. While this analysis is not without its limitations or areas for continued improvement of the algorithm behind CS or the use of CS, the outcomes are such that continued use of Course Signals as a means of helping instructors provide detailed feedback to their students, and to ultimately assist students in their academic endeavors is highly warranted. 6. ACKNOWLEDGMENTS. Our thanks to Kyungmin “Mike” Ahn for his work analyzing the data associated with the retention numbers."