"Introduction. The promise of learning analytics. Learning analytics employs sophisticated analytic tools and processes in investigation and visualization of large institutional data sets, in the service of improving learning and education (Brown, 2011; Buckingham Shum & Ferguson, 2011). Building on the demonstrated strategic advantages of “business analytics” in the corporate world, learning analytics also draws on the related fields of web analytics, academic analytics (Goldstein & Katz, 2005), educational data mining (see Romero & Ventura (2010) for review) and action analytics (Norris, Baer, Leonard, Pugliese, & Lefrere, 2008) to support decision-making and strategic planning in academic settings. “Academic analytics” approaches are typically applied in educational settings to address administrative and operational concerns such as “advancement/fundraising, business and finance, budget and planning, institutional research, human resources, research administration, and academic affairs” (Fritz, 2011). Projects undertaken under the auspices of “learning analytics” extend the potential of analytics to the level of individual learning, by selecting, capturing and interpreting data on teaching and learning activities, with the goal of improving teaching and learning outcomes. Institutions and senior administrators are key users and stakeholders, and enhancement of institutional decisionmaking processes and resource allocation are core objectives (Romero & Ventura, 2010). In the postmodern context of constant and dynamic change in higher education and technological innovation, then, learning analytics offers higher education institutions a valuable tool in their ongoing efforts to select actions that are “achievable within the capacity of the organization to absorb change and resource constraints” (Kavanagh & Ashkanasy, 2006). The importance of strategic investment in learning technologies and e-learning In this paper, we present a learning analytics case study of LMS implementation and use in a large research-intensive university that routinely ranks within the top five in national magazine league tables (Macleans, 2010). The institution achieves high annual scores in the presage variables compiled in such rankings: institutional resources, research funding and reputation. We know from meta-analytic studies of decades of available data, however, that the quality of education offered by an institution is not predicted by the size of institutional budgets, numbers of or dollar values of research awards, or even by measures such as student: faculty ratios or “hours spent in class.” Instead, the best institutional predictors of educational gain are “measures of educational process: what institutions do with their resources to make the most of whatever students they have” (Gibbs, 2010, p. 2). Citing a major 2004 study (Gansemer-Topf, Saunders, Schuh, & Shelley, 2004), Gibbs (2010) argues that the feature that distinguishes effective institutions from less effective schools is their strategic use of available funding to support “a campus ethos devoted to student success” (p. 14). In other words, decision-making processes relating to organization of institutional resources – human and material – and planning for more effective use of existing resources are a critical feature of excellent institutions. Within the teaching context, Gibbs argues that the most significant predictors of educational gain “concern a small range of fairly well understood pedagogical practices that engender student engagement” (2010, p. 5). At least a decade of research and writing has demonstrated that learning technologies, when used appropriately, can help educators adopt the “seven principles of good practice in undergraduate education” (Chickering & Gamson, 1987) and improve the overall quality of an institution’s educational delivery (Chickering & Ehrmann, 2002). Moreover, recent work in the field of learning analytics has demonstrated that the communicative affordances of ICTs and learning management systems (LMSs) can increase student sense of community (Dawson, 2006) support learning communities and enhance student engagement (Dawson, Burnett, & O'Donohue, 2006; Dawson, Heathcote, & Poole, 2010; Macfadyen & Dawson, 2010). The teaching climate within higher education is becoming increasingly complex. Student enrollment numbers continue to rise (Patrick & Gaële, 2007) and universities are catering to an increasingly diverse student body (living far from campus, studying part-time, returning to education after a long break or juggling the demands of study with career or family life (OECD, 2008; Twigg, 1994)). In this context, learning tools that support and enhance student engagement with peers, instructors and learning materials have become essential enterprise resources. It should be no surprise then, that, like 93% of US-based higher education institutions (Campus Computing, 2010), the institution in this case study has invested heavily in the campus-wide implementation of a web-based LMS since the late 1990s. The institution also hosts and supports a number of additional learning technology platforms (e.g., WordPress, MediaWiki), and subscribes to others (e.g., Turnitin). The LMS is therefore embedded within a wider network of platforms and systems involved in supporting the teaching and learning mission, and is viewed as a core component of the university’s teaching infrastructure. The catalyst for change. Given this university’s substantial investment in an institutional LMS, and the ever-evolving market in LMSs and learning technologies, as well as shifting economic conditions (Campus Computing, 2010), strategic decisionmaking and forward planning regarding technology choices and related resource allocation are clearly of the essence. Reviews of available learning technologies have routinely been undertaken as part of the institution’s standard quality assurance practices. These reviews have aimed to evaluate the current state of use, to ensure that the suite of adopted technologies reflect the broader learning and teaching mandate and to ensure that the university is deploying its limited resources most effectively to support learning and teaching. In addition, in 2010, a further catalyst for a new LMS review was the LMS vendor’s announcement that the current LMS product would not be supported after 2013. Together, these conditions generated the necessary impetus for the next round of institutional review, with the goal of selecting as the next enterprise LMS the product that would best support the university’s teaching and learning goals. In his well-established eight-step change model, Kotter (1996) notes that the critical first step in effectively managing change is one that creates a sense of urgency and the necessary levels of motivation required for sustaining the change process. This step involves a careful examination of the current context, to allow identification of potential “threats” and opportunities, and envisioning of future scenarios. In this light, the application of learning analytics focused on the current state of LMS integration presented an opportunity to understand the specific teaching and learning context, and develop a strategic vision and operational pathway for continual improvement. Employing e-learning analytics to undertake a current state analysis Gathering as much data as possible regarding current LMS usage across a university is no small feat, and is particularly challenging in such a highly decentralized institution as the one under study here, which comprises numerous Faculties/Divisions. Some LMSs do capture and store large amounts of course and user activity and interaction data. Until recently, however, investigators have only been able to access, aggregate, analyze, visualize and interpret this data via slow and cumbersome manual processes. While the majority of commercial and open source LMS are rapidly recognizing the importance for integrating sophisticated learning analytics, the associated reporting functionality is still largely under development (Dawson, McWilliam, & Tan, 2008; Mazza & Dimitrova, 2007). To overcome the challenge of poor analytics functionality in the current LMS, the university considered here has partnered with an analytics software company to customize and implement an analytics reporting tool that allows extraction, analysis and dis/aggregation of detailed information about uptake and use of the enterprise LMS. We made use of this analytics platform to carry out the requested “current state analysis” of LMS usage, and to seek answers to questions about the extent and complexity of LMS adoption. This analysis was undertaken with the goal of informing and guiding the institution’s campus-wide strategic planning process for learning technology and LMS integration. The availability of the new e-learning analytics platform allowed us to undertake, on the university’s behalf, the most comprehensive examination of its LMS use to date. The process revealed, and will continue to reveal, an array of LMS use patterns and practices, finally allowing the university to “know itself” in terms of learning technology uptake and integration in the service of teaching and learning. We hoped that this e-learning analytics exercise would provide compelling data that would generate the sense of urgency necessary for motivating broad scale institutional change associated with learning, teaching and technology. Through participant observation in the review and planning process we were able to investigate the degree to which the e-learning intelligence revealed influenced institutional decision-making. In this paper we present selected examples of data from the analysis. More critically, we discuss the reality that the data developed in this e-learning analytics process did not significantly inform subsequent strategic decision-making and visioning processes, and consider some of the factors that may have limited its impact. Approach and tools. Ethics and privacy. There are very real concerns about ethics and information privacy issues relating to the collection, analysis and dissemination of data on student, faculty and staff online activity and on student achievement and demographics. For this reason, our approach is informed by the institution’s policies on research involving human subjects, and the TriCouncil Policy Statement: Ethical Conduct for Research Involving Humans (“TCPS”) (Government of Canada, 2010). These policies mandate an ethical review process for research projects that involve human subjects. Furthermore, data that is gathered through institutional research is subject to the provisions of the Freedom of Information and Protection of Privacy Act (Government of British Columbia, 2012) (whereas data generated in the course of “traditional academic research” is under the control of the individual faculty members involved and exempted under section 3(1)(e) of the Act.). This research complies with all stated policies and in accordance with FIPPA, our data is protected. Data access is limited to a small number of research investigators; raw data is maintained on secure data servers only; and all individual identifiers are removed from any disseminated data or analysis. Selection of data. LMS usage data was mined for a single academic year (2009-2010). Only data for credit-bearing courses was examined (LMS adoption by the institution’s various continuing education units for non-credit programming, and usage by non-teaching units, was excluded, as well as numerous non-credit LMS-based online “training and orientation” modules created by Faculties and support units). Data on the total number of courses offered across the institution in the 2009-2010 academic year was prepared by manual analysis of course section listings exported from an institutional database listing all sections available for student registration in 2009-2010. Sections listed as summer courses, registration placeholders for overseas exchange courses, or registration placeholder course numbers for students completing Masters or Doctoral theses were removed, before determining section counts and statistics. The following section outlines some of the analyses that were undertaken. Analytics and data visualization tools. E-learning data presented in this paper was extracted, collated and analyzed using an e-learning analytics platform based on MicroStrategy Business Intelligence software that allows development of customized reports investigating user, course, Department, Faculty and institutional metrics during time periods of interest. User interaction data within the institutional network can be readily captured, categorized, and analyzed. These large data sets can be further interrogated to identify patterns of user-behaviour that can inform teaching and learning practice. The software makes use of a pre-built enterprise data warehouse, optimized for use by educational organizations, with common, conformed dimensions, enabling cross-platform intelligence. At the institution under study here, the analytics platform has been configured to pull course identifier data and student grades from the institution’s student information system, LMS data from LMS tracking and meta-data (captured on production servers). Data is presented to end users via a web interface with extensive report authoring, ad hoc querying, data analysis and report distribution capabilities. Selected data was also visualized using Tableau Desktop 6.1 data visualization software. (For more information http://www.tableausoftware.com/) Participant observation. To investigate the subsequent impact of our e-learning analytics reporting on institutional decision-making processes in relation to LMS selection and learning technology planning, we undertook a longitudinal participant observation process (Douglas, 1976). In participant observation, investigators are also subjects. This qualitative research methodology typically involves direct observation, participation in the life of the group, collective discussions, and analysis of documents produced within the group. It is usually undertaken over an extended period of time, ranging from several months to many years. The institution in question hosts a standing advisory committee on learning technologies that comprises at least 35 representatives from across its academic, information technology and learning technology units, and is jointly chaired by senior figures in academic affairs and information technology. We participated in and observed the activities and collective discussions of the committee over a period of approximately 18 months, during which time it was tasked with evaluating current usage of the institutions LMS and other tools, and development of a vision, roadmap and plan for the institution’s next generation learning technology environment. We also undertook review and analysis of public and private process documents developed by the committee. The lead author of this work participated with ‘observer status’ only, and played no role in decision-making processes. Selected findings. Institutional data indicated that in 2009-2010 academic year, a total of 18,909 course sections were offered (of which 14,201 were undergraduate course sections). This total includes 388 distance learning sections, of which 304 (1.6% of total sections) were offered in fully online format, and 84 in print-based format. Numbers of courses and students using the enterprise LMS. Assessment of the proportion and characteristics of course sections implementing the LMS provides a sound indication of broad-scale institutional adoption rates and overall diversity of adoption across year levels and class sizes. In this instance, 21% (3,905) of all course sections had an associated LMS course site. Based on institutional staffing and student enrollment figures for 2009-2010: - 80.3% of all students were enrolled in at least one LMS-supported course during the 2009-2010 academic year (total student enrollment: 52,917) - Most LMS-supported sections (61%) were employed for medium-sized course sections of 15-79 students. A further 22% of sections were employed for large classes of 80+ students. - 1,118 instructors or roughly 30% of all teaching staff used the LMS for instructional purposes (total teaching staff of 3,061, including part-time and full-time Professors; Associate, and Assistant Professors; lecturers; instructors; and clinical, visiting, adjunct and emeritus Faculty). The institution’s LMS is currently used by courses across all year levels (1st-4th year undergraduate courses, as well as graduate level courses), with roughly 14% of lower level course sections and graduate course sections, and 25% of upper level course sections making use of the LMS. Across the undergraduate years, numbers of unique student users are similar (ranging from 12,000-19,000 unique student users), demonstrating that in upper level (3rd and 4th year courses) the LMS is, on average, being used to support smaller course sections than at the lower level. While fully online courses represent only 1.6% of course section offerings in 2009-2010, 4,661 students or 11% of the total enrollment completed at least one fully online course during this period. LMS user time online. User time online within LMS-supported course sites varied immensely by user role (designer, instructor, teaching assistant, student), by Faculty, by Department, and by course mode (fully online versus LMS-supported). Table 1 shows comparative average user time online per term for LMS-supported and fully online courses. Table 1. Comparative user time online by role for LMS-supported and fully online course sections, 2009-2010. Average user time figures mask real variation between Faculties, Departments and even individual course sections. Students in LMS-supported courses in the Faculty of Arts, for example, spent an average of 7 ± 6 hours per course section using LMS-based course resources, while students in the Faculty of Agriculture spent an average of 16 ± 15 hours online per course section. Similarly, instructor time online varied tremendously, even when courses were taught in a fully online modality. Examination of instructor data for fully online courses shows a range of instructor time online from 61 ± 261 hours per section at the high end, to 5 ± 4 hours per section at the low end in the 2009-2010 academic year. What learners are doing online. Measures of “average time online” using an LMS is a crude indicator of student (or instructor) time investment in teaching and learning. In order to further unpack what students are doing while logged in to LMS-based course sites, we investigated data on LMS tool use. The current institutional LMS offers instructors and students a range of tools for presenting learning materials, communication, collaborative work, assessment and administrative tasks. In addition, a number of web-based products and services (Turnitin, MediaWiki, the Wimba suite of tools) offer “plugins” – known as Powerlinks – that allow their integration into an LMS-based course. Assessment of LMS tool “presence” in LMS-supported course sections during the period of interest shows that the standard suite of LMS tools are typically implemented (i.e., available for use) as well as a number of Powerlinks (data not shown). However, a more nuanced representation of LMS tool use is provided in Figure 1, which illustrates average actual tool use time per student (measured in minutes) for all available tools and Powerlinks in the 2009-2010 session. Figure 1. Student usage of LMS tools shown as minutes of use time per student enrolled in LMS-supported course(s), 2009-2010. Figure 2. Overall composition of the institution’s LMS-based course content, represented as relative numbers of each file type. To explore the nature of actual learning materials (i.e., course content) we investigated which file types are contained in the entire LMS course content database. The diversity and proportion of file types is represented in Figure 2. This data can provide insight into the types of learning and teaching approaches adopted for a particular course, or can allow a more generalized assessment across a Department or Faculty. Figure 3. Relative distribution of average student time per “learning activity category”, by Faculty, 2009-2010. The aggregation of tool use data based on tool “purpose” provides an effective method for interpreting the broad pedagogical intention of online learning materials and activities. Dawson (Dawson et al., 2008) has previously proposed that LMS tools can be broadly organized into four categories representing the core activities within LMSsupported and online courses: - Engagement with learning community - Working with content - Assessment - Administrative tasks Table 2. LMS tools assigned to “learning activity categories”. Figure 4. Correlation between student achievement and selected LMS tool use frequency in LMS-supported course sections. This categorization offers a useful approach for interpreting LMS tool use data, especially in light of increasing evidence that student engagement with peers in a learning community has the strongest positive effect on learning success (Astin, 1993; Light, 2001; Macfadyen & Dawson, 2010; Tinto, 1998). Table 2 outlines our assignment of currently available LMS tools into these learning activity categories. Figure 3 represents average learner time using each tool category in LMS-enabled course sections proportionately for each Faculty, regardless of absolute time use figures. This allows easy comparison of relative tool category use time per student between Faculties. LMS tracking data and student grade data for 95,132 undergraduate student enrollments in LMS-supported courses was merged, visualized and analyzed using Tableau 6.1. Student grades were binned into deciles and best fit lines determined. Correlation coefficients for the selected LMS activities shown here with binned student final grade are as follows: number of discussion messages posted, r = .83, p<.01; number of discussion messages read, r =.95, p<.0001, number of discussion replies posted, r =.94, p<.0001); number of content pages viewed (0.89, p<.001); number of visits to the “My Grades” tool (0.93, p<.0001). Subsequent data analysis confirmed and extended our earlier reporting of significant correlation between student learning outcomes (as represented by student final grade in the relevant course) and their use of engagement tools (discussions, mail) in fully online courses (Macfadyen & Dawson, 2010). Visualization of LMS use data for LMSsupported classroom-based courses again shows significant positive correlation between student participation in course-based discussions and their final grade (for number of discussion messages posted, r = .83, p<.01; for number of discussion messages read, r =.95, p<.0001, and for number of discussion replies posted, r =.94, p<.0001). A significant positive correlation with final grade is also observed with student use of LMS-based course content materials (0.89, p<.001), and also, most interestingly, with student visits to the “My Grades” tool (0.93, p<.0001) that allows students to monitor their own progress (Figure 4). Outcomes of participant observation. The institution’s standing committee on learning technologies convened monthly during 2010 and 2011, with the goal of developing a vision, roadmap and plan for selection of the new institutional LMS. A detailed analytics report on the current state of implementation and use of the institution’s existing LMS was presented at an early stage in this process. Subsequently, meeting minutes and reports on later stages of decision-making were made available to the university community (data not shown). From review of these documents, and from participation in continuing committee discussions, we observed that although completion of the current state analysis was noted, no further references to or interpretations of the findings were made in later meetings or documentation. Discussion and implications. Benchmarking the institution’s LMS usage. This e-learning analytics case study revealed multiple layers of data that can be re-purposed, aggregated and analyzed in new ways. By revealing details of actual LMS use patterns and their relationship to student learning outcomes, these data not only re-emphasize the value of the LMS in supporting student learning at the institution, but offer benchmarks by which the institution can measure its LMS integration both over time, and against comparable organizations. The 2010 Campus Computing Survey (Campus Computing, 2010) reports that US public universities now make use of an institutional LMS in delivery of an average of 60% of their course sections, suggesting that LMS uptake in the university under study here, at only 21% of course sections, is comparatively low. Similarly, a 2007 study (Allen, Seaman, & Garrett, 2007) reported that US higher education institutions offered an average of 10.6% (median 5%) of their course sections in fully online mode, and that this number appears to be increasing rapidly as institutions further embrace distance and flexible models of education. In this case study, the institution’s small offering of online courses (1.6% of total course sections) again indicates a low level of penetration. Furthermore, at least 70% of teaching staff did not make any use of the institutional LMS during the 2009-2010 academic year. While instructor use of the LMS may therefore be considered to be in the early adoption phase (Rogers, 1995), students are heavily exposed to the LMS, confirming the impression that the LMS is a core component of the institution’s overall learning experience. The vast majority (>80%) of students were enrolled in at least one course that made use of the LMS in 2009-2010, and 11% of students completed at least one fully online course. This latter figure, in particular, suggests that the student cohort is beginning to recognize the strategic advantages of online courses as they plan their timetables, meet program requirements and attempt to manage the time demands of work, study and commuting to campus. Workload and “time online” are core issues for teaching staff, and can also be significant areas of concern for students. With particular relevance to online courses, in which almost all courserelated activity is mediated by the LMS, data show a very wide range of total student engagement time with peers and course materials across different disciplines. These data begin to provide lead indicators of the appropriateness of the course load as a result of the implemented learning activities. A more detailed understanding of what, exactly, is occupying student time in LMS-supported course sites provides a more meaningful representation of how an LMS is being used, and therefore the degree to which LMS use complements effective pedagogical strategies. Contemporary educational theorists emphasize the importance of peer to peer interaction for facilitating the learning process (Astin, 1993; Chickering & Gamson, 1987; Light, 2001; Tinto, 1998). Nevertheless, when we examine the categories of activity that are occupying student time in LMS-supported course sites, it is clear that across the institution (and regardless of course mode), the dominant use of the LMS is for content delivery. This observation is further supported by the number of static text files contained within the LMS (Figure 2, text and pdf files). Adoption of technological innovations in a manner that simply replicates existing hegemonic practice (Reiser, 2007) is not limited to LMSs. Such “first stage” adoption appears to allow a familiarization phase, before broader innovations can be undertaken. It is only at this later innovation stage that learning technologies will be fully utilized to support a pedagogical practice of engagement that will significantly enhance the overall student learning experience. However, this will also necessitate the kind of cultural changes described by McWilliam (2005). A wealth of literature describes the enriched learning possibilities permitted by such a shift. In relation to LMS functionality, while more than half of all LMS-supported courses at the case study university implemented a common suite of tools, mining the data on actual student use time for tools assists with overall interpretation for informed action. The current LMS tools that support online discussions, presentation and organization of course content, and assessment activities (usually quizzes) are the only tools which are heavily used. It can be argued that use of some LMS tools simply require little time investment – for example, tools that allows students to read a quick announcement, check their grades, or upload assignments. It is clear, however, that a range of available tools that could be used to increase student engagement and collaboration (MediaWiki, the Wimba suite of voice and video tools) remain poorly utilized. Further investigations are required to better understand why and how the adoption of these resources can be improved. Together, these findings indicate that the institution has some distance to go in maximizing effective and strategic use of its enterprise LMS. Informing strategic planning?. Although the data gathered in this case study analysis suggest that the potential use of the enterprise LMS is yet to be realized at this institution, it nevertheless confirms that the LMS is central to the student learning experience – a reality that should highlight the importance of careful planning for future learning technology uptake. The mandate of the standing committee on learning technology at this university is that it will support the institution's teaching and learning mission by assisting in the development of a campus-wide vision for technology use, and will lead the planning process for technology implementation. It is the only group explicitly tasked with integrating technology with the institution’s learning and teaching mission. With this in mind, it might seem surprising that subsequent steps in the institutional LMS review process did not appear to incorporate or build upon the intelligence revealed by this learning analytics exercise. While this committee might be considered to be the “powerful coalition” that Kotter (1996) identifies as a key actor in motivating and managing successful change, their subsequent discussions and deliberations did not include any critical consideration of current LMS use patterns in the development of a vision and strategic plan. Presentation of data indicating apparent correlations between student online engagement and student achievement did not catalyze debate about the pedagogical benefits of technology, or about whether the institution as a whole appears to be making best use of available learning technologies. In essence, the findings derived from the learning analytics process failed to generate the sense of urgency or motivation for change as it related to technology adoption within the institution. Diverse approaches to change management and leadership (see, for example, literature cited in Kavanagh & Ashkanasy (2006)) agree that development of an organizational vision, and a strategy by which to reach it, is a critical step. In this case study, learning analytics offered the institution a means of measuring its current state and future progress towards an institutional vision for teaching and learning with technology. However, through participant observation of committees responsible for moving the institutional LMS review and selection process forward, we noted that subsequent deliberations and decision-making focused almost exclusively on technical questions relating to “ease of migration.” Critical interpretation of the implications of data describing the institution’s current LMS use was almost entirely absent. These observations are reflected in public and private reports documenting the committee’s activities (not shown). While there is an obvious imperative to ensure that any new enterprise technology is functional, scalable and reliable, an exclusive focus on technology integration issues, in the absence of development of a pedagogical vision, quickly neutralizes the likelihood that learning analytics data may catalyze organizational change with a focus on the student experience and learning out"