"1 Introduction. The field of Learning Analytics is emerging as a combination of business intelligence, business logic, educational data mining and action analytics [8] where data is collected, analyzed and interpreted to derive so called actuators to optimize a learning experience. Much the same way in which a regular web site monitors the operations performed by the users to then infer patterns to suggest or modify the experience of future users, in the context of learning, students usually interact with a Learning Management System (LMS) that records all the operations. This wealth of data can also be analyzed and transformed into useful information to ultimately infer and apply changes to the current environment aiming at improving the process for the student, the instructor and/or the institution. There are numerous key developments that are behind the emergence of this new field. LMSs, both commercial and open source, include modules that automatically register every event taking place in the platform. The higher the percentage of course activity that takes place in the LMS, the more detailed information is stored. For example, if a learning experience contains support for on-line quizzes, formative assessment, a chat room and its own email service, the platform may easily keep record of who did what, when and with whom. When combined with the well established techniques in the area of web analytics, having a detailed account of the interaction between students and instructors, or among students becomes a reality. Once the data is obtained from the tools used in a learning experience there are multiple objectives that can be tackled. For example, The Signals project a Purdue University [4] is an example of how learning analytics as described in [10] are applied at an institutional level to create an early detection system for student failure. The system uses the data already collected by the institutional LMS to detect in real time the type of events that, based on previous information, have a high probability of leading to student failure. Detecting these patterns translates then into a set of measures that are taking to anticipate the problem and thus reduce student failure rates. Interaction has been shown to be an important factor influencing student success [3]. The amount of interaction a student has with peers has a positive correlation with the academic performance [12]. As a consequence, having a detailed account of the interactions that occur in a learning experience will likely offer a good predictor of academic performance, which itself is one of the most important aspects of an educational institution. With these new tools, a learning community can be “seen” in ways that were never considered before [7]. But in this scenario, there are several challenges that need to be overcome. Campbell and Oblinger [5] characterized the process of learning analytics as an engine with five stages: capture, report, predict, act, and refine. The first step already faces the challenge of having an adequate observation capability. A detailed account of any event that takes place in a learning scenario is the first requirement to have a solid foundation upon which to build the reporting, predicting and acting mechanisms. After the data has been obtained, it should be reported in meaningful forms to all stakeholders. Several visualization techniques have been applied specifically to data gathered in courses (see for example [14, 15]). The next challenge is to determine which factors are truly significant to achieve accurate predictions. In [13] a detailed analysis is performed considering initially a set of 22 variables recorded by the LMS (BlackBoard Vistat m (www.blackboard.com) . Out of these 22 factors, only 13 were found to have a positive significant correlation with student final grade. A final multi-variable linear model is proposed with only three of these factors accounting for 33% of the variability. These factors are: total number of discussion messages posted, total number of mail messages sent, and total number of assessments completed. Once a model has been created, the following steps face the challenges of inferring relevant interventions, and finally, to design a feedback process to refine the overall mechanism. There are already several approaches that close this cycle. In [11] a system is presented in which all the interactions of the students with the course material and among each other are recorded and made available to the instructors when modifying the course content. The system uses Semantic Web techniques to translate LMS logs into resource annotations that are then inserted into the editing tool used by the instructors to create content. 1.1 The challenge of recording the interaction. But recording the interactions that take place in a learning environment is becoming more difficult. The initial model used during the early stages of LMS deployment in educational institutions could be called “LMS-centric”. There were numerous analogies between LMSs and conventional knowledge management tools. But with the advent of the Web 2.0, the “LMS-centric” model has failed [6]. Although the latest LMSs offer an increasing set of features, students are beginning to reach the educational institutions with solid experience on how to interact with their peers in ways that are not covered by the LMS. The main consequence is that a significant part of the interaction during a learning experience is beginning to take part outside of the LMS in what it could be called a “de-centralized approach”. Even the LMSs themselves have contributed to this de-centralization. For example, most LMSs offer the option of receiving email notifications when new messages are posted in forums. The chances of students using an email client outside the LMS are increasingly large. Email support is another example. LMSs offer internal an email account to each user, but they are no competition in terms of features other platforms available on the Net. This tendency is more exacerbated when a learning experience contains a significant amount of activities that cannot be embedded by any means in an LMS. In experimental sciences, students typically require the use of special resources for procedural activities. The extreme case of this tendency is in ICT education where most of these special resources are applications that can be installed in the student personal computer. Furthermore, some studies are beginning to confirm that students use conventional ICT tools to access an increasing number of resources outside the institution LMS [21]. The main consequence of this tendency is that in order to maintain the effectiveness of learning analytics, new techniques are required to extend beyond the LMS-centric approach and adapt to the Web 2.0 style. 1.2 Observation to support assessment. Another factor that is changing the educational landscape is the transition from a purely expository instructional method to a “learner-centered” approach [17] where the tutor adopts a more supportive role, and the learner explores, participates and is more active during the learning process. This change of philosophy is having numerous ramifications within the academic world. Entire degree programs are re-organized in order to accommodate the new role of the student. Teaching staff needs to adapt their pedagogical techniques to a, sometimes, totally new approach. Together with these changes, numerous accreditation institutions have emerged with the objective of assuring that educational institutions embrace quality assurance and sustained innovation techniques. For example, ABET (Accreditation Board for Engineering and Technology) is an institution that provides accreditation for degrees in the area of applied science, computing and engineering education. The focus of the accreditation process is on “what is learned rather than what is taught” (www.abet.org). The approach described in this document is being deployed in an engineering degree that currently pursuing accreditation by ABET. The institution describes what students are expected to know and be able to do by the time they graduate. Again, in the specific case of engineering education, some of these outcomes have a strong procedural nature. For example: “(k) an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice.” [1]. In order to assure that on graduations students are capable of using modern engineering tools, they need to practice with them through activities. This is an example of the type of new outcomes that are being requested from applied science degrees that are difficult to accommodate by conventional LMS. At most, LMSs may cover this aspect of the learning process by supporting on-line quizzes, but as an indirect measuring tool. A second example of the limitations of LMSs is highlighted by another program outcome: “(d) an ability to function on multi-disciplinary teams”. Teamwork requires a high degree of student-student interaction. There are studies that rely on interaction through forums hosted in an LMS to gain insight on the level of collaboration within teams [2]. But if students are already used to communicate using a variety of Web 2.0 type of tools, it is highly unlikely that when immersed in a collaborative setting, they would use an LMS for these tasks. From the previous observations, there are several questions that lay ahead in the area of learning analytics: – How much do they rely on interaction taking place in the LMS? – How can they cope with new forms of interaction? – How are they affected when analyzing interaction in collaborative environments? This document describes the approach to obtain learning analytics in a concrete scenario of collaborative activities within a course of an engineering degree. Although still in the preliminary stages, we believe there are several observations that can help shed some light on the previous questions. 2 Approach. The approach was deployed in the face-to-face course “Systems Architecture”, which is part of the degree in Telecommunication Engineering (www.it.uc3m.es/labas/syllabus_en.html). The total number of students that initially signed for the course was 248 and were divided into five sections groups. The course contained the following learning outcomes: 1. Design and development of applications in the C Programming Language. 2. Use proficiently the tools for application development. 3. Apply team working techniques to develop an application for a mobile device. 4. Use of self-learning techniques. Outcomes 3 and 4 refer to generic methodological aspects. Team work was used during the second half of the course (six weeks) in which groups of four students were created by the instructors to work in a project. Several documents about team dynamics were requested as readings and a class session was devoted to discuss teamwork, agree on a team contract and discuss the different type of conflicts that may arise. The measures to achieve outcome 4 were applied throughout the entire course. Each session had two sets of activities, previous and in-class. The set of previous activities required an objective that would be reviewed in the following class. Students found this methodology significantly different to those used in other courses. The course followed a continuous evaluation scheme. Five partial examinations spread along the semester were combined with small exercise submissions. The goal was to engage students to regularly work in the course. The final course grade was simply the sum of all these partial scores; no final exam was given. 2.1 Providing a fully configured development environment. The main complication from the point of view of analyzing the interaction derived from outcomes 2 and 3. In order assure that students use proficiently the tools for application development, they required a development environment fully configured and, most importantly, with high availability (to promote offclass work and do not overload computer rooms). This type of environment was clearly beyond the reach of the institutional LMS, and therefore, the possibility of observing the interaction with these tools was initially non-existent. The adopted solution was based on the use of a virtual machine. Lately, virtualization has been considered in education in order to easily facilitate students fully configured machines that can execute with barely any configuration steps in their personal computers [9]. The use of this this approach had several benefits. First, all students had initially the same exact set of tools properly configured which greatly simplified the design of activities to use them. Second, the machine was configured so that students could access the files stored in their regular personal computers. Third, the virtual machine (although including a fully configured operating system) was portrayed to the students as the application to use when working on the course material. And last, but most importantly, the machine included a system to record the events occurring with respect to the installed tools. More precisely, the monitoring mechanism was capable of recording the following events: – Power-up and shutdown of the machine. – Invocation of a previously selected set of tools. – Internal commands used by the students in some of the development tools – Historic data about the sites visited with the included browser With this mechanism, a wide variety of interaction events that otherwise would be ignored, were recorded and stored in the virtual machine. 2.2 Support for a shared folder. The use of the virtual machine was combined with support for a web-based folder shared among the team members in which students could store any files they needed related to the course. More precisely, a first folder was created for each pair of students during the first half of the course, and a second shared folder was created for the teams formed in the second half of the course. Instructors had access to the shared folders of all the teams under their supervision. This addition turned to be a powerful communication channel not only among students, but also between students and instructors to solve problems, check errors, and generic consultations. During the configuration phase of the virtual machine described in 2.1, the shared folder was configured as the repository where all the recorded events were stored. A non-intrusive procedure would be in charge of sending the recorded events whenever the students submitted a new version of the files in the webbased folder. In order to comply with the current legislation, the virtual machine was downloaded only by those students that agreed with the terms of use described in a document. Furthermore, the machine was configured to boot up with the browser open and showing a page explaining the recording mechanism, the steps to disable it, and the contact person to exercise the rights over the collected data (delete, query and modify). Figure 1 shows a screen capture of the initial desktop of the virtual machine. 2.3 Support for actuators. With the configuration described in the previous session, the virtual machine can be thought of as the application that students need to use when working on the course material. But being outside of the LMS not only poses a challenge to record and collect data, but also to the step of acting. Although the experience is only at the first stages, and as such, only the data collection aspect has been deployed, a solution has been considered and configured to be able to act on this environment. A widget displaying the content of a pre-defined folder has been installed in the desktop of the machine. Initially, the folder contains the terms of use for the machine (that the student agreed to). The widget is shown in the upper left corner of the desktop illustrated in Figure 1. Using a technique similar to the one to send the recorded events, a new set of files can be uploaded to this special folder such that their corresponding icons appear in the desktop widget. Fig. 1. Initial screen of the virtual machine. With the described configuration, students would see how the folder shown in their desktop keeps changing its content. The type of resources that can be added range from files (documents, audio, video) to URLs to access remote resources. 2.4 Encoding the events. The events recorded in the student machines offer a very detailed account of the procedures followed as well as the tools that were invoked. The captured information has been encoded using the CAM (Contextualized Attention Metadata) format [19, 22, 18]. CAM provides a data model for representing user activity together with contextual information. Educational application of the CAM framework are discussed in [20], where the tool CAMera for monitoring and reporting on learning behavior is described. CAMera collects usage metadata from diverse various applications, represent these metadata with CAM and reports them to the learner. More complex applications in the scope of adaptation and web-semantics have also been built based on this format [16, 16]. 3 Initial Results. The initial objective of this approach is to explore ways to extend the datagathering phase of learning analytics beyond the LMS and into environments in which a significant amount of interaction is taking place. The initial conditions were also to deploy the data gathering in a scalable and non-intrusive way. The virtual machine was made available at the beginning of the course. Out of the 248 students that signed out for the course, a total of 220 downloaded the machine (88.71%). Out of the remaining 28 students (11.29%), most of them opted to use their own configured environment. The large percentage of students that decided to use the machine shows its acceptance as the course tool. The number of downloads, though turned out to be not a good estimation of the true activity carried out by the students in those machines where the recording mechanism was not disabled. The events received in the first half of the course (in which students worked in all the activities in pairs) were 48, 342 for a total of 115 students (an average of 420 events per student). 3.1 Activity outside the LMS. An important side-effect of placing the data-gathering phase outside of the LMS and into a fully configured environment was to be able to measure the percentage of URLs that were related to the LMS. In other words, by exploring the events encoding a visit to a URL we can have a first look at the percentage of traffic that goes to other sites. Out of the almost 49, 000 events, 15, 507 (32.07%) were events in which a URL was opened with the browser. When counting the number of unique URLs, this number falls down to 8, 669. Out of these, only 2, 471 (28.51%) pointed to the LMS. An initial interpretation (pending a more thorough analysis) seems to suggest that students interact with a large number of resources that are outside of the LMS. 4 Discussion and Future Work. In this paper a context has been described in which in order to assess the degree of interaction that students are having with a previously detected set of tools and among themselves, the LMS offers a very poor coverage. The context is derived from the adoption of learning outcomes that require procedural activities with tools and functionality beyond the scope of a conventional LMS¿ The described approach proposes extending the scope of the data-gathering techniques to include a fully configured virtual machine containing all the required tools as well as a mechanism to record a subset of the most representative events. A detailed description of the terms of use of the machine with instructions on how to disable the recording mechanism, as well as how to check, modify or delete the information, was included with the machine. The machine is also configured to establish a bidirectional communication channel with a central server to send the recorded events and receive new resources that are shown in a desktop widget as actuators on the learning environment. The received data has been encoded using the CAM format and is being prepared to perform a more sophisticated algorithm to detect special patterns to detect early which students are not using properly the given tools. The work is still in its preliminary stage in the sense that only the datagathering stage has been successfully deployed. Still, the approach has been shown to be scalable (more than 200 students) and in-obtrusive (students do not sense that the events are being recorded). The obvious line for future work is to identify those variables of the recorded events are more suitable to make predictions of those students that are not using properly the tools included in the machine. A second line of work has also been conceived to combine the recorded events and the information extracted from the LMS to detect potential anomalies in the collaborative part of the course. The challenges in this context are bigger because teams are suppose to meet regularly on face-to-face meetings in which there is no type of event recording. Acknowledgment. Work partially funded by the Learn3 project, “Plan Nacional de I+D+I TIN200805163/TSI”, the Best Practice Network ICOPER (Grant No. ECP-2007-EDU417007), the Accion Integrada Ref. DE2009-0051, and the “Emadrid: Investigacion y desarrollo de tecnologias para el e-learning en la Comunidad de Madrid” project (S2009/TIC-1650)."