"1. Figure 1. An example problem, “Making Cans”, from Carnegie Learning’s Cognitive Tutor 2005. In the “Making Cans” example shown in Figure 1, a student completes a problem where she is asked to find the area of a piece of scrap metal left over after removing a circular area (the end of a can) from a metal square. The student enters everything in the worksheet except for the row labels, and column and ‘Unit’ labels for the first three columns. In addition to providing students with feedback and instruction, the tutor records the student’s actions and tutor responses, and stores them in a log file, which is imported and analyzed by DataShop. The DataShop logging format differs from existing standard formats in that it attempts to capture student-tutor interaction history at a fine-grained level, while also providing context for such interactions; the format does not attempt to describe, a priori, learning resources and how they’re transferred (e.g., LOM, SCORM) or test content (e.g., IMS- QTI). In this way, the format is essentially descriptive, not prescriptive. The DataShop logging model is represented by the following constructs [16]: • Context message: the student, problem, and session with the tutor • Tool message: represents an action in the tool performed by a student or tutor • Tutor message: represents a tutor’s response to a student action Below we see example context, tool, and tutor messages in the DataShop XML format. In this example, the student attempted problem “MAKING-CANS” in the “PACT- AREA” lesson of the Geometry tutor. Looking at the tool and tutor message pair, we see the student correctly entered “200.96” as the answer. Tool and tutor messages are traditionally paired with each other (by context message), allowing DataShop to interpret the student action and the tutor’s response in conjunction. These pairs are then stored as a single tutor transaction in the database. Table 1 below illustrates how actions from the Making Cans example are interpreted and stored as tutor transactions. A tutor transaction stores details such as the student(s), session, time, problem, problem subgoal (step), attempt number, student input, tutor response, number of hints, conditions assigned to the problem step, as well as knowledge components (skills). Unpaired tool messages can be used to represent untutored actions, such as a student starting audio playback, and are stored in the repository as well. Table 1. A simplified tutor transaction excerpt from the “Making Cans” example. Multiple tool and tutor messages are typically logged for a single problem-solving activity. Problem-solving activity is broken down into “steps” which represent completion of possible subgoals or pieces of a problem solution. Students often make multiple attempts at a step or get instructional help on a step and each of these attempts or help requests are stored as a separate tutor transaction in the database. In the “Making Cans” example, we see the student attempted the “(SCRAP-METAL- AREA Q1)” step three times (transaction numbers 2, 3 and 6 in Table 1). We can ascertain from the transactions that the student was unsuccessful in her first two attempts, providing an answer of “32” and “4”, both labeled as incorrect by the tutor. On the third attempt, the student successfully completed the problem step, providing an input of 13.76 (as can be seen in Figure 1). To allow for fast and easy visualization and analysis of data, tutor transactions are aggregated into a student-step rollup table. This “denormalized” table aggregates steps by student, problem, and step and is used by many of the DataShop tools, such as the Performance Profiler and Learning Curve. An example of how the “Making Cans” tutor transactions are aggregated by student-step is depicted in Table 2. The student-step roll-up table stores an array of useful step information, some of which is depicted above. In the “(SCRAP-METAL-AREA Q1)” step in the “Making Cans” example, we can see that the three tutor transactions are now rolled into a single step entry (line number 9 in Table 2). “Opportunity count” is two since the student saw the “Compose-Areas” knowledge component twice. “Total incorrects” is two, since she made two incorrect attempts. “Assistance score”, which consists of total hints added to total incorrects, is two. “Error rate”, whether the first attempt was an incorrect action or hint request, is 1 (or 100%) because the student’s first attempt was an error. Table 2. Data from the “Making Cans” example, aggregated by student-step. Each step in a problem requires the student to know something—a relevant concept or skill—to perform the step correctly. This small unit of knowledge is termed a “knowledge component”, a key notion in the PSLC’s theoretical framework (learnlab.org/research/wiki). To document this required concept or skill, a tutor author labels steps with a hypothesized knowledge component(s) required for correct completion of the step. In the “Making Cans” example, we see the knowledge component “Compose-Areas” assigned to the correct transaction (row 6 of Table 1) for the “(SCRAP-METAL-AREA Q1)” step. A knowledge component codes for a general student capability to accomplish steps in tasks. Knowledge component modeling, the process of assigning knowledge components to steps, bolsters the usefulness of intelligent tutor data by providing additional context for a step. A step can have zero, one or multiple knowledge components associated with it. Steps that have assigned knowledge components are used to produce a variety of “learning curves”. Figure 2 below depicts error rate learning curves generated by DataShop. In this graph, error rate, or the percentage of students that asked for a hint or made an incorrect attempt on their first attempt on steps associated with the knowledge component, is shown on the y-axis. The x-axis (“Opportunity”) indicates the nth time (e.g., 4 is the 4th time) a student has (in theory) used (or tutored on) a knowledge component to solve a step in a problem. Each unique step in a problem is distinct from other problem-solving steps. For example, in Table 2, “(POG-AREA Q1)” and “(POG- AREA Q2)” are unique steps—they correspond to different questions in problem. Because they both exercise the “Circle-Area” knowledge component, they are counted as distinct opportunities for the student to demonstrate whether he or she has learned “Circle-Area” (and if not, get instruction on it). Figure 2. Error Rate Learning Curve with predicted values from a Geometry Area dataset. The solid curve represents the actual values, each point an average across all students and knowledge components for the given opportunity. The dashed curve represents the predicted curve values, based on the Learning Factor Analysis (LFA) algorithm [5]. Built-in statistical models measure student proficiency, knowledge component difficulty, and knowledge component learning rates. This algorithm allows for search and comparison of alternative knowledge component or cognitive models. DataShop supports users in entering multiple hypothesizes about how student knowledge and learning may be decomposed. 2.2 Importing and Exporting Learning Data. Data may be imported into the DataShop repository through XML or a tab-delimited text file format. Logging to DataShop XML provides the richest and most complete data. If logging via XML, tutors can send messages directly to the DataShop logging server in real time. Logs are automatically processed on a nightly basis, making them available for analysis or export through the web application. Alternatively, a computer tutor can write XML to files on the local hard disk (for example, if the tutor is running off-line) and then send the data to the logging server at a later time. Data in a pre-existing log format can also be converted to DataShop XML and then imported into the repository. This procedure has worked well for data collected by other tutoring systems including Andes (www.andes.pitt.edu), math Cognitive Tutors (carnegielearning.com), REAP (reap.cs.cmu.edu), and Assistments (assistment.org). The tab-delimited format of a transaction table can alternatively be used to import from a preexisting source. DataShop offers various data export options through the web application each delivered in a tab-delimited text file. These include transaction and student-step level exports (as illustrated in Tables 1 & 2), and a student-problem aggregate export. 2.3 Analysis & Visualization Tools. The DataShop web application provides several tools to assist with analyzing and visualizing repository data. These tools can be used in conjunction to jump-start data analysis: one can determine if students are learning by viewing learning curves, then drill down on individual problems, knowledge components, and students to analyze performance measures. The following DataShop tools are available: • Dataset Info: provides dataset metrics, contextual information, quick statistics (number of students, transactions, knowledge components, etc.) as well as papers, files, a problem table, and exporting and importing knowledge component models. • Performance Profiler: multi-purpose tool that visualizes student performance across various dataset-specific domains (problem, step, curriculum level, knowledge component and student) and measures of performance (error rate, assistance score, average number of incorrects, average number of hints, and residual error rate). • Error Report: presents each student’s first attempt at a problem or knowledge component, including if he or she was correct, the number of students or observations, and the text of the student’s answer. • Learning Curve: visualizes student learning changes over time. Learning curve types include error rate, assistance score, correct step time, and others. The Learning Factors Analysis model [6] provides predicted values for error rate learning curves. 3 Uses of PSLC’s DataShop. As indicated above, many recent analyses of data from DataShop have been performed in a variety of domains. A number of other studies have used, tested or extended the analysis techniques employed in DataShop including investigations in reading [10], Physics [12], and Geometry [15]. Often analyses have been targeted at finding ways to improve student learning. In some cases, the work has been taken full circle such that an analysis led to an instructional redesign that was demonstrated to improve student learning beyond that realized by the original instruction. We provide a couple examples. Cen, Junker, and Koedinger [6] performed a learning curve analysis using the Learning Factors Analysis (LFA) algorithm based on data from the Area unit of the Geometry Cognitive Tutor. They noticed that while students were required to over-practice some easy target knowledge components or skills (see square-area in Figure 2), they under- practiced some harder skills (see trapezoid-area in Figure 2). Based on observation and further analysis, they created a new version of the geometry tutor by resetting parameters that determine how often skills are practiced. They ran a classroom experiment where students in a course were pre- and post-tested and randomly assigned to use either the previous or the new tutor version. Students using the new version took 20% less time to finish the same curriculum units (because over-practice was eliminated) and learned just as much as measured by normal, transfer, and long-term retention tests. A second demonstration of a datamining project that “closed the loop” is work by Baker et al. [3], who had done formal observations of student behavior in computer labs while working through lessons of a middle school math Cognitive Tutor. Among a number of categories of off-task or otherwise disengaged behavior, he found that “gaming the system” had the largest correlation with poor learning outcomes. Gaming refers to student behavior that appears to avoid thinking and learning through systematic guessing or fast and repeated requests for increasing help. Baker used machine learning techniques to build a “detector” capable of processing student log information, in real time, to determine when students were gaming. The detector became the basis for an intervention system, a “meta tutor”, designed to discourage gaming and engage students in supplementary instruction on topics they had gamed. A controlled experiment demonstrated student-learning benefits associated with this adaptive selection of supplementary instruction for students observed to be gaming. 4 Discovering better cognitive and affective models of student learning. An important general use of this kind of data is to drive the development of more precise computational models of human cognition, motivation, and learning. The work on gaming is just one example of using interaction log data to assess and remediate student motivation and other detectors of student engagement are possible. With respect to cognitive and learning assessment, we have been pursuing a strong hypothesis that the correct representation of knowledge (facts, skills, concepts, strategies, integrative models or schemas, meta-knowledge, etc.) in a domain is an empirical question. Why is this a strong claim? First, many do not believe knowledge is decomposable, for instance, Barab and Squire say “learning, cognition, knowing, and context are irreducibly co-constituted and cannot be treated as isolated entities or processes” [4]. Second, most designers of instruction, like textbook authors, assume that the instructional designers determine the knowledge decomposition, the target concepts and skills, and students learn these knowledge components as they are taught. In contrast, the PSLC takes the position that the nature and grain size of mental knowledge representation is driven as much or more by the student and the explicit and implicit learning mechanisms in which they and their brains engage. Others, too, have emphasized that students do not learn knowledge in pieces as they are taught [7]. If the nature of human knowledge representation is an empirical question, then we need both vast and detailed educational data and associated data processing algorithms to answer this question for all domains of academic learning. In addition to the LFA technique mentioned above, which employs the “smooth learning curve” criteria [13], a number of algorithms have been created for empirically evaluating knowledge representations against student performance data including the rule space [18] and knowledge space [8] approaches. The collection and use of on-line learning data will further drive such developments. As noted in a major national report “psychometric validation of [on-line] assessments is needed so they can be compared with conventional assessments, and complement and ultimately supplant them” [1]. 5 Conclusion. We described PSLC’s DataShop, an open repository and web-based tool suite for storing and analyzing click-stream data, fine-grained longitudinal data generated by online courses, assessments, intelligent tutoring systems, virtual labs, simulations, and other forms of educational technology. In contrast to other types of educational data such as video and school-level data, data in DataShop includes a rich set of semantic codes that facilitate automated analysis and meaningful interpretation. The PSLC DataShop uniform data format is an initial attempt to develop a common standard that we hope will be useful to field if not as is, then in driving better or more useful common standards. In addition to being a source for learning data, it is also a place where researchers can deposit data and then get help from other researchers who can perform secondary analysis on this data. DataShop allows free access to a wide variety of data sets and analysis tools. These tools help researchers visualize student performance, difficulties, and learning over time. Such analyses can lead to demonstrably better instructional designs. The data can also drive improved models of student cognition, affect, and learning that can be used to improve on-line assessment and on-line learning. We take as a premise that the human brain constructs knowledge based a variety of input sources (e.g., verbal, visual, physical) and in a fashion and at a grain size that may or may not conform to the structure as conceived by an instructor or domain expert. In other words, the nature and content of human knowledge representation is a deep and important scientific question, like for instance, the nature of the human genome. To answer this question requires a vast collection of relevant data, associated analysis methods and new theory. Acknowledgement. Research supported by the National Science Foundation award number SBE-0354420."