1. INTRODUCTION. Learner models [1] are at the core of adaptive learning systems, as they enable a system to adapt to individual learning needs. The accuracy of a learner model is the baseline for the usefulness of the adaptation decision. A variety of learner characteristics (knowledge, interest, learning style, etc.) may be represented in a learner model. Within this paper we concentrate on the learner’s knowledge state. A learner’s knowledge state is highly dynamic: it may increase (through learning) or decrease (through forgetting) from session to session. To make sure that the learning system can adapt to the knowledge of its users, continuous maintenance of the learner model is necessary. In our previous work, we have suggested Knowledge Indicating Events (KIE, related to evidence bearing events as described in [1]) as a means for non-invasively diagnosing user knowledge in an adaptive learning system [2]. Similar approaches to assessing user knowledge from different sources of evidence have been proposed with the Cumulate server of KnowledgeTree [3] and the Personis server [4]. Conceptually, the idea of KIE is in line with evidence-centered assessment design as suggested by [5]. In a nutshell, KIE are naturally occurring user actions (e.g., selecting a link, accessing a learning hint) that are interpreted as evidences for a user’s knowledge state. The main benefit of assessment based on KIE is that the diagnosis of user knowledge happens non-invasively, i.e. no additional interaction with the user is needed. This makes the KIE approach especially useful for systems in other fields than school or university settings such as workplace learning (e.g., [6]) where tests of knowledge and skill are typically not an option. While having obvious benefits, assessment based on KIE bears at least two serious drawbacks. first, the method makes a number of assumptions (e.g., “a person who clicks on the ‘help’-button for a concept has little knowledge of this concept”) that may not be accurate in all cases. The accuracy of the learner model, however, is a key issue in adaptive learning environments. Second, the algorithms for inferring user knowledge are based on aggregations of all KIE that occurred for each user. This leads to complex data and inferences that are not necessarily understandable for the users. Open learner models (for an extensive overview see [7]) have been proposed to improve understandability and accuracy of the learner model and, as a consequence, the adaptation. In an open learner model, the contents of the learner model are made visible to the learners. Some open learner models also allow the user to correct entries or to suggest additional information. Besides their benefits for improving accuracy of the content in the learner model, we regard an open learner model as a powerful means to enhance transparency and understandability in KIE based assessment approaches. There is broad agreement in the open learner model community that users should be offered an overview of the information in their learner model ([7], see also [8]). This is difficult for large and complex data sets, and most of the existing open learner models presented in [7] are not able to provide such an overview as the number of concepts increases. For a different context than learning, visualizations providing an overview of large models exists [8]. However, these visualizations are not applicable to our problem, as the structure of the underlying models is entirely different. In this paper, we present MyExperiences, an open learner model for evidence-centered assessment based on KIE. MyExperiences is implemented by using the design principles of information visualization which are condensed in Shneiderman’s well-known information visualization mantra ‘Overview first, zoom and filter, then details-on-demand’ [9]. The space-filling approach of MyExperiences is applicable to large and complex learner models. The aim of this paper is to present an innovative way of visualizing learner models that are based on the KIE approach. 2. MyExperiences: APOSDLE’ S OPEN LEARNER MODEL. MyExperiences is the open learner model of APOSDLE, an adaptive work-integrated learning system. The aim of APOSDLE is to improve knowledge worker productivity by supporting learning within everyday work tasks. Within APOSDLE, the learner model is used for ranking learning goals, recommending useful learning content, and for suggesting knowledgeable people (for details see [2]). As an integral part of APOSDLE, MyExperiences has been developed in an iterative three-year participatory requirements and design process. In this process, users from several knowledge-intensive work domains have been involved from the start through such methods as use cases and personas, scenario-centered design, formative and summative evaluations in the lab and the field. These design methods are reported elsewhere (e.g., [10, 11]). MyExperiences allows each user to access his or her own learner model in order to (i) understand recommendations and suggestions within APOSDLE, and (ii) improve the learner model’s accuracy by contributing information to it or altering information in it. 2.1 The APOSDLE Learner Model. The APOSDLE learner model is designed as a layered overlay of the APOSDLE domain model. APOSDLE can be instantiated to various domains by creating new semantic domain models. Typically, an APOSDLE domain model consists of approximately 100 to 150 domain concepts. For each concept, one of three knowledge levels is distinguished: learner, worker, and supporter. A learner in a topic is a user who uses the system with regard to this topic mainly for learning purposes, e.g. in order to receive in-depth information and hints. If a person is considered a worker in a topic, the person has worked on the topic without requesting further help. A supporter in a topic is a person who provides additional information for that topic, or who is contacted by another person to provide support for the topic. The users receive different recommendations for a topic (different types of resources etc.), depending on the detected knowledge level. The three knowledge levels are automatically diagnosed applying the KIE approach. To define KIE within APOSDLE, we first analyzed possible user actions with regard to how typical they are for one of these three levels. Then, we assigned the most typical events to each of the three knowledge levels. For instance, one learner event is “asking for a learning hint for a concept”. An example for a worker event is “performing a task which requires knowledge about a concept”. “Being contacted by someone else about a concept” constitutes an example for a supporter event. For maintaining the learner model, all KIE of a user within APOSDLE are logged. The inference to one of the knowledge levels is based on the proportion of KIE assigned to the different levels: The knowledge level of a user in a topic is that level for which the highest proportion of KIE has occurred. Clearly, different inference algorithms are conceivable. In the following, to make it easy to understand, we assume the simplest case where the inferred knowledge level of a user in a concept always is the level for which the most KIE have been observed. 2.2 MyExperiences: Overview, filter, Zoom and Details-on-demand. In order to fulfill the first aspect of Shneiderman’s mantra, ‘overview first’, the data structure of the data needs to be considered. In terms of data structures, the APOSDLE open learner model can be viewed as a forest of trees with the knowledge levels being the roots of the trees. Then, the children of each root node (knowledge level) are all concepts for which the algorithm inferred the corresponding knowledge level. The children of each concept again are the KIE that occurred for the respective concept. Different visualization techniques are conceivable for visualizing tree structures (e.g., node-link diagram, hyperbolic tree, tree map) [9]. Studies reveal that explorer-like visualizations (further referred to as tree view) are superior to all other tree visualization techniques for almost all tasks [12]. The drawback of tree views is that they do not allow a big picture overview if the data set is large and complex - which is the case for the APOSDLE learner model. In contrast, tree maps [13] enable a big picture overview of large hierarchical information structures and have shown to be quickly learnable by users unfamiliar with the visualization [14]. Therefore, we chose to combine the familiar tree view and the powerful tree map as coordinated multiple views (i.e., interactions in one view are instantly reflected in the other view). figure 1 shows an example of the resulting visualization for all three knowledge levels for a user who has been working with the APOSDLE system for several hours. The domain for which the system was created is innovation management. Typical tasks in this domain are writing an offer for an innovation project for a customer, or designing a creativity workshop. MyExperiences is divided into three rows, one for each knowledge level. Each knowledge level has a specific color. These colors are also used throughout the APOSDLE system whenever a visual item refers to that knowledge level. Each row in MyExperiences consists of a tree view (left) and a tree map view (right). Both views are coordinated, i.e. when a user clicks on a concept in one view, the concept is also selected in the other view. figure 1. MyExperiences: The tree map-based open learner model of APOSDLE. Each tree map gives an overview of all the concepts for the specific knowledge level. The number in brackets indicates the number of evidences that occurred for the specific concept. E.g., for the concept “Marketing” (in the topmost map) five KIE occurred so far. These events belong to two different types of KIE indicated by the concept rectangle separated into two parts. The color of such a sub-rectangle is derived from the number of events for this type of event and the total number of events for this concept. The lighter the color, the higher is the relative occurrence of this specific event for the specific concept. In other words, an equal color distribution means that all events occurred equally often, whereas one light color between darker ones indicates that one type of event occurred much more often than the others. The second aspect of Shneiderman’s mantra, ‘zoom and filter’, is realized with the search functionality on the top left. Search has shown to improve usability, if the user’s task is search rather than exploration [12]. With regard to Shneiderman’s third aspect ‘details on demand’, the user has several possibilities to interact with MyExperiences. Common interaction techniques like selection and zooming enable the user to investigate the open learner model, either as an overview or in detail. Zooming one step into the tree map allows the user to understand why the specific knowledge level was inferred for this concept. figure 2 shows this zoomed view for the concept “Kreativitätstechniken” (creativity techniques) of the bottom tree map of figure 1. In this zoomed view, the user is provided with the additional information concerning the frequency of events occurring for this specific concept. Remember that the inference of a knowledge level is a majority voting. Hence, the events that occur for one and the same concept may vote for different levels. This is the case, for instance, for the event “Select Learning Goal” in figure 2 (second from left), which is a learner event. We will address this issue in the discussion in Section 3. The user also can alter the knowledge level for any concept with right-click on the concept in either the tree view or tree map. Concepts with manually altered knowledge levels are then marked with an asterisk, like the concept “Kreativitätstechniken” (creativity techniques) in the bottom tree map in figure 1. This enables the user to differentiate automatically inferred (observation) and manually changed (explicit evidence) levels. Manual changes can be reset at any time either individually or all at once. figure 2: Zooming into a concept reveals the knowledge indicating events. 3. DISCUSSION AND FUTURE WORK. We have presented MyExperiences, an open learner model for evidence-centered assessment based on KIE. MyExperiences was implemented by using the design principles of information visualization and allows overview and detail for large and complex learner models. The search functionality facilitates direct access to specific areas of interest which may be useful, e.g., for reflection. Users can interact with MyExperiences to improve the learner model’s accuracy by correcting the inferred knowledge level of different concepts. Outcomes of informal interviews with APOSDLE users indicate that MyExperiences helps users to understand the underlying models and inferences, and supports reflection and meta-cognitive processes. Because of its unusual look-and-feel (MyExperiences is one of the first attempts to use tree maps for learner modeling), the users did not understand intuitively at the first glance what the purpose of the tool was. However, once the concept was explained to them, they did not seem to have difficulties to understand and use the interactive visualization. Still, these findings need to be evaluated systematically in usability studies, with control groups where MyExperiences is compared to other visualizations of KIE based learner modeling. Strictly speaking, the current version of MyExperiences visualizes the approach of KIE in a simplified manner: As described above, KIE voting for different knowledge levels might have occurred for one and the same concept. However, this is not represented in the second level of the tree map where all KIE for one concept have the same color (figure 2). The main reason for this design decision was that the visualization should be usable for all different kinds of algorithms underlying the inference of the knowledge level. While in the case of a simple majority voting, the visualization of the second level in the tree map might be straightforward, this would not be the case, e.g., if the inference algorithm uses a weighting scheme. Concerning the visualization, we see three main avenues for follow-up. first, we plan to provide users with visual clues about uncertainty in automatically inferred knowledge levels. Second, it might be helpful for the users to see the development of their knowledge levels over time (history). Third, it may be interesting to take into account the representation for the rest of the users such as the gathered history of transaction data, or group average. In this line of research, an approach for visualizing user models of other users with tree maps (not based on KIE) has been recently presented by Peter Brusilovsky’s group [15]. While being aware of the fact that in-depth usability studies of MyExperiences are still missing, we believe that using tree maps for representing information about users combines research into open learner models and information visualization is an innovative and promising way. ACKNOWLEDGEMENTS. The Know-Center is funded within the Austrian COMET Program - Competence Centers for Excellent Technologies under the auspices of the Austrian Ministry of Transport, Innovation and Technology, the Austrian Ministry of Economics and Labor and by the State of Styria. COMET is managed by the Austrian Research Promotion Agency ffG. APOSDLE (www.aposdle.org) has been partially funded under grant 027023 in the IST work programme of the European Community.