"1. INTRODUCTION. Commons-Based Peer Production (CBPP) is the process by which internet communities create and maintain digital public goods [2]. Generally, these systems work in what would appear to be a very chaotic manner. There is no fixed or coordinated schedule for contributors to a peer production project and users tend to contribute when and how they want. At the same time, there is no fixed management structure, and users are generally promoted in the community by virtue of their contribution. Often there is an area of the community’s space reserved for users to participate in discussions that relate to the community’s operation, such as the Village Pump in Wikipedia1 or Meta in the Stack Overflow2. This is, in essence, how the community learns as an organisation – users put forward their individual experiences that are then discussed with the boarder community. This can result in the implementation of new community policy as the community begins to evolve and adapt to how it once currently existed. While analytics are increasingly being explored as reflective mechanisms for leaning in a variety of contexts, there has been little research conducted on the use of analytic interfaces to support the organisational aspects of online communities. We believe that visual analytics can be used to support organisational learning in peer-based communities by not only providing a space for reflection but also promoting discussion and debate in the broader community. To asses this claim, we have developed a novel visual analytics tool, called Explore.su, which we have embedded in a collaborative architecture. This work builds on previous research into both social visualisation, which provoked reflection and story-telling in groups and communities [7], and social data analysis, which opened collaborative data analysis to a non-expert community [14]. We divide this paper into two sections. In the first section we consider CBPP systems in terms of socio-technical learning systems, which supports three interrelated levels of learning: individual, group and organisational. We argue that each level of learning contributes to the system as a whole and then concentrate on how organisational learning is currently addressed in online communities. In concluding this section, we emphasise how the process of reflection is considered essential to organisational learning and argue that visualisation and the application of visual analytics can be used to support organisational learning in peer- based systems. In the second section, we introduce Explore.su, a collaborative visual analytics environment, which we developed to explore the process of reflection in a large Q&A community. We describe the design, implementation and evaluation of Explore.su and conclude with a discussion on reflection and the implementation of visual analytics in online communities. 2. LEARNING & PEER PRODUCTION. Commons-Based Peer Production (CBPP) is an approach to the creation and maintenance of digital public goods3 that harnesses the capability of communities organised, principally, without traditional management hierarchies. Communities engaged in CBPP have complex socio-technical infrastructures, which are predicated on cooperation, yet are subject to continuous change. We conceive of CBPP communities as socio-technical learning systems in which learning is undertaken at a number of interrelated levels: individual, group and organisational. 2.1 Individual Learning. Individual learning is one of the key motivating factors to the success of Open Source Software (OSS) [25]. Developers join OSS communities to learn, hone and improve their skills. In these communities, there is a much stronger emphasis placed on the social recognition of human-capital and on the autonomy of the individual. OSS developers are free to choose projects they consider cool, in vogue, or they feel could benefit from their contribution. The result is a more concerted effort on behalf of the individual both as a participant in the project and as an individual learner. The sort of learning can be generalised to other peer- production systems. In technical Q&A communities, for example, users can learn a great deal about a subject domain in a more interactive and participative manner. 2.2 Group Learning. When individuals come together the dynamics of learning is transformed as those individuals begin to work towards a shared consensus [3]. Peer production provides ample opportunity to work in groups, however, the nature of the collaboration will impact upon the approach to learning. Haythornthwaite, for instance, considers peer production as either lightweight or heavyweight, depending on the degree of freedom or ambiguity that exists around an individual’s role in the system [11]. Both lightweight and heavyweight models of peer production provide opportunities for learning; however, this occurs in the form of a learning collective or community of practice. 2.2.1 Learning Collectives. Thomas and Brown [6] argue that in communities, individuals learn to belong while in collectives individual learn to participate. While subtle in some regards, this shift recognises the importance of participation without over-emphasising the nature or indeed level of participation. Thus, learning can involve much less intensive forms of action such as rating or commenting on web content. 2.2.2 Communities of Practice. One of the most widely recognised (and highly cited) models of group learning is the community of practice model originally proposed by Wenger [26]. Communities of practice are founded on the recognition that learning is a continuous process and should not be separated out from the everyday activity of the learner. Rather, effective learning occurs when individuals are engaged in meaningful practices as part of the communities they value. Communities of practice require more intensive and often situated collaborations, than those found in learning collectives, and can often develop in many peer-based communities. 2.3 Organisational Learning. There is a broad academic and practitioner literature on organisational learning and a variety of perspectives have emerged in regards to the use of the term. In this section, we will briefly and rather broadly, address some of the core tents of organisational learning and then focus on how online communities, and in particular peer production communities, typically learn to adapt and change. Schon and Argyris published one of the first theoretical accounts of “organisational learning”, in which they described organisational learning as a method of “detecting and correcting errors” at an organisational level [1]. Their account introduced the notion of single loop and double loop learning and placed a particular emphasis on the role that reflection plays in effective organisational learning. Other authors, such as Kim, reinforced this view suggesting that organisational learning “increases an organisation’s capacity to take effective action” [18]. Levitt and March looked at the practice of organisational learning. They argued that experiential learning is prone to human interpretation and thus there is a need to address organisational learning with more objective analytical tools (such as statistical analysis for example) [19]. Crossan et al.’s 4I framework conceives of organisational learning as a process of dynamic renewal that occurs across three levels of an organisation: individual, group and organisational [5]. Their framework articulates how experience, intuited and interpreted at the individual level, is shared and integrated at the group level and then institutionalised, as new policies and procedures, at the organisational level. Organisational learning has been also addressed in the literature on online communities, however, rarely is the actual term organisational learning used. The majority of this research has been concerned with how online communities manage to adapt and change, given that power is distributed amongst the community members and that the technical and social elements of the community are tightly integrated. Butler et al., for example, looked at the policy mechanism in Wikipedia. They found that the Wikipedia had grown substantially - from a relatively open system to a much more sophisticated bureaucracy [4]. Jahnke found that a student online community evolved in a similar fashion from an undefined and amorphous state to a defined and rigid social structure [17]. Forte and Bruckman found that smaller autonomous projects had emerged within Wikipedia, each with their own set of operating norms and policy rules yet the basic norms of behaviour were inherited from the original site [8]. Schindler and Vrandecic describe how the introduction of new technical features in the German Wikipedia was the result of increasing pressure by the community [21]. Our aim is to investigate ways to support organisational learning in online communities so that the policy mechanisms move beyond an experiential based model, which is prone to the problems of human interpretation, to one that is based on the application of more objective analytic interfaces. As a first step in this direction, we have developed a novel visual analytics tool that provides a community with an area for reflection in their social space. We believe that visual analytics cannot only be used to passively inform organisational learning in online communities but also provides a means for discussion and debate as regards to the ongoing evolution of community processes. 3. EXPLORE.SU. In the rest of this paper, we present the work done on Explore.su – a visual analytics tool that was developed to explore reflection in the Super User4 online community. 3.1 Related Work. Before continuing, we will briefly review the related work on social network visualisation and social data analysis. There is a wide range of software that supports social network visualisation. UCINET, Pajek and Gephi, for example, provide both statistical analysis and graph visualisation. While popular, many of these packages are developed for professional analysts and require substantial knowledge and expert interpretation during use. Researchers have, as a consequence, sought to develop social network visualisation tools that support the less- expert end-user. Figure 1 shows a node-link (graph-based) visualisation from Vizster (Heer and boyd 2005). Vizster, by Heer and boyd, is an example of a node-link visualisation that enables end-users to explore their own social network [13]. While provoking cycles of analysis, reflection and story-telling, Vizster (as illustrated in Figure 1) was designed for ego-centric networks, which are focused around a single user, and the approach will not scale elegantly with larger networks, such as the Super User online community. Indeed, this is a broader reflection of node-link (graph-based) visualisations in general [9]. Networks of over a 150 nodes are generally incomprehensible when visualised using node-link diagrams. While algorithms such as spring or force-directed layout help to organise the clutter of nodes and links into more meaningful representations, when conducted on large networks, the execution time is prohibitive. Perer and Shneiderman argue that many professional network analysts render the visualisation for publication only after having conducted the analysis [20]. When wishing to visualise change in a network, this problem is significantly exacerbated, as node-link layout algorithms remain sluggish or inelegant when dealing with the evolution of a network over time. These difficulties have led researchers to investigate the use adjacency matrices as an alternative approach to node-link diagrams. Henry developed Matrix Explorer, a social network analysis tool that visualises large-scale social networks at both global and local levels [16]. Matrices are used for global representations while the more familiar node-link visualisations are used to show local relationships. The visualisation aims to shift towards the end-user (historian or sociologist), moving away from a complex interface to a more user friendly visualisation environment. To handle scale, Henry implements block modelling, a common clustering technique used in network analysis, to order and represent the social network. While certainly a more compact approach, the result of block-modelling can often appear arbitrary and difficult to interpret. In an attempt to deal with such constraints, Van Ham et al. developed Honeycomb, a visualisation for large scale social networks [10]. Their approach, as illustrated in Figure 2, involves using “concrete organisational hierarchies to drive the analysis”. This is a more intuitive clustering approach that, unlike block-modelling, reflects a user’s prior understanding of the social network. Adjacency matrices can also handle the dynamics of a network in a much more elegant fashion than typical node-link diagrams. A matrices representation remains static as each node is already assigned a position in space. Thus, reflecting change is more accurate, and the result is generally more intuitive. This can be beneficial in two ways. First, change can be illustrated as a heat map in which significant change is represented by the intensity of colour (Figure 2). Second, the visualisation can accurately represent network spread, changes in the network that result from emergent patterns, such as population growth. Matrices are also able to portray areas of inactivity (or the absence of connections) as well as those of intense activity [10]. Again, this is because each node already holds a position in space that, unlike node-link diagrams, can show the absence of information over time. Figure 2 shows a snapshot of the Honeycomb visualisation developed by Van Ham et al 2009. Connections are organised by country and then continent. Blue indicates negative change, red indicates positive change and grey indicates a connection. Colour intensity is used to illustrate the quantity of change. Other approaches, which are not strictly network visualisations but have been used to visualise online communities, include History flow [24], AuthorLines [23] and Communication-garden [27]. While each helps to illustrate user activity in online communities, and portray the characteristics of a community, none of these methods convey community dynamics (or the interaction between users) as successfully as node-link or matrix visualisations. 3.1.2 Social Data Analysis. Social data analysis seeks to take advantage of social as well as cognitive and perceptual process, during the visual analysis of data. Much of the work to date has sought to render visualisation and collaboration tools more usable to communities of non- professional end-users. By far the most popular example is Many Eyes, a collaborative visualisation site developed by IBM [22]. Many Eyes has a complete suite of visualisations, from wordles to graphs, and users are supported in uploading their own datasets or exploring those uploaded by others. However, the collaborative functionality of Many Eyes remains relatively un-advanced, as users are unable to embed annotations in the visualisations. Heer’s Sense.us implements more advanced annotation mechanisms, such as view sharing, doubled-linked discussions and embedded annotations [15]. While illustrating that visualisation, and analytic reasoning, can include social as well as perceptual and cognitive processes, asynchronous collaboration has yet to be applied to social network visualisation. 3.1.3 Summary. While, at present, node-link visualisations experience difficulty with large and dynamic networks, other social visualisations do not express the dynamics (or interactions between members) of an online community effectively. Matrices visualisations, on the other hand, handle large and dynamic networks with reasonable elegance, and can illustrate areas of little as well as much network activity. Although social data-analysis has been implemented with varying degrees of success, there has been relatively no work on implementing asynchronous collaboration mechanisms with social network visualisation. 3.2 The design of Explore.su. We outlined several design goals that helped inform the development of the system. 1. Visualise the social and temporal patterns of the entire Super User community. The social patterns describe interactivity between members of the community and the temporal patterns describe how these interactive patterns change over time [7]. 2. Visualise different user actions. Members of Super User can post questions and answers, vote, comment as well as revise their own and other member’s posts. Each of these actions is considered as a separate communication-act and thus each is addressed with a separate visualisation. 3. Provide annotation and collaboration tools to encourage discussion and reflection 3.3 The Super User Online Community. The focus of the evaluation is the Super User online community, an implementation of Commons-Based Peer Production, which was introduced to address the domain of computer hardware and software. Super User is one of a series of community-driven question and answer sites called the Stack Exchange Network, which, with Stack Overflow, has risen to a position of prominence over the last number of years. The socio-technical implementation of Super User is of particular interest as it seeks to address many of the perceived deficiencies with traditional question and answer sites. First, Super User has both comments and posts. Posts (as questions and answers) are considered first class entities, while comments provide a way for users to seek clarification in regards to a post or suggest an amendment to a post. Posts can be considered as analogous to Wikipedia article pages while comments can be considered similar to an article’s talk pages. Second, each community is specific to a domain (such as programming, gaming or cooking) and questions that are considered off topic can be migrated from one community to another. Again, this approach is analogous to Wikipedia’s name spaces. Third, Super User has an explicit reputation reward, of badges and reputation, which systematically encourages and rewards behaviour. Finally, Super User is collaboratively edited, a feature again drawn from Wikipedia, and has implemented a similar approach to the Wikipedia's Village Pump called a meta site. 3.4 Visualisation. We choose to use adjacency matrices as a representation of community’s network. This was largely because matrices can handle scale and evolution in a more elegant manner than node- link diagrams. Furthermore, matrices illustrate the absence, as well as the presence, of network activity. Figure 3 illustrates the user interface for Explore.su, Figure 4 shows revision patterns for a 24 hours period and Figure 5 shows a popup, which lists all the individual communication-acts for a glyph in the matrix, and is displayed when a user clicks any glyph in the matrix. This provides access to both a global view (illustrated by the adjacency matrix) and a local view of individual interactions. Figure 3: The Explore.su interface comprises of the following components: A) Adjacency Matrix – visualises the communication-acts (questions and answer patterns or commenting patterns or revision patterns or patterns of accepted answers) for the entire community over a given 24 hour period. B) Adjustable Timeline (graph and slider) – allows the user to traverse time, which in turn updates the matrix to a given 24 hour period. Dragging the slider animates the adjacency matrix, thus visualising temporal patterns as shifts in the community’s social patterns. The green line graph illustrates variations in community activity. In the above image, each dip in green line graph illustrates a dip in activity at the weekend. C) User Actions, Graphical Annotation Tools and bookmark. - User Actions – enables the user to visualise different communication-acts. For example, a user can visualise questions and answer patterns or commenting patterns or revision patterns and or patterns of accepted answers. - Graphical annotation tools – allows users to graphically annotate the visualisation. E.g. point, draw or highlight. - Bookmark – allows the user to attach an annotated visualisation to a comment. This bookmark will retain the state of the visualisation (date and annotations). D) Threaded discussion – a set of threaded comments with visualisation bookmarks that allow users to load a visualisation’s state (including any contributed annotations) into the adjacency matrix. 3.4.1 Social Patterns. The matrix visualises community activity for a given 24 hour period. The visualisation is updated nightly (12.01 GMT), giving a relatively up-to-date representation of the community’s communication-acts. As is generally the case with social network visualisations, the Explore.su visualisation is created from the community’s reply-to graph. So if user A replies to user B, a connection is created between those two users. However, communication behaviour in online communities is often non- reciprocal. So, user A replying to user B creates an in-link between those two users without the need for user B to reply-back to user A. Plotting the in-links between users creates an asymmetrical visualisation, which not only highlights areas of activity but also illustrate areas of inactivity (via the absence of connections). Figure 4 illustrates revision activity in the Super User online community. Colour intensity is used to illustrate the number of connections between users of different reputation categories. There are two noticeable patterns. The first dark diagonal line illustrates users revising their own posts, an activity encouraged by the community. The second clump of activity on the left, lower section of the diagonal illustrates users with a high reputation revising the posts of users with a low reputation. This is a convention of the community, as rarely do new users revise other user’s posts. Given the scale of the community (~50,000 users at the time of writing), rendering a single one-to-one matrices was unfeasible. We therefore adopted the approach proposed by van Ham et al. and used a hierarchical overview to drive analysis. We choose reputation as a way to structure the visualisation as it is both intuitive and meaningful to the community. As discussed under the heading Common-Based Peer Production, reputation is a systematic mechanism for rewarding user contributions to the community. So users with a high reputation are generally trusted by the community, while users with a lower reputation are considered more recent members or are members who have yet to contribute significantly (in terms of reward-able activities) to the community. The distribution of reputation can be represented with a log scale. There are a large number of users (~26,655), for example, with a reputation of below ten, while there are only a few users with a reputation of over 50,000. To reflect this distribution, we divided reputation into 37 categories, starting with 0-10 and finishing with 90,000-100,000. The first 10 categories are increased by 10, so 0- 10, 10-20, 20-30 etc. The next ten categories are increased by 100, so 100-200, 200-300, 300-400 etc. The next ten categories are increased by 1000, so 10000-2000, 2000-3000, 3000-4000 and finally the remaining categories are increased by 10,000, so 10,000-20,000, 30,000-40,000 etc. The aim of this representation is to visualise a large community yet, at the same time, reflect the scale-free network topology familiar in large, open web-based systems and provide an intelligible visualisation for end-users. 3.4.2 Temporal Patterns. We also aimed to visualise, and hence provide access to, the community’s unfolding temporal patterns. Temporal patterns describe the naturally occurring rhythms of collaboration, which, when visualised, can help to illustrate aspects of a community’s evolution, such as sudden growth or network spread. At the same time, temporality is proven to play an important role in collaboration activities, enabling groups to coordinate their conception of time and improve efficiency and effectiveness. While there are several approaches to visualising temporal patterns in adjacency matrices, including thumbnail views for example, we implemented an adjustable timeline (as a line graph and slider in Figure 3), which enables users to adjust the date, which in turn updates the adjacency matrix. Dragging the slider continually updates and thus animates the matrix, enabling the user to quickly observe temporal shifts in the community’s social patterns. Figure 5: Clicking on any square on the adjacency matrix will show a popup, which displays the communication-acts represented by that square. In this example, the popup displays questions and answers from users with a reputation of 100 to 200. This approach provides both a global and local representation of the community’s activity. 3.5 Collaboration Support. Collaboration support draws from Sense.us and from Heer’s recommendations for social data analysis [12], providing simple yet intuitive ways for users to gesture towards items of interest and to share observations with other users of the system. 3.5.1 Graphical annotation tools. We implemented several graphical annotation tools to help users communicate intent and contextualise an observation. Pointing is the most important tool in this context, providing users with a way to gesture towards a feature of interest or simply say “look here”. Other forms of annotation, such as highlighting and drawing, are also provided. 3.5.2 Bookmarking and view sharing. To analyse the visualisation, users must be able to see the same view, providing context for observations and user actions. In Explore.su, this is achieved with simple collaboration functionality in the form of visualisation bookmarking, commenting and view sharing. 3.5.3 Threaded discussions. Comments enable users to elaborate on their annotations, to describe the observations in more detail and to share their observations with other users. Comments provide a platform for discussion. 4. EVALUATION. We conducted a two staged evaluation, which involved a pilot lab study followed by an exploratory user study with the Super User community. The pilot study was conducted to ensure that the visualisation was comprehensible and that the overall system design was coherent. The exploratory study use was run as a live deployment over a two week period. 4.1 Pilot study. The pilot study involved 7 subjects (6 males and 1 female). All subjects are computer science PhD researchers from our faculty. None of the subjects are online community researchers and their specialities range from semantic technologies to ubicomp systems. In addition, none of the subjects were familiar with Super User, however, 4 of the subjects had used Stack Overflow, and all of the subjects were familiar with Yahoo Answers (an alternative non-domain specific question/answer site). The first 5 subjects were given a brief tutorial of the system, asked to play around or familiarise themselves with the different interactive elements and then asked to comment on their findings (submit a minimum of 2 comments). The second 2 subjects were given a tutorial video, outlining the system and the process of the study, asked to play around and familiarise themselves with the tools and then comment on their findings (min of 3 comments). While all the subjects were familiar with the core tenets of information visualisation, none are visualisation researchers and none were familiar with collaborative visualisation. No tasks were given, instead users were asked to browse around the visualisation, identify patterns of interest and comment on those patterns using the graphical annotation and collaboration tools. Each session lasted between 50 and 60 minutes. Each session was recorded, notes were taken by an observer and the subjects were asked to think aloud. Having submitted 3 comments to the system, subjects were asked to complete two short questionnaires - a participant questionnaire and a SUS (Simple Usability Score) test. 4.1.1 Evaluation Feedback. Some users expressed frustration at not being able to apply more fine grained filtering to the visualisation, to drill down a little deeper, or to compare and contrast the different user actions – such as following a specific thread over time or identifying the actions taken during the lifecycle of a thread. Other users expressed frustration at not being able to see what people were talking about. For example, one participant asked “what is the relation between the different topics and the activity represented by the visualisation?“ Another participant inquired “what are they talking about?” Some subjects also found it difficult to initially understand the matrices – as a visual metaphor – especially considering the use of reputation as an abstract representation. Few users were familiar with use adjacency matrices as most indicated more familiarity with node-link visualisations. However, once they grasped the general idea, and understood the visualisation as an overview tool, most users set about identifying patterns of interest. 4.2 Exploratory user study. The pilot study allowed us to refine our initial implementation and to evaluate how users perceived the matrices over the more familiar node-link diagrams, the second evaluation provided us with a platform to study reflection in the Super User online community. We recruited members of the Super User online community. A proposal for the study was posted on the Super User meta-site5. This was initially endorsed by a community moderator but subsequently removed within twelve hours by a Stack Exchange moderator who argued that this was an advertisement and against Stack Exchange policy. As a consequence, each day we advertised in the Super User chat rooms, which tend to be used by the more committed user-base. 28 people signed into the system, and 17 completed the evaluation. As with the lab study, participants were asked to sign in, watch a short tutorial video, submit a minimum of five comments and then complete two short questionnaires. The first questionnaire related to the process of the study, inquiring about their impressions of the study and if they would be amenable to further contact, while the second questionnaire was, again, a SUS (Simple Usability Score) test. The process, they were informed, would take a minimum of 30 minutes. Six Amazon vouchers were raffled to encourage participation. 4.2.1 Findings. We received positive initial feedback about the prototype, especially from the more senior members of the site. Initially, some interesting comments were submitted outside of the Explore.su system – in the Super User chat room and on the Super User meta site. For example, on the Super User meta-site, one user commented “considering how insightful this is, maybe SE (Stack Exchange) should think of implementing?” In the Super User chat room, another commented on an insight they discovered using the tool “fascinating, 100-rep people are the ones who're asking the most, 1000-3000 rep ones are the ones to answer the most”. Figure 6: Results of the content analysis on Explore.su. commentary. Each category is considered mutually exclusive. Observations were most prevalent. However, several participants submitted hypotheses or questioned a previous user’s observation. Later, the same user contributed “comments and revisions have the same pattern of distribution, obviously comments distribution is more dense [sic]”. Participants also submitted comments as part of the Explore.su system. For example, one participant commented “these two clusters, based on the Questioner's rep, make me wonder how users cross the gap from 50 to 100 rep (reputation)”. Also, several users identified clusters around reputation of a 100. For instance, “here there's a concentration around 100 rep (reputation)” and “there's a lot of activity concentrated around the 100 rep (reputation)”. Finally, one participant remarked that “while there is a large variation of the reputation of commenters [sic], there is good correlation of users presumably actively responding to comments (diagonal line), especially in the 100-1000 range”. 4.2.2 Content analysis of comments. We conducted a content analysis on the commentary from the exploratory user study. We wanted to learn to what extent the visualisat"