"1 Introduction to iSpot. Fig. 1. The iSpot home page, showing images of latest observations (top left), observations from this time last year (bottom left), and the sign up/search/log in panel (right). iSpot [1] is a website that enables learners to post observations of wildlife, and get help with identifying what they have seen from experts and other users. The aim is to encourage learners to engage with the natural world, to enable them to identify species and hence to learn about what they have seen. The main activity on iSpot is the posting of observations (see Fig. 2). Learners upload photos of wildlife they have seen, and log when they saw it and where. If they are able, they also add an identification: the name of what it is they have seen. To help with the quality of the identification, iSpot can look up scientific names from common names, and check spelling with the dictionary of UK species curated by the Natural History Museum. Other users of the site – whether experts or beginners – can then click ‘agree’ on the identification if they think it is correct, or post an alternative identification if they think it is not. A sophisticated but simple-to-use reputation system, described in more detail in section 2.2, gives users of the site feedback about the expertise of other users. The site is organised into eight broad taxonomic groups: Amphibians and Reptiles; Birds; Fungi & Lichens; Fish; Invertebrates; Mammals; Plants; and Other organisms. Fig. 2. An observation on iSpot, showing the user who submitted it (top), the image captured (just below), and an identification, the number of agreements with that identification, and links to the Encyclopedia of Life and the NBN (shaded box at bottom). The UN declared 2010 to be the International Year of Biodiversity, to celebrate the diversity of life on earth and of the value of biodiversity for human lives [2]. However, expertise in correctly identifying observations, for baseline recording of biodiversity, is in such short supply that automation has been proposed [3]. But social networking, as harnessed in iSpot, can make the process more efficient, and spread taxonomic knowledge [4]. The goal of iSpot is not to produce new data, but new data recorders. (1) Fig. 3. The iSpot ecosystem, showing the iSpot website (centre) surrounded by users, experts, formal learning, broadcasts, iSpot Keys, and mentors (around). The iSpot website does not stand alone. It is part of an ecosystem of activity specifically designed to support and encourage learning at scale (see Fig. 3). In the UK2 there are many (largely) voluntary expert organisations who are the repository of expertise in identification and recording of particular taxa of wildlife. There are also many millions of enthusiastic viewers of nature programmes. With strong links with BBC TV and radio broadcasts, iSpot helps motivate users to take their interest further, providing an informal learning space. Learning pathways that take users beyond the website were also designed in from the start. There are the expert organisations, who are generally very keen to welcome and nurture new members. There is also a carefully-designed route through to formal learning, via a short open-entry universitylevel course called Neighbourhood Nature, offered by the Open University in distance mode, which could form the first step on a journey through to a related degree. The ecosystem also includes a network of biodiversity mentors, who engage in outreach activities in their local regions, taking pro-active steps, particularly to reach under-represented and under-privileged groups. Finally, there are iSpot Keys, which are designed to provide a new, more learnerfriendly way to identify species than traditional dichotomous keys. A key strategy of the project was to aim to do one thing well (social networking to support learning about identification of wildlife), leaving other sites to do other tasks more effectively. So apart from the iSpot Keys, the site does not itself provide significant help with identification, but links instead to other excellent resources that are available online, such as field guides. Similarly, the site does not contain information about particular species, but provides direct links to the Encyclopedia of Life (EOL) [5], an online reference which aims to cover all 1.9 million species known to science. (1) However, useful data has been generated: some expert organisations working with iSpot are very interested in transferring records to formal recording schemes. (2) At the time of writing, iSpot is funded to operate only in the UK and Ireland. No geographic restriction is in place on use of the site, but as species – and expertise in identification – vary so considerably around the world, we are working to set up versions of iSpot tailored to other areas. 2 Design Rationale. 2.1 General. iSpot is an example of what the MacArthur Foundation describes as participatory learning [6]. The learner is an active participant, engaging in authentic, open-ended creative activities, developing their interest and passion. This type of participatory learning is an excellent complement to traditional approaches to Open Educational Resources (OER), providing a motivating link between the worlds of informal and formal learning [7]. The design of iSpot was underpinned by the notion of shared social objects (see e.g. [8][9][10][11][12]): there was a focus on sociality, rather than pure functionality; and on social interactions mediated by a social object (the observation), rather than a pure social network. Both the object (the observation) and the social conversation related to it (the identifications, agreements and comments) are important. Fig. 4. Fairy rings of participation, showing the visible layer of participation operating in four different modes, underpinned by an invisible network of interactions. In a community of practice [13], members learn from legitimate peripheral participation [14]. Preece and Shneiderman [15] categorise successive levels of social participation in online communities as reading, contributing, collaborating and leading. An analysis of participatory activity on 50 sites [16] suggests that users connect, participate and collaborate around a shared object, transferring information and pooling knowledge within and between communities, in four different modes: 1. Browsing, gathering and sharing content 2. Giving and receiving feedback and expertise 3. Collaborating and jointly deciding about actions 4. Sharing control with other members over the community and content A user’s participation is influenced by several hidden elements (e.g. reciprocity, identity, habits, real-world probes), which may not be immediately evident in their actions, but can be inferred by analysing user reactions. Engeström [17] compares such underground activities to mycorrhizae: symbiotic associations between the fungus and roots of a plant. We liken this process to a “fairy ring”, a circle or arc of mushrooms, where the mycelia of a fungus grows invisibly beneath the surface, but its presence can be inferred from occasionally wildly fruiting mushrooms, and also from careful examination of the surface. Figure 4 shows what we call a “fairy ring of participation”, which informed the design of iSpot and the analysis presented below. All activity on iSpot is visible in the sense that there are no private or restricted spaces. This is to maximise the potential for learning through legitimate peripheral participation, and to reduce the potential for unpoliced abuse. The development process of iSpot was underpinned by this theoretical understanding, and followed a simplified version of the socio-cognitive software design approach [18][19][20], drawing on aspects of Agile development [21]: extensive consultation and envisioning with stakeholders, followed by rapid iterative release informed by feedback from users. 2.2 The Reputation System and Learning Analytics In formal learning, a teacher (loosely defined) arranges learning activities for students, and motivates and tracks their progress through formative and summative assessment. Indeed, it is widely attested that assessment ‘defines the de facto curriculum’ [22][23]. The results from assessment can also be used to inform improvement of the learning activities. This feedback is very important for both the learners (to improve their own learning) and the teacher (to improve their teaching). In informal learning, there is no ‘teacher’ as such, although there may be someone who arranges learning activities for others. The participants do not usually have the same access to external feedback about their learning. Additionally, a teacher in formal learning can often track student progress through direct observation of learning situations, whether face to face or online. Learning analytics can be useful in making learning visible in ways it would not otherwise be. Learning analytics promise to be particularly useful for informal learning contexts, where direct observation is even more problematic. The iSpot reputation system was designed to support learning on the site by providing a form of external feedback, recognizing and rewarding the activities that the team wanted people to engage in on the site. There is a large literature on reputation systems (e.g. [24] [25] [26]), which has informed the design of the iSpot system. The purpose of any reputation system is to facilitate trust between users, by making actions and feedback transparent, and encouraging reciprocity. Particular attention was given to incentives for users to provide honest, positive feedback. Some users find negative feedback disproportionately demotivating; some might seek to escape it by creating a new, separate account. So no facilities for negative feedback are provided. One concern is the potential problem of users colluding to rate each other positively, perhaps using multiple ‘sockpuppet’ identities. This is known to be very hard to defend against without a central, trusted authority [27] that can identify a trusted user or users to be the source of reputational score [28].3 iSpot uses just this approach, using the network of experts from expert organisations. The use of ‘experts’ as a source of reputation does not depend on a particular disciplinary epistemology: for subjects outside ‘hard’ Science, Technology, Engineering and Medicine, it might be more appropriate to describe them as what Northedge [29] would call fully functional users of the discourse within a particular specialist discourse community. There are two aspects to the iSpot reputation system, which are displayed together on the users’ profiles (Fig. 5): social points, and scientific scores. Fig. 5. Part of an iSpot profile for a single user, showing social reputation (Social Points), and then, for each iSpot group, a ‘star’ rating of icons, with a count of observations added, identifications made, agreements received, and agreements given. Note a single icon for Fish (denoting score > 0), a gold icon for Fungi and Lichens (denoting an expert), and three icons for Plants (denoting score > 10) – see Table 1. Social points are gained for engaging in activity on the site: the number of observations posted, the number of identifications made, and the number of agreements received and given. Even these ‘social’ measures are not entirely separate from technical/scientific approval: ‘agree’ appears superficially similar to a social network’s ‘like’ functionality, but here it carries the connotation ‘agree that this identification is correct’, an arguably less subjective judgement. (3) Or for each user to calculate reputation scores for themselves – which is mathematically equivalent to them treating themselves as a trusted user. This is unwieldy in a social network, since different users would see different reputational scores for the same user. Scientific scores are intended to be a rough measure of expertise in accurate identification. Because the skills needed for accurate identification vary between species, the scientific scores are recorded separately for each of the eight groups. The scores are accumulated as follows: experts (known to the iSpot team to be such) are given a starting score of 1000,4 and all other users start with 0. If user A ‘agrees’ with an identification of user B, user B’s score increases by user A’s current score divided by 1000. So, for instance, if an expert agrees with user B’s identification, user B’s score will increase by 1000/1000 = 1. If user B – with a score of 1 – then agrees with an identification, that will increase the score of that user by 1/1000. For any particular identification, a user can accumulate a maximum of 1 to their score. Table 1. Relationship between score and icons shown. The scientific scores are not shown directly, but indicated by a number of icons representing the score, calculated on an exponentially-increasing basis5 as shown in Table 1. These icons are displayed on a user’s profile page (Fig. 5). The relevant icons are shown next to a user’s name in any context where their reputation in that group is pertinent. So, for instance, it is shown next to their name on an observation (Fig. 2), and next to the list of users who have agreed with an identification (Fig. 6). Fig. 6. Part the list of users who have agreed with a particular observation of a Bird on iSpot, showing the names of users who have agreed with the identification, icons indicating their reputation in the Bird group (blue bird icons, gold-circled bird icon), and relevant ‘badges’ for expert organisations (e.g. black butterfly on yellow square). (4) The factor of 1000 was chosen based on many factors, including: providing a clear initial distinction between known experts and novices; providing scope for reputation to grow over time; and making it very tough but feasible to achieve the maximum score. (5) An exponential was chosen over a linear approach in expectation of the highly non-linear pattern of activity and reputation that is observed – see p.11. The ‘agree’ mechanism is designed to encourage and reward reciprocity on the site: nearly anyone can agree with an identification, regardless of their knowledge, and give it a virtual ‘thumbs up’. However, the reputation score calculated on the basis of these agreements is not so egalitarian. The site displays ‘badges’ for users who have been identified by the iSpot team as being members of expert organizations. This provides a reputational marker for those experts, and also provides a way for those expert organisations to connect with interested users (the ‘badge’ is a link to their organisation’s web page). The experts thereby have an incentive to contribute constructively to the community. In addition, a key source of expert contributors to the community is the iSpot project itself, through the expert team members and through the network of mentors. 3 General Learning Analytics. Having explained the design and functionality of iSpot, this paper now presents empirical data to illustrate how participatory learning and reputation unfold in a real setting. There are three main sources of data: Google Analytics, active on the site since before the launch on 27 June 2009; the database of the site itself; and qualitative analysis of the activity on the site. The dataset analysed covers the period from the earliest online availability of iSpot on 29 September 2008 up to 4 October 2010. There is very little activity before the public launch on 27 June 2009. The dataset contains 6,487 users, 27,493 observations, 47,355 images, 33,088 identifications, and 83,029 agreements. Users have posted an average (mean) of 4.2 observations; observations have an average of 1.7 images and 1.2 identifications; and identifications have an average of 2.5 agreements. However, some of these averages disguise very asymmetrical distributions, as will be shown. Fig. 7. Visitors to iSpot over time, from Google Analytics. The number of visitors to the site (Fig. 7) shows many interesting patterns. The number of visitors increases over time, but shows a series of very sharp spikes. There is also a clear weekly pattern of activity below the big spikes: the site is much more heavily used during the working week than at weekends. The very sharp spikes occur at the same time as iSpot received coverage in the mass media. The first is on 13 October 2009, when a news story about a six-year-old girl spotting (via iSpot) a rare species of moth for the first time in the UK received considerable coverage in the national press. The next spike occurs on 6 April 2010, when a national radio programme (Saving Species on BBC Radio 4, weekly audience over 1m [30]) mentioned iSpot at length, with a call to action. iSpot has an ongoing relationship with the programme. There are subsequent spikes in traffic, on 22 June 2010 and 29 June 2010, when iSpot was again featured with interviews with iSpot users and an explicit call to action. There is a further spike on 18 May 2010, when a popular television broadcast (Springwatch on BBC2, audience 2.1m [31]) mentioned iSpot on air. There is a clear difference in the pattern of activity around these spikes. The first spike, around the moth story, shows little ongoing effect after the initial interest. But the traffic spikes from the broadcasts show evidence that a significant proportion of those visitors became ongoing, long-term visitors and contributors. Indeed, Tuesdays (the day Saving Species is broadcast) are the busiest day of the week on the site. Fig. 8. Observations posted to iSpot, by month posted, from database. The data from Google Analytics are matched by activity apparent in the database. Fig. 8 shows the number of observations posted to iSpot per month. The broad shape is similar, but the pattern is smoothed out. There is some evidence of an overall seasonal peak of activity, matching the number of species active and the desirability of field work. (Only the hardiest insects and naturalists are active in mid-winter.) Table 2 shows the activity on the site, broken down by group. Invertebrates is by far the most active group, followed by Plants, Birds, and Fungi & Lichens. The other groups have very low activity by comparison. There are interesting patterns in the numbers of users who post observations, identifications, and agreements. For Invertebrates, Fungi & Lichens, and to a lesser extent Plants, these figures fall off fairly sharply. For other groups, including Birds, the decline is much less steep. This may be because identification of common species in Birds is easier for non-experts than in other groups. Table 2. Activity on iSpot, broken down by group, from database. The seasonality apparent in the overall numbers of observations (Fig. 8 above) is revealed in finer texture when the data is disaggregated by group as in Fig. 9, which shows the number of observations per month by the date of observation. (6) Fig. 9. Observations posted by month observed (not date added to iSpot) for selected groups. (6) NB The date of observation is different from the date the observation was added to iSpot. Users often upload observations going back some time – there are even two observations in the dataset with an observation date in the late 1980s. The peak for Invertebrates appears to come in July/August, but for Plants slightly earlier. Fungi & Lichens peaks much later in the year. These peaks parallel the abundance and ease of identification of the species: plants are most easily identified when in flower, and it is almost impossible to identify fungi when they are not fruiting without specialist equipment. The Fungi and Lichens peak in November 2009 may be the result of a related survey on air quality, which encouraged observations of lichens. The Birds group appears to have two annual peaks: it may be that the spring peak is associated with an abundance of migratory species, and the winter one with ease of observation (lack of foliage) and lack of other observation opportunities. Thus, the analytics here reveal the texture of the learning experience, with strong echoes of real-world activity beyond the website itself. In this and subsequent sections, the paper will focus mainly on the Invertebrates group, the largest and most active group, for reasons of space. Other groups are largely similar in overall character. Fig. 10. Invertebrates: Observations per user, ordered by number of observations. The activity on the site is far from evenly distributed. Figure 10 shows clearly that the pattern of posting of observations in the Invertebrates group is very unequal. Twelve users have posted more than 200 observations; 194 users have posted fewer than ten. The pattern in other groups is very similar, even in groups with much lower activity. This type of ‘long tailed’ steep distribution has long been attested as a feature of social networks [32]. Anderson argues [33, p126] that it requires three factors: 1. Variety (there are many different sorts of things); 2. Inequality (some have more of some quality than others); 3. Network effects, such as word of mouth and reputation, which tend to amplify differences in quality. It seems highly likely that all three factors are present on iSpot: the reputation system is precisely designed to amplify differences in quality and reflect that in scores. However, the precise nature of these steep distributions is not immediately clear. Figure 11 shows the same data as in Figure 10, but presented on a log-log scale with a power law drawn for comparison. Even if the data were a perfect fit to this straight line, considerable care would need to be taken before one could claim with any certainty to have discovered a power law [34]. The distribution is steep, and has ‘fat tails’, as does a power law, but is not a power law, or at the very least it is not the same power law over all of the data. Fig. 11. Invertebrates: Observations per user, ordered by number of observations, log-log plot, showing power law (dotted line) with exponent of -1.3. 4 Qualitative Analysis. This section presents, for context, comparison and triangulation, a brief qualitative analysis of the activity on iSpot in general, and in two of the groups. 4.1 General Activity. In iSpot, experts, mentors and citizen scientists form the core community, while novices, students from the Neighbourhood Nature course, and nature enthusiasts form the peripheral clusters. The core community helps create links between the different groups by helping out, encouraging and opening up social conversation in the forums and comments section. This conversation shows a transfer of knowledge within and between the different groups, and also with external communities (e.g. Wikipedia, BugLife, British Beetles etc). Initial motivations for commenting, adding an identification or an agreement include content quality of the observed species (e.g. “cool photograph!”), uniqueness (“very rare, I understand it only appears for 3 weeks each year in May and June.”) and relevance (“The hummingbird hawk has it's very own They Might Be Giants song”). However, the level of activity and mode of participation of the different users is highly dependent on the ‘invisible layer’ of underlying elements and motivations (see section 2.2). These include reciprocity (e.g. a user adding content will stop participating if their activity is not reciprocated, while activity that is reciprocated encourages them to add more content; this is especially true for the new members and novices in the scientific field of the group); real-world probes (students asked to post observations as part of the Neighbourhood Nature course); and identity (experts knowing each other will discuss their findings and collaborate to find the right identification for the species). There is considerable evidence of interaction with layers of community beyond the site itself (e,g. “Tompot Blenny? Look at my flickr pictures”, “there's a BBC story about 10 million ladybirds descending on a farm in Somerset”, “all my colleagues now know what a nettle weevil looks like”). 4.2 Invertebrates. The Invertebrates group shows activity in all four modes of the “fairy rings of participation” model: browsing, gathering and sharing content; giving and receiving feedback and expertise; collaborating and jointly deciding about actions; and sharing control with other members over the community and content The rare moth story (the news story in section 3) illustrates this: an initial observation (“My daughter found this strange moth on our windowsill”) sparked increased interest and activity, growing spectacularly when it was stated that a new species had been discovered (“this is indeed the first British record of the Euonymus Leaf Notcher.”), and this knowledge was transferred into the wider global community (“a colleague in Taiwan, within the moth's native distribution, also confirms it”), and the discovery secured a place in the Natural History Museum collection – after the girl had taken it to Show And Tell at school. There are many examples of learning in a social yet scientific environment. One Neighbourhood Nature student and iSpot beginner starts with a shy observation of a micro-moth and request for help from the experts; this grows into several consecutive activities, with the learner making more accurate and scientific descriptions of the observed species and showing increased confidence about identifications of other users’ observations. 4.3 Fungi And Lichens. The Fungi and Lichens group contains one single key expert, plus nature enthusiasts and Neighbourhood Nature students. This is a small group with considerable activity, with users helping each other to identify species and providing many agreements – again, working in all four modes. One learner joined the group as a course requirement for Neighbourhood Nature, with little knowledge. Her activity and progress is extraordinary, uploading images and making identifications on her own observations but also on other users’ observations. She links information from external sites to support her arguments, receiving feedback and many agreements from experts and other users. She improves so much that when an expert makes an observation (“white fungus on log with teeth like outgrowths”), she makes a suggestion that it is something different (“Phlebia rufa – Small crust-like fungus growing on dead wood”) that makes the expert rethink and alter their identification (“after [the] suggestion of Phlebia … I was wondering if this one might be a Phlebia instead since it has the pores of a suitable shape …”). A key element in this social conversation is reciprocity in the community around the learner, including the expert’s support. 5 Reputation Learning Analytics. The reputation scores on iSpot are, we believe, unique as a measure of activity on a social network: they are designed to be a proxy measure of skill in a particular task, rather than a more social measure of acceptability. The analysis presented in this section is exploratory, rather than complete. 5.1 Reputation Received The reputation received by users on iSpot in terms of icons (as per Table 1 above) – is shown in Table 3, along with the number of accredited experts (‘gold’ icons).7 Across all groups, there is a clear tapering off, with large number of users having earning a single icon, and increasingly smaller numbers having earned more. Table 3. Distribution of reputation icons earned by group. The reputation score accumulated by individual users in the Invertebrates group is shown (logarithmically) in Fig. 12. As with the number of observations, it is clearly a sharply unequal distribution. There is also a sharp discontinuity in the curve. This occurs between a reputational score of 1.0 or higher, between N = 618 and N = 619 in the ranked data. The users with a score below 1.0 have not had any expert agree with their identifications, since a single expert’s agreement will add 1.0 to the score. It is perhaps unsurprising that there is a difference in the dynamics of score between those who have received some reputation from experts, and those who have only received it from other non-experts. (7) Not all of these experts are actively engaged on the site in any given period of time. Fig. 12. Invertebrates: log plot of reputation received, ranked by reputation received, showing clear discontinuity at reputation < 1.0. Fig. 13. Invertebrates: reputation received, log-log plot, first 618 users (reputation score ≥ 1.0), showing power law (dotted line) with an exponent of -1.4. The two sections of these data are shown separately in Figures 13 and 14. The shape of the graph for those with reputation ≥ 1.0 (Fig. 13) is not a power law (i.e. not a straight line), but it is very clearly long-tailed. For the population with reputation < 1.0, yet to come to the attention of experts, a logarithmic relationship appears to be a good fit (Fig. 14). These differently shaped distributions are consistent with the patterns of activity being different between the two subpopulations. The data for the other groups are strikingly similar: there is a distinct discontinuity between the population with scores influenced directly by experts, and those without. Fig. 14. Invertebrates – reputation received for users with reputation < 1.0, showing logarithmic curve fit (solid line) giving y = -0.197ln(x) + 0.8317, R2 = 0.98. NB Not log plot. 5.2 Reputation given. Fig. 15. Invertebrates: Reputation given ordered by reputation given, log-log plot, showing power law (dotted line) with exponent of -3.6. The reputation given by a user is the sum of the amount by which they have increased other users’ scores by clicking ‘agree’ on an identification. This quantity is not shown on the site in any way. The sum total of reputation given will exceed the sum of reputation gained, since an individual identification can only gain the user a maximum of 1.0 point, but many experts may agree with the same identification, which is counted as reputation given of 1.0 points times the number of agreements made. Figure 15 shows the distribution of reputation given for Invertebrates. Once again, the data do not fit a power law, but they are clearly very unequal. The sharpness of the fall-off is much higher than for the distribution of observations and reputation received. The data do not observe power laws, but the exponent of a fitted power law can give a (very) rough indication of the sharpness of the decay, yielding -1.3 for the observations (Fig. 11), -1.4 for the reputation received (Fig. 13), and -3.6 for reputation given (Fig. 15): a dramatically steeper curve. The data for other groups are very similar. In summary, the reputation given on the site follows a very, very steeply unequal distribution, even by comparison with observations posted and reputation gained. This is consistent with the intended purpose of the reputation system in magnifying the impact of known experts on the distribution of reputation. 5.3 Reputation Given vs Reputation Received Fig. 16. Agreements received against agreements given for Invertebrates, log-log plot, showing fitted power law (dotted line) with exponent 0.57 and R2 = 0.47. Is there a relationship between the amount of reputation given and received? Figure 16 shows a log-log plot of the number of agreements. The data suggest that if there is a relationship, it is steep and very widely scattered, particularly for lower numbers of agreements: a power law explains less than half of the variance. For reputation given and received (Fig. 17), the picture is similar: such relationship as there is is very non-linear, and the data are widely spread and not well explained by a power law. So there may be some correlation between agreements given and received, and between reputation given and received, but the relationships are different, not remotely linear, and very highly variable: most users give far more agreements/reputation than they receive, or vice versa. Fig. 17. Reputation received against reputation given for Invertebrates, log-log plot, showing fitted power law (dotted line) with exponent 0.345 and R2 = 0.62. 6 Discussion. This exploration of analytics data on iSpot reveals both the gross, large-scale picture of participatory learning activity on the site, and some of the more fine-grained, nuanced texture of activity, illustrated by the qualitative sketches. Patterns of activity"