"1. INTRODUCTION. Today, it is not uncommon practice for faculty to deploy an online instructional environment in order to promote student conversations over course materials and ideally to heighten comprehension and retention of course readings. Co-blogging is an example of a technology-mediated learning activity of this sort. The “blogging” part requires students to explain course material in their own words. The “co-” part makes the students exchange ideas and interact regarding the course material by discussing the alternate viewpoints that emerge within the blogosphere [26]. One of the trends in the American university system over a number of years has been the migration towards larger so-called “gateway” courses [28]. These introductory courses are typically a student’s first exposure to collegiate work. However, these large classes have a negative impact on the student’s learning process. Large lectures are useful for certain things, but less useful for fostering higher order thinking [31, 10]. Technology like co-blogging, however, can be used to make the class seem smaller, and provide the foundation for students to learn more and better. For most if not all learning activities, a substantial amount of an instructor’s time and effort is devoted to evaluating and monitoring the quality of students’ work, and thus, hopefully, the depth of their learning [11]. The purpose of this monitoring, however, is not merely the determination of grades; part of the instructor’s work is entirely self-reflective, enabling the instructor to concurrently, or ideally even preemptively, intervene to make adjustments to course pedagogy based on students’ engagement or understanding [29]. While assigning grades might be facile, some difficulties complicate this second objective: how might an instructor intuit when, precisely, students have understood the material sufficiently? Making this determination manually, specifically in larger gateway courses, would prove an intensely laborious and time-consuming process, far more complicated than simple reading and re-reading of any single student’s work. When supporting learning using technology, however, a positive by-product is that students produce their work in an electronic form, which enables the creation of computerassisted instructional aids [22]. This paper describes an automated solution that can be used by educators to help resolve these tensions. Through the application of lexical analysis to student writing, we have implemented an analysis tool that allows an instructor to track how a student’s written language migrates from mere paraphrase to mastery, isolating the moment when the student’s understanding of core concepts best demonstrates an ability to place that concept in his or her own words, a moment that we’ve chosen to call the “point of originality.” This process recreates the same cognitive activity that educators might ordinarily undergo, yet in an automatic manner that is less labor-intensive. Ultimately, the resulting data is presented to the instructor by way of a custom visualization, which allows for continuous self-monitoring with minimally expended effort. In order to demonstrate the utility and validity of the analysis method, this paper explores the potential of the point of originality to preemptively predict likely student success or failure. The remainder of the paper is organized as follows. Section II explores the benefits of co-blogging in education. Section III and IV provide background on the particular problems of, and potential solutions for, evaluation in larger gateway courses. Section V and VI introduce the theory behind, and the specifics of, the point of originality analysis method. Section VII and VIII report on a case study where the point of originality method was used to evaluate student co-blogging data. The paper ends by discussing future work. 2. CO-BLOGGING AND EDUCATION. Co-blogging is an example of a social computing activity that can be very conducive to learning [13, 26]. Overall, blogging provides a platform that promotes individual expression, enables students to establish their own “voice” and yields a richer conversational interactivity within a community [48, 47]. Each student has a blog, composed of multiple blog posts. Students can read each other’s blog posts and comment on them. Because blogs are easy to use, they can promote students’ digital fluency [23] and encourage students to explore and publish their own nascent ideas under less pressure than in the rough-and-tumble of in-class discussions [3]. Writing a blog forces students to become analytic and critical as they contemplate how their ideas may be perceived by others [47]. Being able to review older contributions affords reflection and enables students to revisit and revise their artifacts, further developing their own viewpoints in the context of each other’s writing as they sense how others understand the material similarly or differently [37]. Conversations emerge when students read, and then comment on, each other’s blog posts, thus enabling them to exchange, explore, and present alternate viewpoints on the course material [17]. This type of social explanatory discussion can benefit learning [12, 9]. Alternative spaces such as asynchronous discussion forums are another example of a technology that is sometimes used to mediate online discussions between students. However, as predominantly shared community spaces, forums give students voices that are heard but are without a distinct, individual identity [14]. Critical thinking may emerge for individuals, but the organization does not promote the coherent and interactive dialogue necessary for conversational modes of learning in the same way that co-blogging does [44]. A blogosphere can function as a repository of information, opinions, monologues and dialogues about course content, where students participate, and leverage each other’s contributions in other educational activities (e.g., when writing term papers) [2, 1]. Blogging enables students to gather their thoughts and come better prepared for class [24, 12] and can be predictive of student performance in a course [13]. Even non-active student bloggers can benefit from the blog’s educational value as it exposes them to different views of the material without necessarily participating directly [47]. Overall, having students discuss and/or “argue” about course readings has significant educational utility [5, 4, 38]. Some discussion might take place during class, however, class time is a limited resource. This is particularly true for larger classes or so-called “gateway courses” that typically enroll large numbers of undergraduates where there is simply not time for everyone to speak up. Using co-blogging, students can both express individual voices and continue conversing with their peers outside the confines of the physical classroom. Unfortunately, the sheer size of these courses presents several challenges. 3. PROBLEMS WITH LARGER GATEWAY COURSES. The ability to monitor and respond to student progress is ever more imperative given the realities of the modern classroom. As noted even a decade ago, the political economies of American universities increasingly mandate large class sizes, particularly in the introductory or “gateway” courses that are typically a student’s first exposure to collegiate work [28]. These large classes have a negative impact on students and instructors alike. There is, for example, an abidingly inverse correlation between class size and student achievement [20, 42]. The large lecture, while useful for reinforcing rote facts, is less successful in fostering higher-order thinking [31, 10], or in encouraging students to construct their own understanding of core concepts [28]. Such a sizable student population further constrains instructors’ abilities to familiarize themselves with students’ individual learning styles [28], thereby forcing instructors to assume that their audience consists of uniform types of learners [10]. Although the extent of feedback that students receive is one of the most powerful predictors of student achievement [46, 40], instructor feedback in large lecture courses is often slow and sporadic; students typically need to wait weeks - from, for example, one midterm assessment to the next - to put their course-related skills into practice, and even longer than that to have their assignments evaluated by an instructor [10]. Pedagogical adjustments, in other words, become both more unwieldy and more unlikely in the precise environment where they would be most necessary. Given the problems inherent to large lectures classes, but given also their entrenched status within the American university system, it would thus logically be prudent to find a way to minimize their most pernicious consequences. Any broader attempt to remedy the problems of larger gateway courses should thus aspire to first, foster higher-order thinking; second, to suit multiple types of learning styles; and third, to provide students with feedback as rapidly as possible. These first two objectives are inherent virtues of the co-blogging process (e.g. [24, 12, 37, 9, 5, 4, 38]); the final objective is the focus of this paper. 4. PRIOR EFFORTS. Several attempts to minimize the unintended consequences of large gateway courses exist. Almost all efforts call for resizing the large class group, either by literally subdividing the class or else by designing activities to make the large class “seem” small. This latter method, it might be argued, is the one already pursued by student participation in a coblogging environment, where conversations take place in an ad hoc and freeform manner. Known interventions can be roughly classified into two major groups: those interventions that are specifically meant for in-class use, and those interventions that are intended to take place between classes. Those activities that take place during class typically interrupt the lecture itself [32, 35, 8], asking students, for instance, to respond to a series of prompts which they answer through remote devices. These same activities, however beneficial, generally disrupt the actual process of knowledge transmission and tend to reward rote memorization rather than higher-order thinking; what feedback students receive reflects only whether or not they got a prompt right or wrong, and not how well or how comprehensively they understood the material. Since the activities take place in the classroom, and in front of the entire student population, the activities themselves moreover treat all students in exactly the same manner regardless of learning style. Interventions intended for use between classes are roundly invested in providing instructors with observable statistical modeling in near real-time [39, 8, 19], which can then be referred to before the next session. These activities typically attempt to breed higher-order thinking by forcing students to reflect on their own learning, asking, for instance, that students rate their level of confidence before responding to prompts [8], or that they engage in a collaborative peer review of one another’s written work [39, 19, 21]. The benefits of this type of activity are directly analogous to the benefits of co-blogging. 5. ORIGINALITY IN STUDENT WRITING. When students engage in a writing activity, the final evaluation of their work cannot only assess whether or not the student has provided the most closely correct answer. Process is just as relevant to student writing as content [43]. Student writing that exhibits exceptional higher-order thinking is generally seen as that which demonstrates a mastery of the course material in new, profound or statistically unusual ways [33]. The ideal is not only for students to confirm that they’ve understood lectures, but to do so in ways that even the educator might not have thought of. This process of mastery need not take place all at once. As a student is continually exposed to the same material, or is given the independent opportunity to rethink, reframe, or revisit that material [45], their writing on the subject has the chance to evolve, from rote regurgitation to wholly original expression [34]. At the level of language, this evolution is reflected through recasting. Recasting is the learning process whereby a student refines his or her understanding of a concept found in course lectures or readings by putting that concept into his or her own words [41]. In the acquisition of new languages especially, this process can be useful, because it allows students to acquire new vocabulary using the assortment of words already available to them [41, 30]. Even where the student’s understanding of a language is not an explicit concern, recasting can mark a student’s attempts to graduate to more sophisticated or professionalized terminology, or, inversely but to the same end, to place new concepts into terms that are nearer to what the student would naturally be more likely to say [15]. “Originality,” fully defined, can of course take numerous forms. The concept of recasting, however, spans a number of theoretical orientations, with an influence on theories of schema formulation [25], the sensemaking process known as “scaffolding” [18], as well as the express principles of educational constructivism [27]. “Originality,” as deployed here, does not therefore strictly refer to a student’s creativity or capacity for non-conformity as might be suggested by more colloquial uses of the term. Rather, what the Point of Originality tool seeks to gauge is students’ ability to interpret, to place core concepts into new and diffuse usages. This definition of originality straddles the tiers of learning that Bloom’s taxonomy [6] associates with “understanding” and “application.” By interpreting core concepts and extrapolating to different terms, students demonstrate their understanding of the material, and when putting those concepts into play through iterative exercises like co-blogging, they begin to apply that knowledge is newer and more diversedomains. For an instructor, the simple identification of recast terminology within a student’s written work can provide an effective barometer for pedagogical self-reflection. If a subset of terms or concepts are deemed vital to the syllabus, repetitions and recast iterations of those same terms will at least suggest that those terms are being acknowledged and reflected upon. Although the presence of recast terminology is not the only metric representative of a student’s mastery, the central role that recasting plays in a host of pedagogies (e.g. [25, 18, 27]) suggests that writing demonstrating high or low levels of recasting will reflect other aspects of performance within the course. Yet if the instructor hopes not only to identify instances where key concepts are deployed, but to determine how comprehensively the concepts are being internalized, it is first necessary to possess a method of scoring how original any given recast might be. In order to do this, we have developed a metric for isolating a specific point of originality within student writing. 6. EVALUATION OF CO-BLOGGING. The process of computer-assisted evaluation of student writing is primarily composed of two parts: the analysis method, and a custom-made visualization depicting each student’s “originality” at any given time throughout the duration of the semester. 6.1 Analysis Method: Theoretical Background. WordNet is a lexical database that arranges nouns, verbs, adjectives, and adverbs by their conceptual-semantic and lexical relationships [16]. Whereas a simple thesaurus would be able to identify any two words as synonyms or antonyms of one another, WordNet is able to note the similarity between two words that don’t have literally identical meanings. These relationships are ideally meant to mirror the same lexical associations made by human cognition. WordNet’s arrangement is hierarchical, which is to say that certain terms are more closely related than others. Within WordNet, these relationships are displayed as “synsets,” clusters of terms that fork, like neurons or tree branches, from more specific to more and more diffuse associations (see Figure 1). If two words are found within one another’s synset tree, it stands to reason that these terms are, in some way, related, be it closely or distantly. As discussed in the next sub-section, these distances between two terms can be calculated, and assigned a value commensurate with their degree of semantic relatedness [7]. Figure 1: Model synset tree (by hyponym relation). The hierarchical arrangement inherent to WordNet provides one method of determining the relationship between two terms. If the synset tree of one term encompasses another term, it is simple enough to note how many synset jumps it takes to move from one to another. In Figure 1, a “Dalmatian” is a type of “dog,” which itself belongs to the subcategory of “domestic animals;” thus there are two tiers of associations between the concepts of “Dalmation” and “domestic animals.” Unfortunately, however, just how closely any two terms might be related is not a purely linear relationship. WordNet organizes related terms by their precise lexical entailment, such that nouns might be categorized as synonyms, hypernyms, hyponyms, holonyms and meronyms, as seen in Table 1. These possible entailments provide a rudimentary roadmap for all the ways in which two words might be related. Since WordNet attempts to map the cognitive associations automatically formed between words [16], a student’s evocation [36] of the holonym or hypernym of a given noun instead of the noun itself is more likely to form an associative recast of the original term. To put these abstract concepts into the same terms used to describe the original problem, if an instructor in a large lecture course in animal biology wanted to monitor how students had grappled with the (admittedly basic but obviously fundamental) concept “animal,” he or she would want to know not only that the students had deployed the literal term “animal,” but were also conversant in the other associated concepts found (in immensely abbreviated form) in Figure 1. This association is consistent with the pedagogical principle of recasting, and with the concept of “original” expression as defined here. Where an instructor wants students to know one thing - in this case, about “animals” that those students can deploy more diffuse and more concrete examples of animals would be the readiest evidence that they actually understand. Yet while this simple index displays just how any two terms might be related, all the possible relationships noted are not necessarily equal. Some relationships, like that between synonyms smile and grin, are obviously bound to be more strongly associated than that between mammal and dog. Following a method first noted by Yang & Powers [49], it is possible to install a series of weights that can best calculate the semantic distance between any two terms. This method in particular is useful because of all known methods, it bears the highest correspondence between its own distance calculations and the intuitions of actual human respondents (at 92.1 percent accuracy). Table 1: Possible lexical entailments for nouns in WordNet. 6.2 Analysis Method: Implemention. Determining the point of originality of a student’s blog post depends upon the manual input of a specific query term by the instructor. The term relates to a key course topic and manual input of the topic reinforces the pedagogical utility of the process. For the query term, the process generates a WordNet synset tree. Words within the tree are then compared to the body of words extracted from a student’s blog post. Where matches are found, a summation of distance calculations between the original query term and the matches is performed as follows: Let q be a query term supplied by the instructor. Then, let W = {w0 , w1 , ..., wn } be a set containing all synset word matches (w ) from the WordNet database for q. Let B = {b0 , b1 , ..., bn } be a set of all words composing a blog post by a particular student and let S = {s0 , s1 , ..., sn } be a set of stopwords, a list of common words in English usage (like “the” or “and”), to be omitted to speed up processing time. Then, M = {m0 , m1 , ..., mn }, the set of synset term matches found in a blog post for query term q can be defined as: (FORMULA_1). WordNet stores synset matches in a tree structure with q as the root node. Then, δ, the distance (depth) for any given synset match (m ∈ M ) from the root node (query term q) is defined as: (FORMULA_2). WordNet also supplies the lexical entailment of each synset term. Thus, t, the “word type” of any given synset term match m ∈ M , is defined as: (FORMULA_3). Then α, the weight of any given synset term match is calculated as: (FORMULA_4). The depth for any given synset term is multiplied by a constant value of 0.7, which reflects the diminished associations between two terms the farther separated they are along the synset tree. This value is selected because it corresponds with the calculation of distance between terms that yields the nearest match with human intuition [49]. Then, C, the cumulative originality score for a given query term q in a student’s blog post, can be defined as: (FORMULA_5). The point of originality for a particular course topic is in many cases defined by the presence of several related query terms, or in other words, the synset matches for those terms. By defining Q = {q0 , q1 , ..., qn } as the set of query terms supplied by the instructor at any one time, then P, the overall point of originality of a given student’s blog post for a particular course topic (defined by Q), is: (FORMULA_6). Finally, repeating the point of originality calculation (Equation 6) for each blog post written by a particular student, and plotting all instances of originality on a horizontal timeline, allows for an optimal instruction comprehension whereas the instructor can see recasts of a particular course topic (defined by Q) across the entire body of a student’s writing throughout a single course. Although this paper focuses on the analysis of blog posts as students’ writing examples, given some additional programming work, any electronic form of student writing could be made compatible with the tool for subsequent analysis. 6.3 Visualization for the Point of Originality. The timeline visualization, as seen in Figure 2, displays a horizontal timeline that represents the time interval for the writing activity of any student for the duration of a particular semester. The numbered components of Figure 2 correspond to the following features. 1. This drop-down menu allows the instructor to select which student’s writing samples are currently being displayed. 2. This is where query terms (Q) are input by the instructor. 3. This timeline displays the date/times of each of the students’ writing samples. Each marker is color-coded, from colder to warmer colors along the ROYGBIV spectrum, the higher the value of the point of originality (P ) score for any given writing sample. These color assignments present an intuitive way for the instructor to quickly recognize that the sample has been assigned a higher originality value. 4. If a writing sample marker is selected in the timeline window (see inset 3), the text of that writing sample is displayed here. This assortment of visualization options allows the point of originality calculation to be displayed in a number of intuitive ways: both within chronology (inset 3) and in context (inset 5). 7. CASE STUDY. This section reports on a case study that explores the capability of using the Point of Originality tool to assess the originality of student writing in a semester-long co-blogging activity. More specifically, the study focuses on correlating originality scores assigned to students’ blog posts to their activities in the blogosphere during the semester and the final grades assigned to a term paper covering the same topics. Although primarily aimed at testing the validity of the point of originality method, this study models a likely use case. By demonstrating how low point of originality values correspond to poor performance in other aspects of the course, the Point of Originality tool could provide instructors with an early, near-instantaneous diagnostic of which students might require additional help. The tool might thus ideally streamline the process of conducting targeted pedagogical adjustments or interventions. The co-blogging data was collected from a course taught in the Fall of 2008 in the Computer Science Department at Brandeis University. The course is an introductory course, an elective, focused on exposing students to topics such as the social life of information, virtual communities, privacy, intellectual property and peer-to-peer computing. In the co-blogging activity, each student has a blog where he or she writes opinions on the course readings. Students can read each other’s posts and comment on the posts of their peers. The blogosphere, provides several features focused on increasing students’ awareness of recent activity, and enabling them to find interesting blog posts to read and conversations in which to participate. 7.1 Participants. There were 8 female and 17 male students, all undergraduates, enrolled in the class. There were 3 science majors and 1 science minor in the class. There were 12 students majoring in the social sciences and 8 minoring in the social sciences. The remainder of the class was either in the humanities or fine arts. Three students were omitted from the data set because they did not begin blogging until the end of the semester following a warning from the instructor. As an introductory course, open to non-majors, the technical requirements for enrollment were few. No formal evaluations were done to assess the students’ computer literacy or prior domain knowledge. In class discussions, most of the students expressed moderate or advanced technical skills. The instructor and teaching assistant did not design, or implement, the co-blogging activity in such a way that it pre-assigned students into particular authoring roles in the online blogosphere, thus potentially influencing the students choice of writing topics or styles. Figure 2: The Point of Originality timeline visualization. 7.2 Procedure. At the beginning of the semester, an in-class tour and exercise introduced the students to the important features of the co-blogging environment. The students were required to blog at the pace of one post per lecture: there were two lectures per week. A typical post was 1 or 2 paragraphs in length. The students were also required to read and comment on other contributions to the blogosphere. The coblogging work of each student counted for 35% of the final grade. During the semester, the students read four books and wrote a paper on one of these books. The focus of the analysis presented in this paper is on the co-blogging work that the students did during the time the class read the book for which they wrote their papers. 7.3 Metrics. Lectures were presented using slides that summarized the key points of the presentation. At the beginning of each lecture, hard copies of the slides were handed out to support student note taking. We used the lecture slides as a basis for identifying the inputs to the blogosphere. For each set of slides, a set of key topics that were covered by the lecture ultimately became the query terms used for analysis. 7.4 Method. All of the students’ online work was automatically recorded in a transcript and analyzed using the Point Originality tool. Originality scores were generated for all blog posts and papers, which were then correlated to students’ final paper grades and to statistical data summarizing their reading and writing activities during the co-blogging part of the semester. 8. RESULTS. The analysis was composed of two principle parts. The first part compared the degree to which the tool indicated the originality of the students’ blog posts and how well the originality scores related to the grades that the instructor assigned their papers. In the ideal situation, given that the instructor graded their papers based on how well the students expressed higher order understanding of the course material, or in other words their writing reflected original thought, the tool should provide scores where higher originality values would correspond to higher paper grades. The second part sought to explore to what degree the students’ interactivity in the blogosphere influenced their understanding of the course readings, and in what way their immersion in the co-blogging community positively or negatively impacted their levels of originality when writing papers. Ideally, students would find sufficient impetus to become deeply involved in the co-blogging learning community and their exposure to alternate or similar viewpoints of the same materials would help them develop their own viewpoints or to strengthen existing ones, thus leading to more original thought and better papers. Since the analysis was primarily concerned with ensuring that the tool could be used during a course to preemptively diagnose likely student success, the blog post dataset was filtered to only include blog posts written in what was defined as the “lead-in” period of co-blogging. During this period, the students were writing blog posts and comments on the topics that they eventually wrote their papers on, but at the time were unaware which specific topics they would have to address in those papers. The paper grades were assigned during the fall of 2008, roughly two years prior to the study described in this paper. Furthermore, grading was done by the course instructor, who is not a participant in the Point of Originality project. 8.1 Originality in the lead-in period. We began by collecting the originality scores calculated by our system for the blog posts written by each student on the paper topics and the actual grades that each student received for his or her paper. The average grade for student papers was 80.00 with a standard deviation of 16.83. The highest grade assigned was 95 and lowest was 40 on a scale from 0 to 100. The students’ blog posts received on average an originality score of 10.61 with a standard deviation of 4.29. The highest originality score assigned by our system was 18.30 whereas the lowest score was 3.92. Soon, a pattern emerged indicating that the more original the students’ co-blogging work, the higher the paper grades assigned by the instructor. While this is to be expected, the importance here is that the Point of Originality tool is automatically producing results that potentially correlate to standard approaches to pedagogy. A chi-square distribution test confirmed that there was indeed a positive correlation between the two factors. As students’ blog post originality scores increased, their final paper grades covering the same topics increased as well. In other words, as their blogging activity became more original, the students wrote better papers: (FORMULA_7). To further confirm the potential relationship between originality while initially learning the course materials (during the lead-in period) and how well that work transformed into mastery of course content as reflected by paper writing, students were divided into two groups based on their paper grades. Students whose paper received a grade above the average (80.00) were assigned to one group, the upper group, whereas students who scored below the average were assigned to the lower group. Table 2: Originality variance and paper grades for two different groups of students. As shown in Table 2, the students in the upper group received an average grade of 90.63 on their papers whereas the students in the lower group received an average grade of 66.79. What is more interesting, however, is what can be defined as the originality variance: the difference between how original the students’ blog posts were compared to their final papers. While the lower student group had an originality variance of 21.49, the variance for the students in the upper group was -6.10. Because the variance for the upper group is negative, those students’ blog posts, written "