1. Table 1 : Monthly data summary. Table 2 : Cluster ranks and relative size (%). 4 Algorithms and Feature Analysis. Clustering algorithms are commonly used to find patterns within large data sets [2, 6, 7] and two clustering algorithms were chosen to study the initial data set. First, the K-means clustering algorithm was used. K-means is an unsupervised learning, iterative descent algorithm that partitions n data observations into K clusters. Each cluster is assigned a cen- troid and cluster membership is determined by minimizing the distance from each cluster member and the centroid. The second algorithm in the initial experiment was the Expec- tation-Maximization (EM) algorithm [4]. EM is a model-based iterative algorithm that examines data observations and represents each cluster as a probability distribution. Giv- en n data observations, EM maximizes the likelihood of the observed distributions by es- timating the means and standard deviations of each cluster. Neither algorithm is without flaws and our experimental results in section 6 show this. K- means primary weakness is that the number of clusters must be determined a priori. This weakness, however, is inherent in many partitional clustering algorithms and may require an experimentally selected n. Another weakness is that K-means is sensitive to outliers – data that are distant from the centroid may pull the centroid away from the real centroid in a given data set. Finally, it is difficult to understanding which feature contributes more to cluster membership, since every feature is assumed to have the same weights. EM‟s core weaknesses are the relative speed with which the clusters converge, and the possibil- ity of convergence at the boundary of a cluster. 5 Experiments. Two experiments were performed on the initial feature set with a few variations for com- parison. The first experiment was designed to run the K-means algorithm with n = 12 and the Euclidean distance function, referred to as K12. The n for this initial experiment was derived from the total number of sessions analyzed over the period (~1,400) divided by the number of users invited to participate (~120). This provides a baseline for comparison with the other algorithms. For comparison, the expectation maximization algorithm was chosen for the remainder of the experiments. EM was first chosen to automatically select n clusters using cross validation, referred to as EM*. This provided a baseline to compare the algorithm's performance against the K-means algorithm (K12). The last experiment was run using EM again with a fixed cluster size of 12, referred to as EM12. 6 Evaluation and Results. Table 2 shows each of the algorithms and the sizes of the largest clusters they produced. EM* produced 10 clusters, the top 4 of which represent 84% of all the data. Similarly, for EM12, the top 5 largest clusters represent 84% of the data. Finally, K12 shows a similar trend, with the top 5 clusters representing 87% of the data. For the purposes of evaluation we consider the top 4 clusters in EM*, top 5 in EM12, and the top 6 clusters in K12, since the K12 distribution of clusters was more sparse. Table 3 shows the features with the greatest means of the top clusters for each algorithm. The cluster labels represent UI features for example, A1 represents clicks on the Interac- tive Resources tab, A2 the Shared Stuff for This Concept tab, A6 for the Embedded As- sessments toggle element, A12 for the Images/Visuals tab, and so on. The top features of the largest cluster in EM* (A1, A2 and A4) correspond to CCS UI tab clicks on Interac- tive Resources, Shared Stuff for This Concept and Shared Stuff for This Activity. This top cluster suggests a pattern of activity that is focused on both CCS-suggested interactive resources and shared resources that others have saved, which may indicate the impor- tance of what others have saved as well the automatically generated interactive resource list. The EM12 and K12 algorithms indicate very similar patterns. For example, EM12‟s largest cluster shows the exact same pattern as EM*. Similarly, K12 shows A1, A3 and A4 as its top features. Examining other clusters show cluster 4 of K12 and cluster 11 of EM12 share similar patterns over features A3, A6, A8 and A11. This pattern corresponds to clicks on Instructional Support Materials, Embedded Assessments, Answers and Teaching Tips system areas. Similarly, cluster 2 of K12 and cluster 1 of EM12 share sim- ilar features along A3, A7 and A14, which correspond to the Instructional Support Mate- rials and Activities tabs, suggesting time being spent on preparing or reviewing student activities and corresponding materials. Table 3 : Top cluster features and their cluster membership. Each cluster algorithm revealed data that was consistent with overall system use seen in the server logs, though the smaller cluster sizes show greater differentiation of features. That there was not complete agreement in cluster features or sizes, however, may indicate more experiments are required. 7 Related Work. Much of the work here has been influenced by the body of work in web use analytics, which break down into two categories : (1) content analytics and (2) usage analytics [12]. This work is focused on usage analytics. Broadly, use analytics aims at understanding the aggregate activity and use patterns of a website primarily using advanced server log anal- ysis. Such analytics often aim at understanding aspects of the site that are popular, con- tent that seems to be frequently accessed, times of frequent/infrequent use, etc. with the goal of developing a sense of where the site could be improved or enhanced for optimal performance, increased advertisement penetration or site content enhancement through recommender techniques [13]. Such use analytics are invaluable for developing site con- tent, but also useful in developing models of user behavior. Website session characteris- tics are commonly studied to determine how users are accessing the site and statistical techniques are used to determine tasks being performed within a website, revealing clus- ter usage patterns in the ways we have discussed here. Markov models have been used to derive the meaning of certain behaviors within a session by observing page transitions and their probabilities to develop behavioral models of use [5]. Work has also been done to connect page semantics to web usage, for example [9] use Probabilistic Latent Seman- tic Analysis to determine if the content and subsequent usage of a page implies an under- lying task. Finally, user interface event mining [8] aims at developing techniques to ex- ploit detailed user experience and interaction data. 8 Discussion. The initial experiments presented in this paper offer some insights into the planning be- havior of teachers online. However, two areas of improvement can be immediately dis- cussed : improved feature selection and expanded algorithm experiments and compari- son. The initial experimental feature set provides interesting insights into the behavior of teachers for the visual components selected in this initial observation. However, the flat- tened visual hierarchy of the CCS interface only provides a convenient way to discretize each visual element of the system without advancing the notion of the semantic structure of this hierarchy. For example, while it is clear that the Interactive Resources tab of the interface was widely used, there are substructures under that tab which also contain wide- ly used features. The current feature set is not capable of capturing this hierarchy or its implied semantic structure, though considering it might yield new insights into the se- mantics of the features commonly accessed by users. Further extensions to the feature set might also include adding link-to-link features, for example, exploring high frequency transitions might reveal unique relationships between UI features and functionality. The EM and K-means algorithms are commonly used in data mining, and while some clusters in K12 and EM12 had similar characteristics, all of the top clusters were not sim- ilar enough to say both algorithms were converging on exactly the same feature sets. This may underscore the differences in each algorithm or in the way they each treat the fea- tures. It may also reinforce the effects parameter sensitivity (e.g. n clusters) and feature selections have on the results. The focus of the next round of experiments will be to expe- riment further with EM and K-means parameters, and also to expand algorithm coverage to hierarchical-based algorithms. Such experimentation may also fit well with the seman- tic features already suggested and allow a comparison of the hierarchies that are produced from a semantic-based structure with the clusters already observed. As with all learning algorithms, it is challenging to determining if the experimental data would be predicted by and hold up to some gold standard or human expert evaluation. Determining if the discovered behaviors match the observed data in practice is difficult and further research is underway to study actual and reported system use through on-site observation and survey instruments, which should lead to a higher fidelity confirmation of the patterns discovered thus far.