"1. Introduction. Phil Long and George Siemens stated [11] that “the most dramatic factor shaping the future of higher education is something that we can’t actually touch or see: big data and analytics.” In other words if a lot of learning data is measured and analyzed, new insights into the learning process will be given and therefore new didactical approaches will help to improve our learning behaviors – not only in higher education but in the whole educational system. Of course the main question addressed by Erik Duval [2] is about what exactly should be measured to get a deeper understanding of how learning takes place. In this publication we like to answer the following research question: “How can we improve learning the multiplication table with the help of an individualized learning program and a followup data analyzing?” First of all, it must be mentioned that learning the multiplication table is a central subject of mathematics in primary school. Young children gradually develop an intuitive and practical arithmetic that they use to solve problems successfully and confidently in their everyday life. But each child is different and some of the basic knowledge about mathematics is acquired even before children start school [8] [14]. Considering current didactical knowledge about learning math tells us, that there are cultural differences, some obviously based on language implications [3]. Each child needs a general linguistic competence before starting to operate with numbers. Some publications point out that mathematic is the first non-native language [9]. For example, understanding the simple expression “multiply” is implicitly not only a mathematical problem but also a linguistic one. Many children who carry out the algorithms correctly do not really understand reasons for crucial aspects of the procedure [6]. The most common and traditional way to learn the multiplication table is drill and practice without taking care of any linguistic preconditions. “Multiplication Tables in 21 days” [12] is one of many offers of teach the kids online or to use traditional materials. From the perspective of special education it is well-known that this way of teaching and learning math is problematic and leads to disorders and blockades on the learner’s side. The “defective number module hypothesis” indicates a genetic defect, a more physiological cause of basic brain functionality [1]. Teachers gladly accepted such a biological explanation as a relief for their problems with a child. According to current neurological oriented research results, but also based on a very old educational tradition, teaching should use tactile, optical and acoustic processing methods in every single case – intensively when they observe deviations from the mainstream. It is important to allow multisensory math to be taught to children with special educational needs. Kendall [7] offers some practical suggestions of how to achieve this. Furthermore face-to-face teaching has to bear in mind also interlinking between different assignments. But the daily practice is dominated by pure “row learning” of the multiplication table [5]. It has to be borne in mind that learning per se is an active individual process on the learners themselves. The competence “perfect handling the multiplication table” differs enormously in the needed learning time and resulting learning effort – from weeks to years. It is obvious that the simple problem “learning the multiplication table” is not as trivial as it seems to be at first glance. The goal of our research is to develop a web-based system that assists learners as well as teachers. Furthermore it should provide teachers with an overview about the current learning process of their pupils. The research group (consisting of educators, e-learning experts, an educational scientist and IT-developers) specified their requirements as follows: - The system should estimate the competence grade of the learner. - The system should provide appropriate exercises according to the competence grade of the learner. Nevertheless the exercises should tend to be challenging. - The system should ensure that already well-done exercises are repeated and practiced. After succeeding a problem the probability for a repeated display decreases in two levels (analogous to a Leitner System [10]). - In general the system should be motivating and show that learning can be fun. - The system should record and safe fine-grained data of all done exercises, test results and the current competence grade of the learner in order to prepare the next sessions in an adequate way. - Thereby it should unburden the teachers from that task. In contrast to many other approaches to adopt learners in this research work there are no probabilistic strategies for estimating competencies used. Even more the goal is to generate a complete table to inform learners as well as teachers about their problems in every single task. And for sure teachers want to know something about the process, the used time and the up and down of the learning curves. Similar research work in the area of Intelligent Tutoring Systems (ITS) [4] [13] pointed out that such an approach can raise many problems for daily-life-situations of an actual classroom. In other words, our research work concentrates on a particular and small learning problem, but likes to ensure the learning success. Finally, the implemented intelligent algorithm is not only useable for the specific problem of learning the multiplication table. Any problem that is presented in a set of flashcards could be easily transferred to a system with this functionality. 2. Technical details about the application. Beforehand some technical details need to be mentioned that were defined by the project team: The application is written in PHP1 5. The interface is based on HTML2 and CSS3. Because of the different render engines of the browsers the CSS-Framework YAML4 is used. Carrying out a solid software development the application is programmed with the help of Zend Framework5. The main approach is to store the data of each learner to analyze their learning process. 3. Algorithm. The approach of this research study is to develop an algorithm that allows an individual learning by providing exercises in dependence on the learner’s knowledge. Therefore the main task is to consider which exercise is provided next. On the one side, the exercise should be challenging on the other side it must be adequate; not too difficult but not too easy at all. Therefore the core task of the program is the new designed algorithm. This algorithm decides which question is chosen to be next. The general idea is to always find the “best learnable” question in reference to Vygotskys [15] idea of ""the zone of proximal development"" for the user. Therefore two parameters are important: - Difficulty of the exercise: First of all each learning exercise has to get a difficulty rank. If the task difficulties are already known from a preliminary investigation (for example a pedagogical study), they should be used in that way. Otherwise a hierarchy of estimated difficulties must be assumed according to specific pedagogical experiences. The difficulty ranks should then be converted to a scale with values between 0 and 1. This means an easy question gets a high success probability value (close to 1) and a more difficult one gets a low success probability value (close to 0). - Degree of competence: To keep track of the learning progress of a user the degree of competence is calculated (0 ... 1), called learning rate. The calculation of this value depends on the performance of each learner and is calculated in real time. - When we started developing this program we used “degree of competence” in the sense of ratio of items, which have been solved to the sum of items which has been offered. During the further process we found out that it is more useful to have a less volatile variable for determining the level of the next question and two others (learning rate 1 & 2) for describing the success and analyses. 3.1 Degree of competence. The basic idea of a degree of competence (DOC) is to indicate which question is difficult and which is easy in relation to the current user´s knowledge. It must be considered, that for the first session we don’t know anything about the competence level of the student so far. With every answered question we get a more and more reliable estimation of the real competence. The main idea is that each user has their specific degree of competence equaling a value between 0 and 1. 1 means the user is advanced and every item could be selected as a suitable task or question, 0.5 indicates an intermediate user and 0 is a beginner. In general the degree of competence reflects the current knowledge. Questions graded “below” the individual degree of competence are solved and known by the learner; questions “above” are questions to be learned. When the next learning item is chosen it must be avoided that the task is neither too difficult (learners will get frustrated) nor too easy (learners will become bored). To fulfill that demand we have to assess the questions given to be homogenous compared to traditional tasks. The amount of data collected by the application helps us to discuss for example whether this homogeneity is really given, and whether the students achieve the learning goals in typical, well known and logical or different and more individualized learning paths. We use the idea of a so-called “extended learning area” to describe the pool of which learners get the next tasks from. By definition the whole learning area is a combination of the actual learning area (known by the learner) and the extended learning area (unknown by the learner). In our pilot study we defined the extended learning area to be 25% above the learner’s degree of competence. Of course this parameter can be adjusted by teachers. In a situation where the teacher anticipates a low level of competence in this domain, at the beginning of learning the tables, this value must be kept down. Over the time, for example when learners are starting to learn the multi-digit multiplication, it could be become higher. Figure 1 visualizes the whole learning area sectioned by the degree of competence into an area of known questions (actual learning area) and one of future questions (extended learning area) limited by the extended degree of competence. Figure 1 Degree of competences. In the initial test phase, we found out that these settings are complicated, especially during the first session when the students have only solved some few problems. A computer program is restricted in perceiving students’ interaction only towards solved or unsolved learning questions. A teacher perceives of course, many other and different signals (nervousness, facial expression, comments on the task …) that may affect the decision which task is to be given next. With other words especially an improper prea-ssessment of learners’ knowledge can be frustrating by keeping learners too long overstrained or bored. Furthermore it must be avoided, that the tasks’ difficulty escalates too quickly and questions therefore become inadequate. In the following chapter the design of the pretest is pointed out. 3.2 Pretest. At the beginning, before the learning session, each learner will be asked two questions, starting with a moderate one; if the answer is correct; a more difficult one will be presented - the estimated degree of competence is set around 0,75. If the answer is incorrect an easier one will be provided (around 0,25). Figure 2 points out the flow diagram to get the estimated degree of competence in the beginning. For example if the first answer is correct and the second one false, the estimated learning rate results 0,50. Finally to get adequate questions (easy – medium – difficult) the following categories are defined - Easy: A question with a probability for a correct answer of 0,78 – 0,69 - Medium: A question with a probability success value of 0,54 – 0,47 - Difficult: A question with a probability success value of 0,20 – 0,13 Figure 2 Pretest – estimated degree of competence. 3.3 Answer classification. Another issue that must be addressed in our context is the classification of well-known learning problems. From a pedagogical point of view it must be stated that presenting a right solution is not a satisfactory indicator. Especially learning the multiplication table is learning by drill & practice which means if a learner practices one problem more often they will get more confident that they can manage this item. Therefore also the developed algorithm must ensure that a task is “well-known”. This issue is addressed by establishing a “well-known” parameter. The answers of the learners are marked with 0, 1 or 2: - 0 means a wrong answer - 1 shows that the user knew the correct answer once - 2 indicates that the student had two consecutive correct answers (this means a question is “well known”) Questions that have been signed with 2 were set back to 0 immediately in case the student failed. After the pilot stage it was decided to set the “well-known” parameter always to 2 when the degree of competence of the learner is 0,3 higher than the difficulty of the task. We wanted to reduce the frequency of getting too easy tasks for good students. Each time a student solves a problem, the result will be stored by the application. We calculate a “learning rate 2” to be the ratio of the sum of solved items signed with 2 and the number of items. We also compute a “learning rate 1”. This ratio is based on the sum of all solved items, e.g. signed with 1 or 2. It is to remark, that these two levels of knowledge and the set procedures have the functionality of a Leitner System [10]. The results of psychology of memory give the recommendation that learned tasks should be used again and again, but there is only a decreasing frequency necessary to avoid forgetting the right solutions. 3.4 Adjusting competence. In the first session we only have the results of the pretest to estimate the competence of the respective student and to allocate a suitable question. With every new test we get more and better information for specifying the student’s current competence. At the beginning of our tests we simply calculated the ratio of the number of correct answers to all given answers. This leads to the following formula: (FORMULA_1). This formula resulted in a too high volatility at the beginning of the learning progress. For example if a learner answers the first 2 questions in the pretest correctly the program will assume a degree of competence of 0,75. After answering the next 2 questions wrong the degree of competence decreases to 0,5. The fast increase/decrease of the factor of course will cause that learners become quickly frustrated. A second approach with the intention of getting a more stable value was to calculate the degree of competence by counting answers only with the classification “well-known” (2). The resulting formula is: (FORMULA_2). number of items = number of learning problems/flashcards provided by the system. This formula has some disadvantages as well, especially for learners with a too low estimated degree of competence resulting from the pretest. For these learners it might be hard to increase their degree of competence, i.e. to correct the erroneous data. Finally the degree of competence is defined by the procedure as follows: We compute the ratio of both - number of correct answers as well as well-known answers - and the number of items. (FORMULA_3). The results are saved in the database. In general learning progress is considered as completed, if every question is signed with 2 (well-known). When a learner starts their first session the following procedure combines the result of the pretest and the actual degree of competence: This special procedure is only used at the beginning if (FORMULA_4). Then we compute a “weight”:. (FORMULA_5). With this weight, the degree of competence is calculated as a combination of the estimated degree of competence of the pretest and the previous performance: (FORMULA_6). This degree of competence is important to find the next task in the learning area. Therefore it must be guaranteed that the DOC is not in-/decreasing too fast in order to provide adequate questions. 3.5 Selecting items. After the degree of competence is calculated the next item presented to the learner must be chosen. For this purpose, the algorithm chooses items out of three categories. - Extended and Actual Learning Area (items signed with 0) - Actual Learning Area (items signed with 1) - Actual Learning Area (items signed with 2) A random number out of the interval 0 and 1 is used to decide which category is activated. Therefore three cases are defined: - Case 1: If the random number (x) is smaller or equal 0,05 a well-known question marked with 2 is chosen. - Case 2: If the random number is 0,05 > x >= 0,15 a known question marked with 1 is chosen. - Case 3: If the random number is x > 0,15 a (unknown) question out of the extended and actual learning area is chosen. In general all items are preordered according to their difficulties and have corresponding rank numbers. In case 1 another random number is drawn to get a position in the ranking of the known items. Beginning with this position the algorithm is looking for the next less difficult item, which is signed with 2. In case 2 the algorithm is starting with the rank of the easiest item and is looking for the next item, which is signed with 1. In case 3 our selection is adjusted with the degree of competence. We compute an upper rank for the selection: (FORMULA_7). Afterwards the algorithm is starting to search beginning with rank position rank or less difficult items. The next item, which is signed with 0, is selected. In the case the algorithm is running out of the ranking, because no suitable item could be found, the procedure is repeated again and again until it finds an item. We are sure that an item is found, because the program selects already known problems for repetition with a p=15%. If all items are signed with 2 the student will receive this information together with an option to finish. The student may proceed with the training if preferred so. 4. Algorithm overview. Finally a short overview of the workflow (algorithm) is given:. 1. Pretest (see section 3.2, figure 2) 2. Calculate the degree of competence (see 3.4) 3. Choose next question according to the degree of competence (See section 3.5). For this, the probability to choose a question out of the three classes is (could be adjusted): o 75% learning area o 10% known questions o 5% well-known questions 4. Process the given answer (See section 3.2) 5. Calculate new degree of competence (See section 3.4) 6. Check if learning progress is finished. If yes output a message to the user otherwise go to step 3. Repeat the algorithm until the user stops playing. 5. Prototype. The program is implemented as a game. Learners can earn points for each correct answer given and reach new stages (called rank). Points are gathered or can be lost if the answer given was wrong. Furthermore the answering speed is also important. The quicker a question is answered correctly the more points are gained. Finally the game requites also consecutive correct answers by giving more points. The prototype is a web application currently available following the URL (last visited October 2011): http://vlpc01.tugraz.ac.at/~georg/index.php/user/login. After registering an account learners are able to login. Depending on the learner’s user type they are redirected to the according page. The learners will get the main game interface (see figure 3). Administrators will see an additional navigation bar, which allows choosing between getting statistical information about all existing learners and simply playing the game. Due to the fact that it is a first prototype the interface is kept simple with a first design suggestion. Figure 3 Main screen of the application.. Figure 3 displays the main game interface (currently only a German version is running). The markers 1 to 6 refer to the most important areas of the interface: 1. The questions which have to be answered (item of the multiplication table) 2. Free space for feedback to the user, for example “Correct answer”, “Wrong answer” or “Not entered a number” 3. Input field for providing the answer. It can be submitted by clicking on the green button “Antworten” or pressing “Enter” on the keyboard. 4. The timeline shows the remaining time for giving an answer. Actually the learner has 60 seconds to answer (predefined by the teacher). Depending on the answering speed points are calculated. 5. Shows the actual rank of the user. 6. Displays the actual points of the learner. Of course the more points a learner gets, the higher is their rank. 6. Study. After finishing the web application the first research study was carried out at a primary school in Austria. In summer semester 2011 the program was handed out to 42 pupils of the primary school Laubegg6 (age: 9-10). After a short introduction of the main principles of the program and setting personal accounts for each learner the study was started. It lasted at least 4 weeks. Some of the learners ignored this time restriction and played the game again and again over months. Learners learned on computers at the school as well as on their personal computers at home. It can be summarized that in the time frame of the study 12.926 answers where given which means that on average each learner answered 308 questions or in summary they did the whole multiplication table 3,4 times. Bearing in mind that there was no real pressure from teacher’s side using the program it is a considerable pleasant high number. Furthermore it can be stated that pupils seemed to enjoy using the application or at least got not bored. 7. Discussion. 7.1 General Overview. First of all an overview of the general result is given (figure 4) by displaying the number of trials versus the final number of wellknown items. In detail figure 4 shows that in summary 18 learners (43%) mastered every 90 single multiplications at least twice (to get marked 2) of the multiplication tables. Many of them obviously enjoyed the game: One of them solved even 1.486 assignments on voluntary basis. If learners make no mistakes, they need approximately 190 trials to show their competence for each item and get marked “well-known” (2). According to our defined algorithm and due to the fact that the probability of getting a correctly solved item twice in the very first beginning of the game is only about 10% the number of 190 trials seems to be a quite solid one. More than half of the learners are below the aspired 90 correct single multiplications of the multiplication tables. There might be several reasons for this: The learner - did not get used to / has problems with the interface - does not know the necessary operations; is not able to solve the learning problem correctly - misinterprets an assignment - is distracted by their environment and makes wrong clicks - is badly concentrated for several reasons. As mentioned before, six learners proceed to work with the program, even after the program offered to stop because every assignment had been answered correctly twice. In such cases, the program serves as a diagnostic instrument: It is more reliable than a paper-pencil test when a learner masters the multiplication table with this application. Figure 4 Well-known items versus trials. As mentioned before, more than half of the learners did not reach the 100 percent level. We already listed possible reasons why a learner does not solve the assignments correctly. Besides these arguments, it is necessary to reflect whether weaker learners could have fewer possibilities using the program, especially at home. The above-mentioned “high-trial” learner for example is known as someone who was able to train and use the program also at home. Eventually, the better students had more chances to work at the computers in classes, so that the teachers were able to work with weaker learners. Perhaps, learners with worse performances had general problems, like to open and to handle the program. Nevertheless, these are only further assumptions, which must be considered in future studies. 7.2 Demotivated learner (ID 21). One learner with a weak performance attracted our attention because of a very high number of trials (513) (figure 5). A detailed inspection showed that she/he did not work very intensively. In the first two tasks he/she failed, then eleven tasks were ok, her/his performance rose abruptly. However, afterwards, she/he continued approximately 400 times to wait the whole answering time without doing anything but asking for a new assignment. Figure 5 Unmotivated performance. 7.3 Motivated captain. Figure 6 shows the learning history of the most diligent learner. In the beginning, the assignments were solved correctly, then some mistakes occurred, afterwards a learning process can be recognized and finally with some occasional mistakes the learner works on a high performance level. Obviously, the learner was highly motivated to deal with the assignments given by the program: She/he absolutely wanted to be and stay the first in the high score. Furthermore there are an amazing high number of trials, which leads to the assumption that the learner likes to do the exercises. It seems that there is a dependence between the high number of trials and the almost perfect knowledge of the learning field. Figure 6 Motivated captain. Teachers have to face such situations. Why did the learner become demotivated? Why did he/she did not proceed going on with learning and gave up? This case is quite dramatic, because the learner her/himself disrupted the ongoing learning process. 7.4 The medium learner (ID 115). One of the most interesting learning histories can be seen in figure 7. It points out the way the application works if a learner is not performing very well. In the beginning, the learner made mistakes in every second assignment (0,5), followed by 7 mistakes consecutively. This is the reason for the big decrease (0,15). Afterwards, the learner gave a number of right answers and the rate of correct answers increased back to 0,5. In the following phase an up and down can be seen till a number of right consecutive answers helps to reach a level of 0,7. But then the number of mistakes rose again and the rate went down to about 0.5. This characteristic curve illustrates how a learner is learning from mistakes and is getting better by failing an assignment and slowly solving it next time - “learning by failing”. Figure 7 The medium learner. This effect can be seen more detailed in figure 8; it shows the learning rate 2 of the same student. As mentioned in previous chapters an item is marked as 2 if it was solved at least twice consecutively. In the initial phase the same items are not presented very often due to the fact that the probability is defined with 15% (see chapter 4) and because the program prefers assignments that were not correctly answered beforehand. Afterwards, the performance increases fast to a certain stage (20 items are marked as 2) and falls a little bit (“learning by failing”). The next 140 questions caused a so to say sideward trend; the student answered some questions right some wrong. Then the curve jumped up. The student has learned the solutions of a more difficult group of items, we do not know whether they got training beside the program or not. It needed about 120 questions more until the curve was growing again to foster the learning efforts. Finally we see that the last 15 assignments obviously lastingly cause some problems. It can be concluded that the learner needs just some additional time or other exercises and materials to perform the whole multiplication table perfectly. Figure 8 Unsteady and slow growing learning curve. Figure 9 – the learning rate 1 of the student – underlines the “learning by failing” effect. There were about 300 trials almost every problem is signed as known (one time) - but only 20 items are signed as well-known (2 times). Furthermore in about 20 percent of the items marked as “well-known” mistakes occurred (figure 8). Finally, the learner seems to be concentrated better or they learned the tables and solved 70 of the 90 items twice. Figure 9 Learning rate – the DOC. 7.5 The weak learner. Figure 10 illustrates the exercise list of one learner, who had performed easy questions with simple assignments, but got worse afterwards, when the difficulty of the questions jumped over their degree of competence. We call the effect the “fast increase effect”, which means the increase is too fast and not appropriate to the learner’s knowledge. Figure 10 ""Fast increase effect"". The learning rate curve of the same learner in figure 11 describes the same behavior: a fast increase followed by a very unstable phase and decreasing output. In comparison to the case before, the learner is only able to manage about 25 percent of all items correctly. It is assumable that the learner is not mastering the whole multiplication table. By all means, they work with the program, make mistakes, are corrected, learn and for sure need much more time to master all items finally. Figure 11 Learning rate of a weak learner. 7.6 Lazy bones. Figure 12 illustrates the learning curve of lazy learners because of the very little activity in relation to the competence. Of about 100 problems 80% were solved. We need more precise observation, what is going on there. Is it a problem, that these students need an additional (social) motivation? The process is very slow and the performance is not good in this test situation. Could some other problems be the cause for this lack? Figure 10 A typical learning rate for learners with low trials. 7.7 General remarks. We think it is useful to test and to train the students with such programs. They have possibilities to get “out of the stream” with provocations to learn, which is the same in a normal class. But here we have a protocol of learning progress. In every case the delivered tasks are more frequent and more precisely orientated on the individual level and next reachable tasks than in a standard situation, with a teacher without such an information-processing capacity tool. 8. Conclusion and future work. In this publication we discuss the implementation of a new and intelligent algorithm to assist school children training one of the basic learning goals in primary schools – the multiplication table. Furthermore a field study with 42 learners was carried out and analyzed. The power of learning analytics allows the research team to think about the outcomes and carries out different types of learning curves. In general we can state that there are some major types of curves – learners who are very knowledgeable, those who are in a stage close to being knowledgeable and those who still need a lot of learning effort to reach the learning goals. However it is recognized that analyzing just one curve (correct answers/total number of process items) is not sufficient enough to cover the state of the learning process in most of the provided cases. There is a need to have a look at learning rate 1 and 2 too. Moreover sometimes the curves gave the researchers and teachers only a hint that a pedagogical intervention is absolutely necessary to enhance the learning results. For our future work we consider a couple of ideas how the current application can be improved, but mainly it must be stated that there should be a much closer look at the learners. This can perhaps be done in some more intelligent analyses of the data or in more cooperation with the teacher. We will have to analyze the reasons why some learners are demotivated or why their learning rate decreases during longer intervals. We also discussed whether it could be more appropriate to change the design from gaming to a more informative display of the actual knowledge. This could be a matrix, which displays the results with the 0,1 or 2 signs. In the future we also need a better aggregation, compression and visualization of the learning outcomes. This could indicate those people, who need more attention from the teacher and probably immediate interventions. Another point is that we measured the time to solve the questions only to limit the time "