Insights into “The behavior of tutoring systems” 

Analyzing the perspective of a practitioner vs. a research scientist

Among all the rapidly expanding facets of Computer Science, Artificial Intelligence or AI is probably the hottest area of research and development in the era we are living in. In this fast paced area of machine intelligence, sometimes even experts working in the very field get lost while trying to model something unique, something new. Building Intelligent Systems for Education has been in practice for quite a long time now(since the late 1960s), though it has not reached it’s extreme yet. More and more genius minds from AI fraternity are shifting towards this emerging field. However, unlike many of the other fields of Computer Science, this field is totally multidisciplinary in nature and often very hard to dive into a practical implementation stage for the same, even for the scientists working long in related research areas. In such a scenario, we find Kurt VanLehn’s(KVL) framework in “The behavior of tutoring systems” undoubtedly the most appropriate starting point for the people of almost all kinds of expert groups. His brilliant layout of the whole description makes it a piece of cake to understand the bigger picture of the whole area of “Intelligent Tutoring Systems(ITS)”. He primarily describes his work as a guide for both novice and experts, but we find the work equally well suited for bridging the gap between the theoretical research standpoint and practical implementation of such intelligent systems. It starts with a soft introduction which even a newbie would be able to follow without any trouble and gradually guides us through the design and implementation overviews of such systems. His descriptions are very well complimented with appropriate examples from few commercially and academically available ITS being used in practice from quite a long time. In this paper, we try to shed some lights on how the different portions of the KVL work help us understanding the core concepts of ITS while giving us enough information to start thinking about a practical implementation of such a system. However, we might not hope to be able to build a fully fledged ITS after reading the paper, which was never his intention of the work anyway, we get a very enthusiastic and structured direction of thoughts about bringing such systems into existence. We believe that it’s the most appropriate starting point for any mind diving into the sea of Intelligent Learning Environments. We summarize the comments about the different sections of the paper while showing the differences between the theoretical research scientists’ viewpoint and that of the practitioners’ in the field whenever applicable.

Core terminologies:

The paper starts with the explanations of some of the basic terminologies of ITS frameworks in a very intuitive way. The explanations are also well backed by relevant examples from some commercial ITS frameworks like Algebra Cognitive Tutor, Andes, AutoTutor, Sherlock, SQL-Tutor and Steve. These details serve as a base for understanding the rest of the paper and is very useful for anyone reading it. The theoretical research guy will find it useful to understand the rest of the theories while the practitioner guy will get an idea of what are the possible components he might probably have to design and develop. We give a very brief snapshot of the terms described here.

First it talks about the “Task Domain” and “Task” associated with the system. The former refers to the area of expertise the ITS wants it’s users to learn while the latter refers to the actual “chronological sequence” of actions that the learner has to perform on the ITS to achieve the modularized goals. Next two terms described are “Knowledge Component” and “Learning Events” which are very closely related in their meanings. The paper describes the subtle differences between the two very cleverly with appropriate examples along with the descriptions.

The last set of terms are the ones that probably serves as the spinal cord for the rest of the description. The paper describes the whole architecture as a combination of an “outer loop” and an “inner loop”. Very broadly, the outer loop iterates through the different “tasks” which the ITS wants to impose on its users. The inner loop is the iterator over the actual physical “steps” the user will perform per task assigned by the other loop. However, the inner loop is also responsible for few other things like intelligent feedback, student modeling and giving signals to outer loop for better selection of next set of tasks to be assigned. This kind of a description is so intuitive and easy to understand that even a layman can understand what’s going on inside an ITS. We feel that breaking the whole system in terms of these two loops clears up almost a major part of the whole system as to what we will be dealing with. The theoretical scientists will find it easier to map the upcoming descriptions to one of these loops while the practitioners would get a high level implementation overview. However, these notions of “inner loop” and “outer loop” has little practical usage and actual implementations will be much more complex than these simple loop structure, which the paper very humbly mentions as “In real tutoring systems, the inner loop may be more complex that the one shown in Table 2”.

Bigger Picture: The Outer Loop

The design of the outer loop decides how the system proceeds along with the learner. The main responsibility of this component is to cleverly select new problems to be presented to the learner at each step, which has to be very much learner specific in nature. The paper describes four major approaches which can serve as a baseline for designing any protocol to solve the problem by the research community. At the same time, the practitioner will find it very useful to start a basic implementation of the task selection module by using one of these basic strategies for the same, which he can later modify to incorporate many complex ways to handle it. The first two approach of task selection is very simple, the first one lets students select problems from a pool while the second one relies on the teacher model to assign the next problem in a “predetermined sequence”. The latter two approaches namely “Mastery Learning” and “Macroadaptive Learning” are somewhat more complex in nature and the description in the paper handles it very smoothly with appropriate examples. Finally, he concludes with some techniques to model the corpus of the problems which included “Human Authors” and “Problem Generators”.

Internal Dynamics: The Inner Loop

The rest of the description talks elaborately about the internal dynamics of a task which he refers to as the “inner loop”. It’s a hypothetical concept which embraces most the complexities of an ITS in terms of a simple loop that keeps running unless the learner successfully completes a particular task. The whole idea of the design of different aspects of this module is to keep students engaged in the most effective ways and reducing the possibilities of potential misbehaviors by the learners while executing the steps. It’s a natural tendency of one group of learners to abuse the facilities given to them and hence the design aspects of the inner loop are very crucial.

Among the most common services of the inner loop, giving minimal feedback is one of the most common ones. The feedback may be anything from correct, incorrect and non-optimal or unrecognized solutions. Hints are the next set of important component of an ITS. We should very clearly define the aspects of when, what and how to give the hints to the learner. The differences between all the 6 tutoring systems are explained with respect to these parameters and we get a fairly clear view of how to go about the whole hint generation mechanism after reading this section. The notion of “Help Abuse” and “Help Refusal” are well described. One of the major issues we encounter while devising a hint generation mechanism is the exact mapping of the thought process of the student model and what the hint module thinks about it. Sometimes the student might do the same problem in a correct but a different manner which ITS might reflect back as a wrong solution or sometimes student makes incorrect steps but receives minimal negative feedback. This is a very crucial issue in this regard and the author has given some nice explanations about the same and also ways to handle this. One of the ways to handle such hint mismanagement was implemented in Andes, where it asks questions back to students whenever the ITS is unsure of some step made by them. The students then answer back the ITS with some menu based fixed answers which helped Andes to narrow down on a decision about the step. The last aspect of the hint generator is the ways to generate hints. They are normally given in sequences, starting with very basic hints, it goes on giving till bottom-out hints. However, there are many other internal complications which he explains beautifully in the paper and gives pretty much everything we need to start exploring more on the hint mechanisms.

Error-specific feedbacks are yet another important area to concentrate on. Though the author has not described the insights of the “black box” error recognition system(step analyzer), it does talk about how to provide feedbacks to the learner based on the “error description” provided by the black box. The basic idea behind almost all kind of such module is to drive the students towards a “self-debugging” mode. He also describes as to when is it appropriate to present the feedback and analyzes different combinations of implementation which serves as a baseline for the practitioner to choose a design from.

One important part of all the ITS frameworks, thought not related to the above aspects, is the student modeling. This assessment of the student helps the system decide the next course of actions to be taken in order to achieve a fruitful learning experience. The author talks briefly about the coarse-grained assessments and elaborates on fine-grained assessments. He beautifully models the assessments matrices in terms of probabilities of both successful applications of learning components and in terms of failures. The details are described with example and comes very handy while designing our own tutoring system’s student model. The work also points out the common issues and questions related to the same. However, he doesn't give any pointers on how to handle those issues and we may find it little incomplete in it’s part. Finally he concludes with some discussions on delayed feedback mechanisms and on reviewing the whole solution at once paradigm of hint mechanism.

Concluding Remarks:

In his final remarks, he humbly mentions about all the topics that he missed in the literature like Evaluation of tutoring systems, Role of instructors in the process, about Step Generators and Step Analyzers, User interfaces etc. Among everything else, we feel that Evaluation of ITS frameworks is one thing which the author should have considered including in the work as we feel that it is one of the  most inevitable topics of any Artificially Intelligent System. Similarly, as we already mentioned earlier, a little more details into the black boxes of Step Analyzer and Step Generator would have also been nice. For the research fraternity it may not be of a much big of a deal as they will be reading dozens of other literatures which will cover those topics definitely. But from the point of view of the practitioner, it may become a bottleneck at times if he solely relies on this work as a starting point. However, it will be very unfair to judge the marvelous work in KVL framework just because it lacks one or two topics when it has so much more to offer already on its part. Finally, as a concluding remark, we want to say that keeping the small glitches aside, the KVL framework is undoubtedly one of the best literature in the field and we certainly recommend it to any individual irrespective of his field of interest or expertise, to have a read of the same if he or she is interested in ITS frameworks.

NB: The article is a critique on Vanlehn, K. (2006). The behavior of tutoring systems. International journal of artificial intelligence in education, 16(3), 227-265.