«Abstract A number of prior studies have found that using animation to help teach algorithms had less bene cial e ects on learning than hoped. Those ...»
Rethinking the Evaluation of Algorithm Animations as
Learning Aids: An Observational Study
Graphics, Visualization, and Usability Center
College of Computing
Georgia Institute of Technology
Atlanta, GA 30332-0280
Technical Report GIT-GVU-99-10
A number of prior studies have found that using animation to help teach algorithms had
less bene cial e ects on learning than hoped. Those results surprise many computer science instructors whose intuition leads them to believe that algorithm animations should assist instruction. This article reports on a study in which animation is utilized in more of a homework" learning scenario rather than a nal exam" scenario. Our focus is on understanding how learners will utilize animation and other instructional materials in trying to understand a new algorithm, and on gaining insight into how animations can t into successful learning strategies. The study indicates that students use sophisticated combinations of instructional materials in learning scenarios. In particular, the presence of algorithm animations seems to make a challenging algorithm more accessible and less intimidating, thus leading to enhanced student interaction with the materials and facilitating learning.
Keywords: Algorithm animation, evaluation, empirical study, learning, software visualization 1 Introduction There is something di cult about understanding and analyzing algorithms; ask any computer science student. What that something" is and how to reduce the di culty" are two problems whose solutions are anxiously awaited by many students and instructors.
Meanwhile, guided mainly by intuition, instructors have been looking toward algorithm animation Bro88 as a tool to help their students learn. It is certainly possible to learn about an algorithm without using an animation, but to many it seems almost obvious that students could learn faster and more thoroughly with one: The dynamic, symbolic images in an algorithm animation help provide a concrete appearance to the
notions of algorithm methodologies, thus making them more explicit and clear. In addition, students using animations report that they feel the animations assist them in understanding an algorithm SBL93. Imagine the surprise of students and instructors when empirical research about the bene ts of algorithm animation began to show disappointing results SBL93, BCS96.
This article is a further step in our examination of the e ects of animation on learning about computer algorithms and programs. We are motivated by the disappointing, mixed results of prior studies and a general curiousity about both how and why animation and multimedia technologies can assist instruction. Much prior research in this area has focused on algorithm animation technologies. This work continues our e orts on analytical, cognitive aspects of the domain. Mayer provides a tting motivational prologue to our
At this time, the technology for multimedia education is developing at a faster pace that a corresponding science of how people learn in multimedia environments. Technological advances in computer-based graphics|including animation| and text-based graphics|including the use of animations|have not been matched by corresponding scienti c advances in understanding how people learn from pictures and words." May97, p. 4 To begin, we shall brie y review a few of the prior empirical studies of algorithm animation that have provided mixed results. A study conducted by Stasko, Badre, and Lewis in 1993 used an interactive animation to teach a complicated algorithm to computer science graduate students SBL93. Their results showed a non-signi cant trend favoring the animation group" in scores on a post-test used to evaluate understanding. The study hypothesized that the animation would aid procedural understanding, but the animation group did not perform any better than the control group on questions testing procedural knowledge. The authors attribute the lack of a performance advantage with animation to a property of most visualizations that they represent an expert's understanding of the algorithm, not a novice's. For a student to bene t from the animation, the student must understand both the mapping from the algorithm to the graphics and the underlying algorithm on which it is based.... Students just learning about an algorithm do not have a foundation of understanding upon which to construct the visualization mapping."
A more recent study conducted by Byrne, Catrambone, and Stasko BCS also found limited e ects for undergraduates using interactive animations BCS96. The study examined the relation of animation to evoking predictions in students. In learning new algorithms, some students viewed animations and some were prompted to make predictions about an algorithm's operation on novel data sets. For a simple algorithm, the use of animation and or prediction was bene cial on challenging questions, as measured on a post-test. For a more complex algorithm, however, animation and or prediction provided no signi cant bene t.
Not all algorithm animation studies have had disappointing results, however. Lawrence's dissertation research uncovered a variety of results, but one particular experiment showed a positive bene t to the use of animations in after-class laboratory sessions when students were allowed to interact with animations by entering their own data sets as input to algorithms LBS94.
Hansen et al. built a hypermedia environment with animation as a key component to help teach students about algorithms. The use of the system exhibited signi cant learning bene ts, though these bene ts may have been caused by any aspect of the environment, not just animation HSN98.
The use of animation to help learners has been studied in a broader context than just learning about algorithms. A number of studies have focused on the contribution of animation as an aid to learning in other domains such as physics and user interfaces on computers.
Rieber, Boyce and Assad conducted a study in 1990 using a computer-based science lesson to teach introductory Newtonian mechanics to adults RBA90. In short, their results showed that neither the addition of static graphics, nor animated graphics had any e ect on learning as measured by a multiple-choice post-test. The study mainly attributes this to a maturation e ect: older students consistently rely less on external images than younger students. Pre77 " The claim is that adults can and will generate internal images given suitable explanations which the material provided and therefore the external images, the static and animated graphics, were not necessary for learning. On a more promising note, students who viewed animations were able to complete the post-test in signi cantly less time than the other students. According to the study, the retrieval process requires students to construct images in short-term memory. They hypothesize that the animations aided students in the retrieval process, presumably by facilitating the initial encoding."
A study by Palmiter and Elkerton in 1991 compared the use of animated demonstrations, written text, and a narrated animation for teaching users how to operate a particular graphical interface PE91. They expected, based on the results of earlier studies, that the narrated animation users would perform the best. Animation would aid the initial learning and narration would aid retention and transfer. Their results showed, however, that the performance of the animation-only and narrated animation groups was very similar;
both had problems with retention and transfer. They found evidence that users in these two groups may have been simply mimicking the procedures and only processing them super cially. As to why the narrations did not have the e ect seen with the written text, they give two possible explanations: that auditory text is processed di erently from written text, or that users were not paying attention to the narration well enough to process it thoroughly.
Pane, Corbett and John conducted a study of students learning about time-varying biological processes using multimedia software PCJ96. They compared a multimedia system that included text, graphics, animations and simulations to a control environment that used only text and carefully selected images. They found little evidence of bene ts from the multimedia system, and argue that these kinds of instructional materials must be prepared very carefully. They state, Merely using animation and simulation capabilities of modern computers does not guarantee improvement in students' learning. Well-designed static graphics and text may be just as e ective, and much cheaper to produce and use, than animations and simulations."
In a series of experiments between 1989 and 1994, Mayer et al. demonstrated that illustrations both static and animated can have a dramatic positive e ect on learning under certain conditions. Results from the early experiments, using only static illustrations, showed that students who viewed labeled illustrations showed better explanative recall and problem-solving transfer than students who saw only labels or illustrations or neither May89. Mayer claims that the labeled illustrations played two roles: guiding students' attention and helping them build internal connections i.e., connections between ideas in the text, as opposed to connections to previous knowledge.
Another set of experiments in 1991 and 1992 with Anderson, considered the use of animations to help students understand scienti c explanations MA91, MA92. In the experiments, college students with limited mechanical knowledge viewed animations and or listened to narrations explaining the operation of a bicycle pump and a hydraulic brake. Students who saw an animation and listened to an explanatory narration outperformed those who did not see the animation on a creative, problem-solving test. In a later experiment, this result was tightened to show that the bene t of animation occurred when it was viewed concurrently with hearing an explanation, not when the two occur contiguously MS94.
Mayer and his collaborators explain these e ects by noting how animation contributes to multiple representations of the problem domain. More speci cally, they cite the integrated dual-code hypothesis, adapted from Paivio's dual-coding theory Pai90, CP91, which posits that learners can build both visual and verbal modes of mental representations as well as connections between them." Further, they cite the importance of simultaneity in di erent multimedia explanations, claiming that a serial presentation animation and narration makes it more di cult for students to build referential connections between the two presentations.
While these experiments with animation in domains other than algorithms certainly help to inform our studies, we are reluctant to make any direct connections between their results and those to be expected for algorithm animations. All these other studies focused on an animation involving a tangible, visual usually physical phenomenon such as a pump, a brake, or a user interface on a computer. In these cases, learners have a pre-existing visual basis to draw from and leverage in knowledge construction. Algorithm animations, conversely, provide visualizations to computer data structures and operations which do not have any pre-existing visual basis. So, animation is being used not only to explain a dynamic process, but also to depict entities without existing visual representations. In some sense, algorithm animation is a broader, more abstract and complex problem domain than those studied in these other experiments.
2 Motivation All of the studies mentioned above either explicitly or implicitly through their design test a theory of how animations could aid learning. This theory is re ected in the choice of subject matter, the content of the animation, the accompanying materials, the method of presentation, the evaluation of learning, and the tasks and participants chosen GC96. In the studies that have failed to nd signi cant bene ts to using animation, at least three
explanations seem plausible:
that there are no or only limited bene ts from animation, that there are bene ts, but the measurements used in the studies are not sensitive to them, or that something in the design of the experiment is preventing participants from receiving the bene ts, or in other words, the theory of how animations could help needs to be re-examined.
This study investigates the third possibility by altering the traditional manner in which animation has been used in empirical studies and by making detailed observations of students using algorithm animations in educational settings. Other researchers, such as Hundhausen and Douglas, have theorized that the manner in which animation has been integrated into empirical studies and the learning assessment methods are inadequate for accurately assessing the bene ts of animation DHM95, DHM96, Hun98. They suggest that the answers to research questions such as How could animations aid learning?" lie in qualitative data gathered from observing students viewing and interacting with the animations in authentic settings. This is in stark contrast to controlled, comparative studies which usually require settings that are not authentic in order to produce clean, quantitative data.
This article describes a study that is a compromise between the quantitative and qualitative approaches, hopefully leveraging the best points of each. First, the study situates algorithm animations in a learning setting in a much more exible manner than previous studies, thus accommodating di erent student uses of the animations. Similarly, students access the learning assessment instruments in the study in a more exible manner. Second, we make detailed observations of each student, characterizing how they use animations and other instructional materials to learn about di erent aspects of an algorithm. Finally, we still include a traditional examination-style set of questions to assess how well the students understood the algorithm being presented.
The purpose of this study, then, was to gain insight into di erent ways that animations could t into successful learning strategies. In particular, we wanted to observe students
using animations in a more realistic learning situation to determine: