# «Abstract: This paper reports on results of a randomized controlled trial evaluating the impact of an online tutoring system on Grade 6 students’ ...»

Randomized Controlled Trial (RCT) Evaluation of a

Tutoring System for Algebra Readiness

Carole Shandy Steven A. Weiling Christopher

Beal Hauk Schneider Li Harrison

University of Arizona WestEd

Abstract: This paper reports on results of a randomized controlled trial evaluating the impact of

an online tutoring system on Grade 6 students’ mathematics learning. Starting with 71 teachers randomly assigned to use or not use the AnimalWatch program to replace 15% of mathematics class lessons, the study ended with 58 teachers’ classes (35 Treatment, 23 Control). Data from over 1200 students are included in the analysis (795 Treatement, 496 Control). Small but significant effects were found. Treatment group gains were larger than Control group gains on a major standardized test and on project-based topic quizzes (Cohen’s d=0.3). Exploration of fidelity of implementation indicated several factors at work and that the minimal effective implementation of this type was in classes where students completed at least 8 of 14 lessons spaced across several months. Software use records indicated that 75% of student-item interactions resulted in students persisting to a correct answer.

Objectives The study goal was to evaluate the impact of an online tutoring system on Grade 6 students’ mathematical proficiency relative to business-as-usual classroom instruction.

In particular, the research question driving the work: Do Grade 6 students who use the program AnimalWatch as replacement for about 15% of mathematics instructional time show greater learning gains than those who do not use AnimalWatch? The project responds to the clear need to improve the mathematics proficiency in the United States.

In international comparisons, U.S. students score in the average range overall, and much lower than students from other nations that are comparable in terms of economic development (Gonzales, Williams, Jocelyn, Roey, Kastberg, & Brenwald, 2008;

Organization for Economic Cooperation and Development [OECD], 2010). Although there is some evidence that mathematics scores have improved over the last decade the growth remains slow, and in many states there has been no improvement at all. There is special concern about algebra as a gatekeeper course for subsequent success, pointing to the need to ensure that middle school students are prepared to succeed in algebra with thorough proficiency in basic computation, fractions, ratios and proportions and other algebra readiness topics (Stein, Kaufman, Sherman & Hillen, 2011; Wimberly & Noeth, 2005). Thus, educators, researchers and policy-makers are searching for approaches that can help to improve students’ learning outcomes in math.

One possibility for addressing the identified need is the use of tutoring software for mathematics learning (Kulik, 2003). Such software is designed with the goal of providing individualized instruction to the learner via the use of features such as integrated explanations, videos and other multimedia resources to help them master concepts. In contrast, standard paper-based worksheets and end-of-chapter practice sets do not provide interactive, customized learning options. The opportunity to direct their own learning while using software may also provide a motivational advantage for some students. For example, struggling learners can choose to look at a multimedia lesson or explanation on the computer in private, whereas they might be quite reluctant to do so in the “live” classroom. Software also provides integrated tracking (e.g., problems completed, number completed correctly and so on) that can inform both students and teachers about progress in real time. These features are argued to lead to more effective learning (Bloom, 1984; Cohen, Kulik, & Kulik, 1982; Graesser, Conley & Olney, 2011;

vanLehn, 2011; Woolf, 2009).

Theoretical framework A considerable body of work suggests that the ideal learning context is one-on one instruction with an experienced human tutor (Bloom, 1984). Human tutors present problems that help diagnose the student’s sources of difficulty, choose problems within the student’s Zone of Proximal Development, scaffold the student to a successful solution, and then attribute the success to the student’s effort and enhanced understanding (Brown, Ellery, & Campione, 1994; Lepper, Woolverton, Mumme, & Gurtner, 1993). Human tutors balance the goal of providing challenge to ensure cognitive conflict, resolution and improved understanding, with the goal of sustaining the learner’s motivation and persistence (Merrill, Reiser, Ranney, & Trafton, 1992).

Unfortunately, it is very difficult for math teachers to provide much individual help in large classes. Many Grade 6 math teachers have four to five classes a day with 25-30 students or more per class. It is physically impossible for even the most dedicated and skilled teacher to diagnose and assist every student. Nor is it necessary when technology based systems can perform some of these functions, providing students with individualized practice opportunities and, in turn, providing teachers with real-time reports of students’ progress.

AnimalWatch is an intelligent tutoring system designed to give individualized instruction by selecting problems that will build a Grade 6 student’s proficiency (Beal & Arroyo, 2002). Like other intelligent tutoring systems, AnimalWatch uses student models (representations of what the student is estimated to understand, in relation to the target domain: here a network of 30 math topics mapped to the California Grade 6 curriculum frameworks) to guide problem selection. Though not the topic of the current report, we note that the student’s actions while working on problems (e.g., answers, clicks to view multimedia help resources) can be researched to update our understanding of the nature and importance of individual tracking of a model for each student.

Although tutoring software is designed to include features intended to promote learning, research results in the K-12 classroom have been inconsistent. A recent meta-analysis of a wide range of computer-based instruction concluded that the use of technology was generally associated with improvements to student achievement (Tamin, Bernard, Borokhovski. Abrami & Schmid, 2011). VanLehn (2011) reviewed a battery of tutoring systems that adapt instruction in ways similar to what a human tutor would do and concluded that such software was as effective as human tutoring. Several studies suggest that there may be benefits to the use of tutoring software specifically for mathematics learning. Arroyo, Woolf, Royer, Tai and English (2010) reported that students who used a tutoring system for geometry along with modules for basic skills practice showed significant improvements in mathematics problem solving. Barrow, Markman and Rouse (2009) concluded that a computer-assisted instructional program for pre-algebra had a significant benefit for students, in comparison to business-as-usual 2 instruction. Beal, Arroyo, Cohen and Woolf (2010) found that a tutoring system for middle school mathematics was comparable to small-group tutoring delivered by a mathematics teacher and also showed benefits in a quasi-experimental design study with a business-as-usual comparison condition. Ritter, Kulikowich, Lei, McGuire and Morgan (2007) reported positive results associated with the Cognitive Tutor, an instructional system for high school algebra that has been widely used in classrooms.

Although results from these studies are encouraging, conclusion are limited because they have typically involved fairly brief interventions, and the use of studyspecific mathematics tests as outcome measures. In contrast, research using more rigorous experimental methodologies and more standardized outcome measures has not demonstrated a consistent benefit associated with tutoring systems. For example, a review of four widely-used mathematics tutoring systems for mathematics conducted by the What Works Clearinghouse concluded that there was no overall evidence that the software was associated with better student mathematics test performance (Campuzano, Dynarski, Agodini & Rall, 2009). Thus, there is a need for more systematic research on the potential benefits of tutoring systems to support mathematics instruction, and how teacher implementation may influence the effectiveness of tutoring systems. The study reported here was designed to address this need through a rigorous evaluation of the impact of one tutoring system for algebra readiness on students’ mathematical proficiency.

Methods Experimental design and participants The study was a randomized controlled trial (RCT). Teachers were recruited and signed participation consent forms before being randomly assigned to either the Treatment or Control condition. Treatment teachers participated in a two-hour online professional development module designed to introduce them to the software and its features, and then integrated it into their instruction throughout the school year. Software use replaced approximately 15% of instructional time, usually through weekly visits to a computer lab or use of laptop carts in the classroom. Control teachers continued with business-asusual instruction. The student sample was consenting Grade 6 students. Though begun with 71 teachers (36 Treatment group, 35 Control), due to student movement and teacher attrition, we ended the year with 1,010 in 35 Treatment classes and 636 students in 23 Control classes across 40 districts. Their districts identified about 20% of students as English Learners. In most districts, a letter sent home to caregivers notified them of study activities and offered an opt-out form to abstain from their child’s scores being included in analysis; in one district, active caregiver consent was obtained for all participants.

Tutoring system The AnimalWatch software evaluated in this study focused on word problem solving, generally considered a central component of mathematics proficiency (Koedinger & Nathan, 2004). As recommended by the National Council of Teachers of Mathematics (NCTM, 2000) and Common Core State Standards (CCSS, 2010) AnimalWatch integrates mathematics with meaningful content. In this case, with authentic environmental science material, focusing on narratives about tracking and monitoring the status of endangered and threatened species (hence the AnimalWatch system’s name). By connecting mathematics problem solving with science, the student encounters 3 many examples of mathematics in real-world problem contexts, at a point in the curriculum when many students begin to complain that mathematics is disconnected from their lives. The topic of environmental science is engaging to many pre-teens, both boys and girls, and aligns with the science frameworks for middle school (Grade 6 focuses on Earth Science, and Grade 7 on Life Science) as well.

Word problems. AnimalWatch includes approximately 1400 word problems organized into 20 learning objectives. Fourteen of these learning objectives were considered “core” for the study. Each learning objective centers on one topic in sixth grade mathematics such as operations with integers, fractions, or rates. Within each learning objects are narratives about endangered and threatened species, including the Giant Panda, Przewalski Wild Horse (Takhi), Right Whale, California Condor, Snow Leopard, White Shark, Great Cats, and Snakes of Asia. Each word problem includes an introduction with authentic background information, a graphic (image, figure or table) and a question derived from the introduction. Scientific terms in the word problems that may not be familiar to students are linked to an integrated glossary. Students enter their answer, including units, into an answer box and receive immediate feedback. For example, in one problem, a white shark is described as swimming from the ocean surface down to a depth of 30 meters below the surface, and then swimming up to a depth of 16 meters. The numbers in the problem are consistent with current scientific knowledge about shark behavior. The student is asked to compute the total distance traveled. Students must understand the concept of the number line, the analogy of the ocean’s surface as the zero point, and the concept of elevation (defined in the integrated glossary) to solve the problem. The problem is linked to the state’s mathematics standards for Number Sense (working with integers) and Mathematical Reasoning (deciding how to approach the problem, evaluating the answer for reasonableness in the context of the original situation). If the student makes one or two problem-solving errors, text feedback about accuracy is presented, followed by an operations hint (e.g., “No, that’s not quite right.” “Are you sure you’re subtracting?”). A third error elicits a recommendation to view the associated help resources. When the student clicks on the “help” icon, a menu window appears, showing the options available for that topic, including text explanations (e.g., how to find the least common denominator), worked examples, interactive solutions, and video lessons. Students’ requests to view help resources are automatically tracked, and these process data will be extracted for later analyses. A fourth error elicits the correct answer, which the student is required to enter before moving on to a new problem. Students also have the option of deciding a question is “too hard” and the answer is shown, but the question marked as incorrect.

Basic skills practice. Royer et al.’s (1999) work indicates that improved fluency with simple computation and mathematics facts is linked to improved problem solving performance on achievement test items. Thus, AnimalWatch includes skill builders that provide students with opportunities to build fluency through practice with multiplication facts, recognizing decimal-fraction-percent equivalents (e.g., identifying that 1/8 = 0.125 = 12.5%), rounding and estimation, one-variable equations (e.g., finding the value of x when 2x = 10), finding the least common denominator of two unlike fractions, and operating with negative integers. Each skill builder set presents the student with a series of 10 mathematical statements, one at a time (e.g., 35 + (-15) = 20) which the student identifies as either true or false; the immediate feedback is a green check if correct and 4 red x if not. Skill builders can be repeated multiple times with new items. The skill builders are motivating for students because they can achieve near-perfect scores with repetition, and repetition in turn helps to builds fluency with the concepts and operations, indicated by decreases in time required to solve the items.