«August, 2006 All rights reserved. Any or all portions of this document may be reproduced and distributed without prior permission, provided the ...»
Teachers’ Uses of Students’ Digital
Annotations: Implications for the
Formative Assessment of Reading
Barry Brahier, Ph.D.
University of Minnesota, Center for Reading Research
All rights reserved. Any or all portions of this document may be reproduced and
distributed without prior permission, provided the source is cited as:
Brahier, B. (2006). Teachers’ uses of students’ digital annotations: Implications for the formative assessment of reading comprehension. Minneapolis, MN: University of Minnesota Center for Reading Research.
Teachers’ Uses of Students’ Digital Annotations: Implications for the Formative Assessment of Reading Comprehension Barry Brahier, Ph.D.
University of Minnesota Reading comprehension has been called the “bottom line” of reading instruction (McKenna & Stahl, 2003) and recent national reports call for improving the assessment of reading comprehension by addressing the complaint that most assessments are not useful to teachers (Snow, 2002; Snow & Biancarosa, 2003). Teachers’ use of their students’ annotations fits well with calls for literacy assessment reform. In arguing for additional assessment types, Tierney (1998) asserted that students’ “learnings may be fleeting” (p. 376) and suggested teachers facilitate assessment by encouraging students “to keep traces of what they do” (p. 375). Winne and Hadwin (1998) argued specifically for teachers to examine students’ margin notes as traces of their metacognitive effort, while Johnston (2003) urged teachers to notice and record the literate practices of their students. While examining students’ annotations would be a step towards the implementation of these reforms, currently there are two significant barriers to the use of annotation in K-12 schools.
The first barrier to the use of annotation is the complete prohibition against marking in the textbooks students use in K-12 schools. This is for good reason as the common practice is that the school district, not the students, own the books. This barrier could be overcome by mandating that families buy textbooks for their children, or the adoption of disposable workbooks, or both. However, these solutions would leave intact the second barrier: annotations on paper materials are not easily shareable between students and their teachers. Both of these barriers are being addressed through the increasing availability of digital annotation systems.
Digital annotation systems allow users to add marks and notes to existing digital text. In 2002, Wolfe described over 25 digital annotation systems, categorizing them as either in existence or development. While their features vary, they all overcome both of the barriers stated above by taking advantage of what Negroponte (1995) called “the differences between bits and atoms” (p. 11). Negroponte used the term bits to label items in the digital realm that have no physical existence (i.e., computer files such as those made by word processors, spreadsheets, and so on) and atoms to label anything that does have physical existence (i.e., paper, books, ink, pencil, etc.). The advantages of bits apply to all digital documents, whether they reside in a digital annotation system or not. These include (a) copies of digital documents are identical to originals and multiple copies can exist simultaneously (e.g., the original, one annotated by the student, a student-annotated version being viewed by the teacher, and one available for viewing by other teachers and parents); (b) additional information (e.g., annotation) can be added and modified as needed; (c) digital documents can be searched with sufficient speed to make looking up a word or phrase practical; and (d) digital files are transportable across networks of computers at the speed of light. Further, in digital annotation systems, hardware and software combine to keep all reading and annotation tools simultaneously ready for use.
Readers use the same tool (i.e., a mouse, a stylus) to turn pages, browse, and add annotations. This capacity, which Marshall (1997) called “ready-to-hand” (p. 5), streamlines the making of annotations to the point where it is easier to make annotations in some digital annotation systems than it is on paper. For example, digital annotation systems, because they allow readers to manipulate bits instead of atoms, permit readers to modify their annotations after they have been made (i.e., the color of highlighted text can be changed or the highlighting deleted).
This study used a qualitative case study design to gather data to answer the question “What are the outcomes of using digital annotation software among secondary school teachers?” Two high school teachers used RepliGo™ digital annotation software (Cerience Corporation, 2003) with their students over the course of one month during the fall of 2005. Teacher interviews, curricular materials, and student work were analyzed using Hughes’ (2000) Replacement-Amplification-Transformation taxonomy to determine outcomes attributable to the use of RepliGo™.
Review of the Literature
minimum is the marking of an existing text. A mark can be any visible trace and a text could be letters and/or symbols, audio (Coates, 2005; Northwestern University, 2005), or video (Smith, Blankinship, & Lackner, 2000). Jackson (2001), in her seminal historical examination of annotations made in books, defined annotation as marks that are “the product of an interaction between text and reader” (p. 100). She described the gamut of annotation in books and included all possible marks regardless of purpose or intent (i.e., a signature on the inside front cover, underlining, written notes). Jackson’s wide-ranging analysis also included the role annotation plays in helping readers understand what they are reading.
The practice of annotating books as an aid to the construction of meaning has been in existence for centuries. Jackson (2001) found numerous recommendations to make annotations in educational treatises and concluded annotation is “a minor theme in educational theory” (p. 48). Teachers as far back as Erasmus in the early sixteenth century “advised pupils on methods of annotation and supervised and encouraged their efforts” (p. 46). Jackson cites Pryde, author of Highways of Literature; or, What to Read and How to Read in 1882, saying, “Pryde promotes a system of note-taking followed by written digests in the reader’s own words” (p. 49). More recently, Frederick (1938) exhorted the student to “…develop a system of marking his books so that he will quickly see what his thoughts were as he reads” (p. 213). Adler (1942) and Adler and Van Doren (1972) called annotation marking and maintained that critical reading relied on it. Finally, a form of annotation called functional underlining was considered by Miller (1980) “as basic to the intelligent reading of literature as a test tube is to a chemistry experiment” (p.
577). Jackson found annotating directly in books had several advantages for learners over making notes in a notebook, among them (a) the writing of annotations requires less concentration and (b) an annotation allows the passage of text in the book to be referred to later as a check against the reader’s interpretation. Despite the accepted usefulness of annotation to readers’ efforts to understand what they read, there are few accounts of how useful students’ annotations are to their teachers.
In contemporary post-secondary institutions, where annotation is a common practice among students, instructors seldom make use of students’ annotations.
Salvatori (1996) was an exception when she described how seeing a student’s highlighted (equivalent to underlined) textbook altered her ideas about students’ annotations:
That class made it possible for me to turn a rather mechanical study habit—the highlighting of a text—into a strategy, one that can make visible the number and the intricacy of strands in a text’s argument that a reader (or an interlocutor) pays attention to and that can show how the selection, connection, and weaving of those strands affects the structuring of the argument a reader constructs as/in
This is a case where an instructor identified one of Jackson’s (2001) advantages of annotation: the text itself served as a check against the student’s interpretation. However, this was a chance, not intentional, encounter. If the student had not chosen to record her interpretation or if Salvatori’s eye had not been caught by the vivid highlighting in the book open on the student’s desk, then the alteration in instruction would not have occurred.
Literacy researchers’ approaches to annotation research. Literacy researchers have studied two types of annotations: underlining and/or highlighting, and margin notes (Berger & Schlitz, 2000; Fowler & Barker, 1974; Hartley, Bartlett, & Branthwaite, 1980;
Hynd, Simpson, & Chase, 1990; Lindner, Gordon, & Harris, 1996; Nist & Hogrebe, 1987; Nist & Simpson, 1988; Peterson, 1992). Participants in these eight studies were post-secondary students except those in the Hartley, Bartlett and Branthwaite study (1980), which studied sixth grade students. All studies used experimental designs, and the dependent variable in each study was student performance on a recall measure.
The literature reviews in two of the studies (Hartley, Bartlett, & Branthwaite, 1980; Nist & Simpson, 1988) surveyed a larger number of studies than the eight studies examined for this review. Hartley, Bartlett, and Branthwaite (1980) reviewed forty-one studies on the effectiveness of underlining and found that “few studies, if any, provide clear-cut support for the effectiveness of underlining, and that little is known of how children benefit (or don’t benefit) from underlining” (p. 218). Nist and Simpson (1988) reviewed 13 studies that either (a) compared the effectiveness of researcher-supplied underlining with student-generated underlining or (b) compared the effectiveness of underlining with other study strategies. They found “the only safe conclusion which can be drawn is that underlining does not to appear to be detrimental” (p. 251).
Underlining was the independent variable in three of the studies, with two (Hartley, Bartlett, & Branthwaite, 1980; Nist & Hogrebe, 1987) comparing researcherprovided underlining with student-generated underlining, while the third (Peterson, 1992) compared student-generated underlining against no underlining. Hartley et al. (1980) found that sixth graders who studied materials with researcher-supplied underlining performed significantly better on a cloze test. Nist and Hogrebe (1987) found no significant difference between the treatment and control groups, and Peterson (1992) found the control group outperformed the treatment group on an inferential recall measure.
When the independent variable was highlighting, Fowler and Barker (1974) reported no significant difference between groups of college students assigned to highlighting, underlining, or control groups. Lindner, Gordon, and Harris (1996) found no significant differences in students’ performance on a multiple choice test in their first experiment, and in their second experiment, the control group outperformed the treatment group. Berger and Schlitz (2000) examined the effect of researcher-supplied underlining that was intentionally poor, comparing it to a group who received researcher-supplied highlighting that was high-quality, and a group that received no intervention. Berger and Schlitz found no significant difference between the two groups who received the intervention, and the control group outperformed both of the experimental groups.
Two studies (Hynd, Simpson, & Chase, 1990; Nist & Simpson, 1988) examined the role of margin notes in promoting recall as measured by performance on multiplechoice tests. In both studies, groups trained to make annotations in the form of margin notes outperformed either the use of self-selected study strategies or the use of a journal.
To summarize the literacy research on annotation, the primary annotation method investigated is underlining and/or highlighting, and the reason to annotate is to enhance performance on recall measures. The results indicate that, at best, the use of annotation to enhance recall produces no significant improvement. Some research does indicate recall improves when students are trained to annotate by making margin notes. However, none of the studies reviewed investigated annotation as a formative assessment. This represents a gap in how the field conceives of annotation, perhaps due in part to the fact that the use of students’ annotations as formative assessments requires teachers to have access to their students’ annotations. This access is inherently impractical when annotations are made on paper materials. This study addressed that gap by placing a digital annotation system into the hands of K-12 teachers and reporting teachers’ uses of their students’ annotations as formative assessments.
Research into the development and uses of digital annotation systems. The research literature surveyed for this portion of the review focuses on the development and uses of digital annotation systems. Six studies (Marshall, 1997, 1998; Marshall, Price, Golovchinsky, & Schilit, 1999; Ovsiannikov, Arbib, & McNeil, 1999; Schilit, 1999;
Waller, 2003) examined how adult readers annotated paper documents to inform the design of digital annotation systems. Two of these (Marshall, 1997; Ovsiannikov, Arbib, & McNeil, 1999) are reviewed here to explicate some of the ways learners use annotations.
Marshall (1997) analyzed college students’ annotations by examining used textbooks selected from the stock of a college bookstore. Marshall did not have access to the students who annotated the books in her sample and thereby relied solely on the highlighted books when interpreting what she found. Marshall found that annotations functioned as place markers and aids to memory (i.e., short passages Marshall supposed were for later use in writing a paper or preparing for a test), records of interpretive activity (i.e., marks that indicated unfamiliar language, commentary, and evidence of misreading), and visible traces of the reader’s attention (i.e., more annotations were present when content seemed more difficult to comprehend).