«FROM POSTS TO PATTERNS: A METRIC TO CHARACTERIZE DISCUSSION BOARD ACTIVITY IN ONLINE COURSES Catherine A. Bliss and Betty Lawrence Center for ...»
JALN Volume X, Issue X – Month Year
FROM POSTS TO PATTERNS: A METRIC TO
CHARACTERIZE DISCUSSION BOARD
ACTIVITY IN ONLINE COURSES
Catherine A. Bliss and Betty Lawrence
Center for Distance Learning
Empire State College
111 West Ave, Saratoga Springs, NY 12866
Telephone: (518)587-2100 ext. 2388
Email: Catherine.Bliss@esc.edu, Betty.Lawrence@esc.edu
ABSTRACTAsynchronous text based discussion boards are included in many online courses, however strategies to compare their use within and between courses, from a disciplinary standpoint, have not been well documented in the literature. The goal of this project was to develop a multi-factor metric which could be used to characterize discussion board use in a large data set (n=11,596 message posts) and to apply this metric to all Mathematics courses offered in the January 2008 term by the Center for Distance Learning at Empire State College. The results of this work reveal that student participation rates, quantity of student posts, quality of student posts and the extent of threading are well correlated with instructor activity.
I. BACKGROUNDOnline learning programs have grown tremendously in the past 15 years. As these programs continue to grow, research in the areas of instructional design and course evaluation have also grown. One area of online course design, delivery and evaluation that has received a particularly high level of attention is the use of asynchronous text based discussion. Providing a framework for the purpose of discussion boards in an online course, Garrison, Anderson and Archer‟s  model of “Communities of Inquiry” for online learning environments place student learning at the intersection of social, cognitive, and teaching presence. This model has been extended by  and  by overlaying interaction with peers, interaction with content and interaction with instructors with social, cognitive and teaching presence, respectively.
At the core of this extended model are Chickering and Gamson‟s “Seven Principles for Good Practice in Undergraduate Education” .
A. Social presence Social presence has been defined as “the ability of learners to project themselves socially and affectively into a community of inquiry” [5 pg. 50]. As such, discussion boards are a place where learners can engage themselves socially and the class may develop a sense of community [6,7]. For adult learners who may not have the traditional support networks that younger students may have, and for distance learners, who are separated by time and physical space, the development of peer support networks via discussion boards JALN Volume X, Issue X – Month Year within an online course may be particularly important.
Research with students in the State University of New York‟s s Learning Network supports the notion that students benefit from this social interaction in online courses. Researchers  and  found a positive correlation between students‟ perceived learner-learner interactions and students‟ satisfaction with their online course. These authors also found positive correlations between students‟ perceived learner-learner interactions and their perceived learning. This suggests that building the social presence and encouraging learner-learner interactions may be one way in which discussion boards enhance the online learning experience.
B. Cognitive presence Discussion boards can be used to facilitate students‟ co-construction of knowledge, engagement in higher order thinking, and the development of critical thinking skills [10,11,12]. Discussion boards can be a place where students negotiate meaning of course content and practice skills before independent mastery is expected.
Constructivist learning theory suggests that knowledge is constructed by learners, rather than transmitted from teacher to student. According to Vygotsky‟s Social Learning Theory, learning occurs through students‟ social interactions . The Zone of Proximal Development is defined as the zone between the abilities which students can perform independently and those which require the guidance of an instructor.
Vygotsky concluded that this is where learning occurs. This theory suggests that discussion boards may be an opportunity where students may bridge the gap between existing skills and new skills to be acquired.
Discussion boards also enhance student learning in an online course because they are an area where higher order cognitive processing may be developed. In 1956, Benjamin Bloom described six of levels for higher order cognitive processing and suggested that engagement in the higher levels would allow a greater transfer of learning beyond the course context . Students may often operate in the first three levels of cognitive processing (knowledge, comprehension and application) via individually based homework assignments, projects, and assessments. However, it is a more challenging task to engage students in the upper three levels (analysis, synthesis and evaluation). It is precisely through class discussions that students may argue, evaluate, assess, construct, design, propose, compare and analyze course content. “Electronic discussion changes the focus of the learning process, replacing the single view of the instructor with the diversity of views from different students” [10 pg. 41]. It is within these levels where a deeper understanding of course content can be cultivated, thus allowing students to “package and bundle” course content to be used in their professional and personal lives. Thus, discussion boards in an online course may be a place where students can engage in higher order thinking.
Discussion boards have a role in teaching critical thinking skills. As an instructional tool, they can encompass the best elements of writing assignments and in class discussions . In an online course, an asynchronous text-based discussion board may hold the added advantage of a time-lag between postings.
This may lead to greater reflection and enriched discussions as compared to discussions in face-to-face settings [15,16].
Several authors have found support for the theory that discussion board use may support cognitive processing. Work by , , , and  indicates significant positive correlations between perceived discussion board activity and perceived learning. Even more convincing,  and  found a significant positive correlation between the quantity of student posts and students‟ scores on final assessments. It is possible that students with particularly good study habits, post frequently to the course
and also work efficiently to master course objectives, thus scoring well on final assessments. At the very least, though, discussion boards can support cognition through peer-dialogue and increased learnercontent interactions.
C. Teaching presence Teaching presence in online courses can be defined as consisting of three roles: “design and organization, facilitating discourse, and direct instruction” [19 pg. 1]. Instructors may use several strategies for facilitating discourse, such as “identifying areas of agreement/disagreement, seeking to reach consensus and understanding, encouraging, acknowledging, and reinforcing student contributions, setting the climate for learning, drawing in participants and prompting discussion, and assessing the efficacy of the process” [19 pg. 8]. Discussion boards are a place where instructors may facilitate discourse and provide instruction. Instructors may select content and re-focus a discussion that may have diverged from the main topic. Instructors may also set a tone which respects the diversity of ideas, opinions and experiences and which models an appreciation for the love of learning. As such, instructors need to be present in discussions and discussion boards are a place where teaching presence may be demonstrated.
The role of the instructor in facilitating online discussion has been discussed widely in the literature [8Surprisingly, considerable debate exists in the literature about the role of an instructor in facilitating online discussion. On one side of the debate, there is the belief that the instructor is crucial to facilitating discussion, but should do so without taking over the discussion [7-10,20,23]. Some researchers  suggest the role of the instructor is to be a “guide on the side,” limited to maintaining focus, moving the discussion along and prompting students to reflect on the discussion. In studies examining discussion board activity,  and  found that “faculty responsiveness” was a salient factor in developing a meaningful online discussion. However, “more is not necessarily better in terms of presence” [21 pg. 143]. Instructors who respond too quickly or extensively may shutdown student interaction.
The view that instructors should be active on discussion boards is supported by researchers examining student satisfaction and perceived learning in online courses. Positive correlations have been found between a.) perceived learning and perceived instructor-student interaction [8,24], b.) the quantity of instructor postings and the quantity of student postings , and c.) students‟ perceived level of interaction with instructors and overall satisfaction for courses . Other researchers [3,25] have found that the extent to which instructors are perceived to facilitate discussion is correlated with students‟ satisfaction of online courses and reported learning.
On the other side of the debate is the view that instructors can interfere with student dialogue and actually inhibit discussion quality rather than facilitate it [22,23]. Although one researcher  suggests that instructors‟ posts can actually shut down discussion between students, this analysis is ad hoc and involves only a single case study. No transcript analysis is included in the report and it is not possible to determine if it is the type, rather than the quantity of instructor posts that may be correlated with less dialogue. The conclusions drawn by  are also problematic. These researchers found that high percentages of instructor posts were correlated with low quantities of student posts and concluded that instructors may be “shutting down” discussions. This index is problematic because in the case of large numbers of student posts, the percentage of instructor posts would be small, simply because the total number of posts is large.
Thus, the need for a new measure is apparent.
JALN Volume X, Issue X – Month Year
II. DEVELOPING A METRIC FOR ASSESSING DISCUSSION BOARD
ACTIVITYSeveral researchers have suggested methods for measuring discussion board activity. Some have suggested multi-factor metrics [18,21,26-28], while others have specifically focused on participation [23,29,30], the role of instructors [3,19,23,26], quantity of student posts [17,18,32], the quality of discussion [11,28,33-35], the discussion board prompt [36-39], guidelines posted by the instructor  and feedback [7,44].
One of the difficulties in measuring discussion board activity revolves around the multitude of factors which may affect discussion board activity (i.e. course design, instructional approach, learner specific characteristics). The other difficulty often revolves around the need for detailed and comparable results.
Most of the work presented in the literature thus far has focused on a single case study or a small handful of courses, analyzed in a cross case study analysis . The goal of this paper is to propose a discussion board metric and to apply this metric to online course offerings in one discipline. This work is valuable in that most of the metrics reported in the literature have dealt with data sets on the order of 1000 message posts or less. The metric described in this paper was applied 11,596 message posts and thus can serve as a tool to analyze differences between courses and between sections of the same course.
A. Student participation For this paper, participation is defined as the percent of enrolled students who decide to post to a discussion board. Participation rate is an important measure to include in designing a discussion board metric and is distinct from the quantity of student posts. In an ideal setting, all students who enroll in an online course will complete the course and all currently enrolled students will engage in all learning activities. In practice, this is not the case. Thus, it is informative to examine participation rates, both within a course and between courses.
This definition is an improvement over previous attempts to quantify participation. Many authors use quantity of posts and participation interchangeably. This blurs the distinction between the proportion of student using the discussion board (participation) and how “vocal” students are once they decide to participate (quantity). By examining participation rates, it may be possible to identify areas in need of revision.
B. Quantity of student postings The quantity of student posts is a common and usually straightforward measure to gather from learning management system software. One approach, as demonstrated by  in a cross case study approach, uses the mean number of messages per forum as an indicator for the quantity of posts. Although this measure is useful in a small case study analysis, this index could be misleading in comparing a large number of courses. For example, a mean value of twelve messages per forum could represent very different quantities of posts, depending on whether these posts were made by twelve students or four students.
Another approach for measuring quantity of student postings is to calculate the number of student responses divided by the number of students in the course [9,17,23,31,32]. Although this index standardizes quantity of posts by course size, it is misleading in that not all enrolled students choose to participate in every discussion. The index for quantity of student posts could be improved by using the “active” students in a discussion board rather than enrolled.