«Julia Kaufman, Sarah Ryan, Candace Thille & Norman Bier1 Open Learning Initiative Carnegie Mellon University October 2013 Julia Kaufman and Sarah ...»
Open Learning Initiative Courses
in Community Colleges:
Evidence on Use and Effectiveness
Julia Kaufman, Sarah Ryan, Candace Thille & Norman Bier1
Open Learning Initiative
Carnegie Mellon University
Julia Kaufman and Sarah Ryan are research scientists at the Open
Learning Initiative (OLI). Candace Thille is the founding director of OLI,
and Norman Bier is the current director of OLI at Carnegie Mellon. John
Rinderle from OLI also provided much helpful assistance and feedback for this report. Additionally, Brian Junker and Howard Seltman provided invaluable statistical advice. The authors would also like to thank the many faculty and students who participated in this evaluation, as well as the funders - Bill & Melinda Gates Foundation, the Lumina Foundation, the William and Flora Hewlett Foundation and the Walter S. Johnson Foundation - who made this work possible.
Introduction Despite high costs for a college education and increasing debts that students face after they graduate, the demand for effective post-secondary higher education continues to grow. In 2013 alone, college enrollment grew by 15.4% across the U.S. 2 The increase in the number of students seeking post-secondary education has brought with it a corresponding increase in learner variability: a larger population of students with broader experiences, differing perceptions and diverse achievement levels.3 In the face of dramatic challenges in access and attainment, policy makers, foundations and administrators have looked to technology as a mechanism for controlling costs and increasing student success. There has been a dramatic increase in the use of online courses over the past decade, 4 and an unprecedented interest in Massive Open Online Courses (MOOCs) in the past year. In the majority of cases, online courses have been executed in a manner that attempts to replicate the experience of a traditional classroom, in effect leveraging the technology as a mechanism for scaling lectures to a broader population of students. Although the MOOC approach has generated great excitement, outcomes have thus far not lived up to expectations.5 A different approach to leveraging technology in support of learning can be found in the work of the Open Learning Initiative (OLI). OLI is an open education research project that began in 2002 at Carnegie Mellon University with a grant from The William and Flora Hewlett Foundation. Unlike many open education projects, OLI is not a collection of materials created by individual faculty to support traditional instruction. The original goal of the project was to develop web-based learning environments that support individual learners, who do not have the benefit of an instructor, to achieve the same learning outcomes as students who complete traditional courses.
To meet the goal of supporting independent learners, OLI built on Carnegie Mellon’s institutional strengths in cognitive science, software engineering, and human-computer interaction. OLI courses are developed by teams composed of disciplinary faculty, learning scientists, human-computer interaction experts, and software engineers in order to make the best use of multidisciplinary knowledge for 2 State Higher Education Executive Officers, State Higher Education Finance, FY 2012.
3 The Chronicle of Higher Education. “Who Are the Undergraduates?”, December 12, 2010 4 November 2011 report by the Sloan Consortium and the Babson Survey Research Group.
5 Ry Rivard. “Udacity Project on ‘Pause’”, Inside Higher Education,
designing effective learning environments. The OLI design team articulates an initial set of student-centered, observable learning objectives and designs the instructional environment to support students in achieving those outcomes.
Although originally designed to support individual learners, OLI courses are increasingly used by instructors inside and outside of Carnegie Mellon as a complement to their instructor-led courses.
OLI learning environments have been used in numerous studies to better understand effective approaches to enhancing learning with technology and to gain an understanding of how the courses support learners and educators. Studies at Carnegie Mellon have shown that students using the OLI statistics course were able to achieve the same or better learning outcomes in half the time of peers in the traditional course. 6 A subsequent independent trial of the Statistics course was conducted by ITHAKA S+R across six public institutions. Using a randomized approach with statistically reliable control and treatment groups, ITHAKA S+R reported that students using OLI had equivalent learning outcomes and completed the course 25% more quickly. 7 The successes that prior studies have demonstrated in accelerating learning in fouryear colleges raises the question of how the OLI approach could be successfully used to support community colleges in addressing the challenges currently facing higher education. In the face of shrinking state budgets, community colleges are being asked to increase attainment rates and reduce the cost of instruction. They are pinched for money, squeezed for space and find themselves under unprecedented pressure to see to it that the growing number of students who matriculate on their campuses each year complete their degrees. Yet, over half (54%) of all community college credits are non-productive (i.e., courses in which students fail, withdraw or that do not contribute to program completion). 8 To explore how the OLI approach might be used and adapted to address some of the challenges confronting community colleges, OLI launched the Community College Open Learning Initiative (CC-OLI) in 2009. Jointly funded by the Bill & Melinda Lovett, M., Meyer, O., & Thille, C. (2008). The Open Learning Initiative: Measuring the effectiveness of the OLI statistics course in accelerating student learning. Journal of Interactive Media in Education.
7 Bowen, W.G., Chingos, M.M., Lack, K.L., & Nygren, T.I. (2012).
Interactive Learning Online at Public Universities: Evidence from Randomized Trials. ITHAKA.
8 “The Institutional Costs of Student Attrition” by Delta Cost Project,
Gates Foundation, the William and Flora Hewlett Foundation, the Lumina Foundation and the Walter S. Johnson Foundation, CC-OLI is an attempt to apply the lessons learned and success from OLI’s earlier work to the unique challenges faced by 2-year institutions. CC-OLI focused on adapting or creating learning environments to support gateway courses (i.e., high-enrollment courses with low student success rates that are critical to many degree paths and that are often an obstacle to degree completion). CC-OLI’s goals were to include community college faculty in the development, use and evaluation of these courses and to scale to at least 40 community colleges and 100 classrooms, all with an overarching goal of demonstrating a measurable increase in student success.
As of spring of 2013, the CC-OLI project has created community college-targeted courses in Statistics, Introduction to Biology, Anatomy & Physiology and Introduction to Psychology, along with pilot modules in English Composition.
Participation in these development efforts has been widespread, with 51 faculties from 48 institutions participating in development. Thus far, 39,000 students in classes offered at 142 academic institutions, have used these courses.
While both the accelerated learning studies and the ITHAKA S+R studies reported similar or improved learning outcomes associated with the introduction of OLI materials into the classroom, neither study investigated differences in outcomes between groups of students who used the OLI materials minimally and those who made more extensive use of the learning resources. Similarly, the earlier studies did not explore faculty use of the courses. We know that the way faculty use the learning assets and the instructor tools available within OLI varies greatly. Gaining an understanding of how use of the OLI resources is associated with students’ learning outcomes would have real value, both for refining the learning environment design and supporting faculty in incorporating OLI resources into their practice.
The current report summarizes the findings from a study of the use of the CC-OLI courses by faculty teaching at 24 community colleges across the United States. The report particularly delves into the relationship between faculty/student use of OLI resources and students’ learning outcomes. Beyond the specific findings with regard to CC-OLI, the study also reports on some of the challenges in executing large-scale evaluation studies in higher education.
Open Learning Initiative Courses in Community 5 Colleges Goals of the Study We had three main goals for the current study that had implications for how we designed our study and analyzed our data. We outline those goals alongside our rationale for each goal and the research methods we used to gather and analyze data addressing those goals.
We first describe our study methods in more detail below, including a description of the Open Learning Initiative courses and the components of those courses we studied. We then provide main findings in relation to the goals listed above. Those
main findings include:
• Key differences in demographics, academic performance, and preferences defining faculty and students who use OLI compared to those who do not
• Positive but non-significant effect of OLI courses on achievement gains for a subset of students of faculty who agree to use OLI with one of their class sections but not the other
• Huge variations in use of OLI courses among faculty and students
• Large achievement gains among faculty and students who engage in “optimal use” of OLI resources as they have been measured for this study W Data and Methods e About Open Learning Initiative Online Courses The Open Learning Initiative (OLI) draws on the expertise of college faculty from particular content areas alongside software developers to create high-quality online resources. The courses on which we focused for this evaluation were introductory “gateway” courses critical for graduation and student success: Anatomy & Physiology, Biology, Psychology and Statistics.
All OLI courses are designed to engage students to put new knowledge into practice and support students to assess their own progress. Activities and assessments embedded within the courses help to ensure that students understand concepts rather than simply memorize facts. Embedded learning activities include “Learn by Doing” (LBD) and “Did I Get This” (DIGT) activities. LBD activities are meant to help students practice new skills or develop understanding of a new concept. While engaged in a LBD activity, students receive automated support and feedback. DIGT activities are meant to help students assess whether they have mastered a skill or understood a specific concept. Upon completing a DIGT self-assessment, a student receives context specific and immediate feedback. In addition to the low-stakes LBD and DIGT activities, the OLI courses include regular high stakes short quizzes and longer exams.
Open Learning Initiative Courses in Community 7 Colleges OLI courses provide tools for educators to easily assess what students are learning and what concepts and skills students have not yet mastered. Faculty have access to the Learning Dashboard, which includes information about student participation and performance both at the class and individual student level. Instructors can use the Dashboard to view the predicted knowledge state of the class or an individual student on each learning objective. Faculty can also take advantage of the OLI course Grade book to view student performance on graded assessments.
Instructors can use the OLI course builder to customize a course by selecting and sequencing the OLI materials and controlling access and due dates. In addition to customizing their OLI course, faculty in the study were also free to decide whether or not to require students to use any or all of the OLI material and whether or not use of OLI would count toward the students’ grades.
Study Recruitment and Participation
With the assistance of Lumen Learning, the evaluation team sent email to over 1800 faculty and administrators in almost 1000 community colleges inviting them to participate in the Fall 2012 pilot study and/or Spring 2013 community college evaluation study. Faculty who teach introductory courses in Anatomy & Physiology (A&P), Biology, Psychology and Statistics were invited to participate either by 1) using the CC-OLI resources to teach their courses or 2) teaching their courses as they normally would (i.e. without using OLI). Both OLI and non-OLI participants were asked to administer pre- and post-assessments to their students, complete a faculty survey, and ask their students to complete a student survey. All faculty who agreed to participate were offered a monetary incentive. About 20% of contacted faculty responded to the initial email and indicated interest in learning more about the study and/or committed to participate. Faculty who expressed an interest were contacted by the evaluation team with a request for some additional information about their courses in order to enroll them in the study. Faculty who had originally indicated interest in the study but who did not respond to follow-up emails from the evaluation team, were contacted one to two additional times to solicit their participation.