«Intelligent assessment of distance learning q Flora Chia-I Chang Graduate Institute of Educational Policy and Leadership, Tamkang University, Tamsui ...»
Information Sciences 140 (2002) 105±125
Intelligent assessment of distance learning q
Flora Chia-I Chang
Graduate Institute of Educational Policy and Leadership, Tamkang University,
Tamsui 251, Taiwan, ROC
Received 17 November 1999; received in revised form 23 December 2000; accepted 18 May 2001
In spite of the great success of many distance learning programs, how to evaluate the
behavior of student performance in an o-line distance learning course still remains a question. If visiting Web course content is a basic requirement of performance assessment, it is possible to make an automatic tool to help the instructor and the students in terms of Web interaction analysis. This paper proposes a solution to the automation. A newly proposed mechanism, called Student±Problem±Course (SPC) table is discussed.
The mechanism is implemented on a Windows-based platform. Instructors and students both bene®t from the system of using the performance evaluation to redesign course material and to review automatically generated tutorials. Ó 2002 Elsevier Science Inc.
All rights reserved.
Keywords: Intelligent distance learning; Virtual university; Distance education;
Student±Problem table; Caution index; Web navigation analysis
1. Introduction Distance learning is a new trend of education. One of the reasons that accelerate this trend is the Internet. The revolution of Internet technology has changed our daily life. From reading news, sending e-mails, to education as well as entertainment, Internet with multimedia technologies provides a new q This project is supported by the National Science Council of the Republic of China, Taiwan, under grand number NSC 89-2511-S-032-012-X3.
E-mail address: firstname.lastname@example.org (F. Chia-I Chang).
0020-0255/01/$ - see front matter Ó 2002 Elsevier Science Inc. All rights reserved.
PII: S 0 0 2 0 - 0 2 5 5 ( 0 1 ) 0 0 1 8 3 - 9 106 F. Chia-I Chang / Information Sciences 140 (2002) 105±125 paradise for information exchange. Among these impacts, distance learning seems to be very important and interesting. One of the attractions of distance learning is its ¯exibility of instruction. Since students and instructors can be separated spatially and temporally from each other, students in a remote area and part-time students can all bene®t from this spatial±temporal ¯exibility as well as a high degree of information sharing. Yet, the revolution of Internet impacts education not only on short courses, or problem-based learning; one of the revolution results from the impact is the realization of virtual university. A virtual university, unlike a traditional university, provides distance learning both from the spatial and the temporal perspectives. Virtual university allows disabled persons or people at a remote location to learn at home. On the other hand, learning can be pursued anytime via information retrieval from the Internet, which bene®ts working people from continue education programs. In spite of these bene®ts, there still exist many diculties of the realization of virtual university, both from the technique and the management perspectives.
One of the drawbacks of current virtual university is the lack of an ecient computer network infrastructure and the integrated software system. In this paper, we propose a software environment for the use of virtual university operations, as well as a new mechanism for the automation of distance learning assessment. The system is designed for administrators, instructors, and students of a virtual university.
There are several dierences between a virtual university and a traditional university, when an instructor is delivering lectures. For instance, in a traditional university, an instructor provides on-site supervision. In a virtual university, an instructor can only provide o-site supervision due to the separation of the instructor and students. Especially, it is dicult to perform summative evaluation to student performance in a virtual university. Similarly, team teaching is easier in a traditional university, as compared to the ineciency of a group discussion over the Internet or on the telephone. In a traditional university, lecture plays the most important role. However, in a virtual university, even with a broadband communication system, which provides real-time video lecture, it is hard to have direct eye contact between the instructor and students. Moreover, the degree of interaction descends in a virtual university.
These are the drawbacks of a virtual university.
Nevertheless, virtual university has other advantages in addition to its convenience of the seamless geographical dierence. Since networks and computers are the essential facilities in a virtual university operation, it is easier to keep track of individual learning records. This advantage results in a better formative evaluation and diagnostic evaluation. Abnormal learning behaviors are easier to be discovered in the middle of instruction delivery. If appropriate guidance is provided, either automatically or semi-automatically, some students can bene®t from the virtual university environment. Moreover, virtual university allows remedy instruction. Since lectures and course materials are F. Chia-I Chang / Information Sciences 140 (2002) 105±125 107 made available in a virtual library, it is relatively easier for an individual to review missing lectures. However, to realize the above advantages of virtual university, sophisticated computer software is essential. In this paper, we propose a software system for part of the total solution.
We discuss some other research projects of distance learning software systems, before the global view of our system speci®cation is discussed. With a focus on automatic assessment, we discuss how to summarize information from three perspectives: the usability of course content, the performance of tests, and the relation between course units and tests. The proposed mechanism comes with a solution, which suggests the portion of course units that should be improved, as well as which students should be supervised. We discuss some implementation techniques, and our future work and conclusion is presented.
2. Related work
WAILE  is a Web-based intelligent learning system, which provides intelligent tools to support distance learning. Group discussion tools are proposed in CHEER . In CHEER, the concept of virtual discussion room is realized by allowing users to choose whatever communication software they need. As a consequence, dierent applications have dierent combination of communication facilities. In a paper, present virtual university (i.e., VLE ), students' classrooms are dynamically located. The discussion also points out that, active data is another challenging research issue for distance learning systems. In the MMU project , virtual university structure is divided into three levels: Micro University, Virtual University, and Macro University.
Micro University can be a software system, which assists an individual to learn from his/her digital documents. Virtual University oers such documents to many students via Web technology and digital communications. The aggregation of Virtual Universities is a Macro University. A join project to integrate many existing Virtual University software systems is currently developed by researchers from USA, Japan, Taiwan, and other countries. Distance learning can be carried out by satellite communications in a remote area. The technique issue and the evaluation of educational bene®ts of a satellite-based distance learning environment are discussed in . CORAL  is a distance learning environment for technical communication education. The system provides a course browser and a group of communication tools. Similar approaches using WWW techniques and Java applets are found in [10,11].
Distance learning systems with interactive classroom and CSCW systems are proposed [6,7]. Laboratory-based distance learning systems are discussed in [13,14]. The discussion of virtual university administration and operation issues is found in [9,17,18]. The bene®t and trend of virtual university are also discussed.
108 F. Chia-I Chang / Information Sciences 140 (2002) 105±125
3. A distance learning environment
How should a virtual university operate? According to a traditional university, instruction delivery is the most important activity. In order to realize the main activity smoothly, administration is required. A traditional university usually has some student activities and organizations, which need to be properly supported by the university's infrastructure. These are some of the important factors of a traditional university. A virtual university also focuses on instruction delivery. Due to the geographical dierence, communication tools should be ecient enough to realize instruction. Communication eciency points out an important factor: the awareness impact. Awareness indicates how strong an individual feels the existence of another person in the communication. For instance, when two persons have an eye contact, the awareness is high. When people are located in dierent cities and are talking on the phone, the awareness is lower. Sending postal mail has the lowest awareness among these three communication channels. Since a virtual university is distributed geographically, how to use computer networks to guarantee a reasonable awareness is one of the considerations. Awareness certainly aects instruction quality. On the other hand, a virtual university needs administration, which includes activities such as registration, course selection, accounting, and so on. Furthermore, a university needs to ensure that students are learning in order to meet some evaluation standard. This step is to guarantee the quality of education. A virtual university is dierent from a traditional university in that assessment is dicult. For instance, if the instructor is given an on-line test, how to make sure the student is answering the problems is a dicult task, unless a teaching assistant is sent to the remote classroom. Conclusively, we believe that, a well-considered virtual university supporting system needs to
meet the following three criteria:
· The Administration Criterion. A virtual university environment needs to have administration facilities to keep admission records, transcripts, accounting records, and so on. These administration tools should be available to administrators, instructors, and students (e.g., checking transcript information).
· The Awareness Criterion. Distance learning is dierent from traditional education. Since instructors and students are separated spatially, they are sometimes hard to `feel' the existence of each other. A virtual university supporting environment needs to provide reasonable communication tools such that awareness is satis®ed.
· The Assessment Criterion. Assessment is the most important and dicult part of distance education. Tools to support the evaluation of student learning should be sophisticated enough to avoid unbiased assessment.
In Fig. 1, the three criteria are illustrated with some activities occurring in a virtual university. The three criteria are dicult to achieve. However, it is possible to design such a virtual university environment up to a certain degree F. Chia-I Chang / Information Sciences 140 (2002) 105±125 109
Fig. 1. A virtual university system infrastructure.
of satisfaction. Fig. 1 proposes an infrastructure, with its suggested network logical connections. Assuming that, virtual university operations are running on a single campus. 1 It is feasible to construct several servers on a local area network with a central repository, which is a large scaled database containing all records. The users to such an infrastructure are administrators, instructors, and students. An administrator accesses admission and student records from the help of an administration server. This server runs some tools such as the accounting tool, the curriculum development tool, and others. Instructors use one or more course development servers, which run course design tools, lecture-on-demand tool, and others. The third type of users ± the students access a The de®nition of a virtual university can be extended to incorporate several campuses on the Internet.
110 F. Chia-I Chang / Information Sciences 140 (2002) 105±125 number of servers. The synchronous communication server runs a number of group discussion tools, which are used by instructors and students. The instructor prepares course video, which is digitized and stored as course-on-demand or video-on-demand (VoD) records. These records are video data that can be retrieved by students using dierent stations on the Internet (or other network infrastructure such as ATM network). A student can also check in or check out references from a virtual library server. Similarly, the student can register for a new course under the registration server. The purpose of dierent types of server is to balance the load among these machines. The virtual university repository contains several databases, which can be centralized or distributed. Conceptually, the repository should maintain referential integrity in a very high cohesion manner.
The architecture proposed in Fig. 1 only illustrates a general need of such a virtual university environment. One of the most important activities of virtual university operations is to deliver course material. This activity starts from the course development server. With appropriate course content provided by course designer or instructor, Web courses are stored in the Web course database, which is a part of the information repository. Students can access these Web courses from any student workstation. The performance of student learning includes the outcome of test score, as well as the interaction between students and the course material. We believe that, through appropriate and correct interaction, a student can learn better. Hence, formative evaluation is important in the environment. In order to achieve this, we have developed a technique based on Window programming skills. In Section 3.1, we discuss how to keep track of student interactions, which is used as the foundation of automatic assessment.
3.1. The Web navigation patrol