Review by: Park, Jonghwi (2005-08-31)

This article compares a traditional lecture/recitation format with an online/recitation format in order to determine whether online courseware can replace large lectures without decreasing learning outcomes.

The online version of the course called Causal and Statistical Reasoning consisted of three components: 1) 17 lessons or concept modules, 2) a virtual lab simulating social science experiments, and 3) a database of over 100 cases. The traditional lecture version consisted of two lectures per week. Both versions included weekly face-to-face recitation sessions for questions and further discussion.

The authors used the two versions of the course materials to conduct five experiments involving over 650 students from the University of California at San Diego and the University of Pittsburg. The five experiments were spread across five semesters allowing the authors to modify later experiments based on lesson learned in earlier ones. For example, based on lessons learned in the first two experiments: 1) students were allowed to choose the method of delivery as the results in the previous semi-randomized design showed selection bias on final scores, and 2) attending recitation sessions was made obligatory because whereas 85% of the lecture-students attended only 10% of the online-students did. The authors collected and analyzed two kinds of data to test their hypothesis: 1) learning outcomes of online vs. lecture students, and 2) student behavior or strategies in the online setting.

The analysis of learning outcomes indicated that the online courses are as effective or significantly more effective than large lecture courses, which supports the claim that online courses can replace large lecture courses. In addition, a regression analysis showed that recitation attendance most strongly predicts final exam performance (2% more per attendance). This means that regular face-to-face meetings are critical to the learning process even in online settings.

The analysis of student behavior in the online setting was based on pretest, quiz and final exam scores, voluntary questions and the use of printouts. A path analysis showed that frequent use of printouts predicts negative effects on both quiz and final exam scores whereas pretest scores and frequent voluntary questions predict positive effects on quiz and final exam scores. The authors tentatively explained that students who printed out the materials would skip the interactive simulations/animations/lab activities as well as voluntary comprehension checks embedded in the online courses and thus, their learning outcomes were lower than those of students who repeatedly visited online and took full advantage of online materials.

The authors suggest that future versions of online courses should use more advanced technology to record student moves and to trace their paths through the online material in order to help them guide what to do more

This article is notable for two reasons.

1. It showed a nice trajectory of research in which the design of earlier experiments affected the design of later ones.
2. It used sophisticated statistical techniques such as a MANOVA for mean differences and regression and path analyses of many online course factors to prove the superiority of online courses.