Review by: Hasanbegovic, Jasmina (2004-09-02)

Most discussions of the effectiveness of computer-based learning refer to the well-established meta-analyses conducted by Kulik and Kulik. Both are experts in evaluation at the university of Michigan and investigated several meta-analyses to analyse the effects of computer-based instruction (CBI) on students.

In this paper, they present the results of findings from 254 controlled evaluation studies that compared student learning in classes taught with and without CBI covering learners of all age levels. They argue that reviewers must take into account the results from numerous studies carried out in different places, at different times, and under different conditions to be able to reach general conclusions. At the beginning of the paper, they introduce several existing positive results on computer-based instruction of different meta-analyses, but point out the timeworn publications these analyses were based on. As computers and their role for instruction have changed, earlier reviews have to be updated to determine whether the record of effectiveness of CBI has changed with the development of new software in recent years.

The authors explain their meta-analytic approach used for this review for which studies of an issue through objective and replicable searches have to be located and coded. The study outcomes have to be coded on a common scale and statistical methods have to be used to relate study features to outcomes. They reveal basic methodical issues like data sources, outcome measures, study features, and unit of statistical analysis in great detail.

The results show a higher examination average of the students in the CBI classes. The moderate average effect size (.30) determined in the 248 studies indicates that the average student from the CBI classes outperform 62% of the students from the conventional classes. However, the effects varied in magnitude from study to study according to the different publication sources, controls for teacher effects, and different durations.

In the discussion section, the authors analyse similarities and differences in the findings between the current study and former studies, emphasizing the relationship between achievement outcomes of studies and study duration, control for instructor effects, and publication source. Furthermore, they discuss treatment features like length of treatment and type of computer use as most strongly related to effect size in this analysis.

This meta-analysis gives methodological insight into the “no significant difference” of computer-based instruction compared to traditional settings. It also discusses different treatment features’ relations to effect size as an explanation for the no significant phenomenon. This article is a must-read for every evaluator. A re-investigation of the last ten years would be of great importance and interest.