Review by: Nash, John (2004-09-13)

Since 2002, within United States policy circles and elsewhere, there has been a growing call for enhanced evidence that educational innovations are working. E-learning initiatives have not escaped this scrutiny. In the United States, federal laws now state that teams developing educational innovations with federal money engage in “scientific research” and “evidenced-based practices” to prove their interventions are working. In this compelling essay, David Berliner points out that these terms are really just code words for “randomized experiments,” and to rely only on randomized experiments to determine the worth of educational innovations (including e-learning initiatives) is a misguided practice.

Berliner does not say that randomized experiments are a bad idea. Quite the contrary, they are a highly effective tool for determining quality in the right situation. But to suggest that randomized experiments are the only form of research that is “scientific” is to suggest that all other methods are somehow lacking. This is where Berliner makes his well-expressed point. Somehow in recent years, policy makers, as well as the lay public, have determined that the methods of the “hard sciences” are superior to the “soft sciences” and therefore the latter should adopt the former’s approaches. Berliner eloquently turns this notion on its head: educational research (typically labeled “soft”) is actually the hardest science of them all. To strictly utilize randomized trials in educational research to examine the quality of interventions does little to take into account the context, interactions and other points of invalidity with which teaching and learning settings are fraught.

As Berliner states:“Doing science and implementing scientific findings are so difficult in education because humans in schools are embedded in complex and changing networks of social interaction. The participants in those networks have variable power to affect each other from day to day, and the ordinary events of life (a sick child, a messy divorce, a passionate love affair, migraine headaches, hot flashes, a birthday party, alcohol abuse, a new principal, a new child in the classroom, rain that keeps the children from a recess outside the school building) all affect doing science in school settings by limiting the generalizability of educational research findings. Compared to designing bridges and circuits or splitting either atoms or genes, the science to help change schools and classrooms is harder to do because context cannot be controlled” (p. 19).

Berliner conclusions are extremely relevant for the e-learning community. If we as e-learning developers accept the fact that there are unique complexities to take into account when learners use our tools, then a single-minded approach to researching the impact on the learner is faulty.

The article contains references within the text to large scale school reform interventions from the United States, many of which may be unfamiliar to researchers outside the U.S. Readers should not focus on the names of the projects, or what their goals are, but rather that, like all e-learning initiatives, the examples Berliner cites are embedded in a social context, and therefore the methods of science that are utilized to research their impact must be chosen carefully.