Review by: Dreier, Matthias (2004-06-10)

Since 1998 Rolf Molich’s usability consultancy “Dialog Design” has conducted four comparative usability evaluations. This article summarizes the results of the second evaluation. Molich et al. compare the results of usability evaluations of the Microsoft Hotmail website conducted by nine independent organisations. The organisations were industry usability labs, university labs and two student teams. The teams were free to choose the methodology, the number of test participants and the user tasks. The results show that 75% of the 310 reported usability problems were uniquely reported by one team, 29 of which are considered to be serious usability flaws. Only one problem was reported by more than six teams. The authors conclude that the results of usability tests are far from being similar, even if multiple organisations choose to apply the same method to evaluate the same tasks.

Comparative studies in the field of Human-Computer Interaction are rare. Therefore the study of Molich et al. is an important contribution to the research in the field. Some of their findings may be arguable because the method chosen by the authors might be inappropriate, e. g. leaving the choice of the usability method and the test tasks to the teams. However, even the tasks inspected by most of the teams revealed few overlaps of the problems found. Rules of thumb like Jacob Nielsen’s “a test with five users reveals 85% of the problems” are misleading. The publication also unveils methodological flaws of some of the teams, for instance giving hidden clues in the task description.

Although Molich et al. did not compare the evaluation of elearning websites, the results are very likely to apply to these websites as well. The authors conclude that the effectiveness of a usability test depends on the chosen tasks, the methodology, and the persons in charge of the test. The article primarily addresses usability professionals and development teams. But the authors’ recommendations deduced from their findings are useful for customers of usability labs as well.