Review by: Meier, Christoph (2005-06-06)

Return on investment: The holy grail of training analytics?

Employees and their qualification are frequently regarded as key to the success of businesses, public services and other organizations. Nevertheless, investments in education and training are frequently regarded as easy targets for cost reduction programs – particularly in economically difficult times. In consequence, methodologies and procedures that allow to determine the value created by training and education take on particular importance. Not surprisingly, the return on investment for training is, at present, widely considered a key performance indicator to be tracked by training analytics (in German speaking countries referred to as “Bildungscontrolling”). One prominent approach to the determination of ROI in training is the one proposed by Jack Phillips.

Two intertwined proceses: “performance analysis” and “ROI-process”

The starting point for Jack Phillips is the conception of four levels of evaluation of training efforts proposed by Kirkpatrick in the late 1950s. These levels are

1. reaction (i. e. the satisfaction of learners with the training offering)
2. learning (i. e. changes in knowledge or attitude)
3. behaviour (i. e. transfer of what was learned and resulting changes in behaviour on the job)
4. results (i. e. organizational impact, effects on production, services, etc.)

Based on these levels, Phillips proposes a fifth level for evaluation of training programs:

5. Return on invest (i. e. the ratio of net benefits to program costs).

In addition to this, Phillips develops a procedural model for performing ROI analyses for training programs. This model, the “ROI-process”, comprises the following steps:

1. Analysis of business problems/opportunities
2. Identification of business needs
3. Identification of job performance needs (includes collection of performance data)
4. Identification of preferred forms of learning
5. Formulation of an evaluation strategy
6. Collection of post program data
7. Isolation of effects resulting from the training program
8. Conversion of effects into monetary values
9. Tabulation of program costs
10. Calculation of return on invest

A graphical representation of the ROI-process reveals that it involves moving down and up the levels of evaluation mentioned above:

It is important to collect data and to document the current status on different evaluation levels prior to program implementation (i. e., prior to conducting the training). Only on the basis of data on levels 3 and 4 is it possible to compare situations before and after training. For the isolation of the effects of training (relative to the effects of the numerous other changes in an organization that may have been going on at the same time) Phillips proposes control groups, trend analyses or assessments by key personnel such as management. For the conversion of effects identified into monetary value, methods such as marginal profit contribution, cost of quality, cost of employee time or estimates by participants can be employed. Only on the basis of these data is it possible to calculate the return on investment for training. In his book Phillips illustrates the ROI-process by discussing a case on sales training.

Critical appreciation

Calculating the return on invest for training programs makes sense only insofar as success criteria are available that can be quantified and for which a monetary value can be calculated with reasonable effort and confidence. As long as “hard” criteria for success are available, such as

• the number of units produced/sold,
• the time required for manufacturing a product or
• the time required for providing a service,

this appears feasible. However, in case only “soft” criteria for success are available, such as

• the motivation of employees and their commitment to the company,
• the service orientation of employees or
• the creativity of employees,

this becomes much more difficult.

It comes as no surprise, therefore, that sales trainings are popular examples of model calculations of the return on invest of training programs. Calculating the ROI for training programs whose effects do not show short term but only after longer periods of time (e. g., management development) is much more difficult to achieve with a reasonable effort and with reasonable confidence. What is more, the calculation of the ROI for a training program does not provide any indication as to what needs to be done to improve such a program in order to make it more successful.