The primary purpose of monitoring and evaluation is always to seek to improve training. That is usually done by collecting information about the training and using that information to measure its effectiveness and efficiency. In that way, decisions can be made on how, and indeed whether, any piece of training or training programme may be improved.

However, it may not always be appropriate, efficient or necessary to formally evaluate every training event. Much depends on the event itself and why you want to evaluate.

In order to make the best use of resources and collect relevant information evaluators should take a Pragmatic Approach to evaluation.

What is meant by a Pragmatic Approach?

This is merely looking at a planned evaluation to determine what is the most pragmatic way to carry it out given all the factors involved.

For example:

• A training event may be carried out several times over a year with different groups of learners. Is it necessary to evaluate each event fully?
• A one off event is organised for a small number of people for a specific purpose? Is it necessary to evaluate this when there is no intention of it ever being repeated?

So how is a pragmatic approach carried out?

Back to top

Using risk analysis

Risk analysis is one technique that can allow each training solution or programme of training solutions to be assessed for evaluation against set criteria. The basic principle of risk analysis is that decisions about evaluation are more consistent. To make this work each training event is assessed for:

• its impact on magistrates carrying out their work, and
• the likelihood of the training not working effectively.

For example, if a training event is concerned with a new piece of legislation (e.g. Criminal Justice Act) then its impact on the working of the courts is HIGH and the likelihood of it not working totally effectively may also be HIGH, as it is new material. This would suggest that a significant evaluation effort should be applied to this.

Alternatively if the training event under consideration is something like Induction Training for New Magistrates, the impact would also be HIGH, but the likelihood would be Medium (MED) because this is a well established course which has been run successfully many times. That would indicate that a more focused approach to evaluation, such as just looking at one sample group if the course was to be repeated throughout the year, may be appropriate.

A further example would be something like an event that is run only once for a specific non-urgent problem. Impact would be LOW and likelihood would also be LOW as the event was not going to be repeated. In such a case, formal evaluation may not be practical and the trainer/facilitator may just wish to check understanding within the training event itself.

Back to top

From the examples given, it is possible to see that adopting such an approach to evaluation will allow evaluators to tailor the evaluation to the training activity in such a way that just the appropriate evaluation is carried out for each event.

It is possible to plot the results of the analysis on a grid diagram as shown in the illustration below:

Squares 8,9 and 6 would indicate that formal evaluation is appropriate.
Squares 3,5 and 7 would indicate that a more focused approach may be appropriate.
Squares 1,2 and 4 would indicate that informal evaulation may be appropriate.

Other factors

When making a risk assessment a number of issues linked to the training should also be taken into account.

The proposed outcomes of the training

• Are the outcomes linked to new legislation?
• Are the outcomes linked to new rules to be applied in Court?
• Are the outcomes linked to a new process being introduced?

Back to top

If so the impact is HIGH and the likelihood is dependent on the nature and scope of the training solution.
There is also the issue of what you intend to do with the outcomes and who you will report to, each of which may influence your decision on the risk.

The proposed extent of the training

• How many people are involved?
• What is the cost of the training?

If either of these characteristics are HIGH i.e. large numbers or high costs (in proportion to overall budget), then impact and likelihood is HIGH if large numbers of learners are involved.

The proposed delivery method

Some forms of delivery are inherently more 'risky', for example, technology-based delivery may be more risky therefore the likelihood is HIGH.

Face to face delivery may be inherently less 'risky', so likelihood may be LOW.

The content of the training

Generally, the more complex the content of the training the higher the likelihood of failure.

Back to top

The resources available

As with all training, the availability of resources such as, evaluator time, is a factor in deciding how and when to evaluate. It is for local decision makers to make these decisions as they are most aware of the resources available to them. However, by carefully using the Evaluation Guidance, it may be possible to select an appropriate methodology that works within the resource constraints which are available to the evaluator.

In all cases, effective planning of the evaluation strategy is more easily carried out at the earliest possible stage in the training cycle.

Conclusion

By systematically considering these factors in a risk-based approach, it is possible for the evaluator(s) to design an evaluation that is appropriate to the event and the environment in which the event takes place. This is the purpose of the Pragmatic Approach.

Back to top