Magistrate Monitoring and Evaluation

Evaluation Guidance: Evaluation Tools

Questionnaires Interviews Focus Groups
Observations self assesment Self-assessment Verbal Feedback
self reflection Self-reflection

Observations

Using observations to evaluate:

Whether Objectives of the training have been met.
Learning - what people have learned from the training.
Long term Impact of training on the work of the individual or group

General advice.
Different approaches.
What you need to do before carrying out observations.

General advice

Observations can easily be used in a classroom situation. However, if being used in court for evaluation purposes it is imperative that its use does not jeopardise judicial independence. Important -Used in the court environment it is essential that it is clear that the observation is to gauge the learning or the impact of the training not of the individual magistrate.

Different approaches

There are 3 basic approaches to observations:

Informal | Semi-structured | Structured

Informal observations

General

Doesn't usually involve preparation of any form of observation brief. It is usually opportunistic, for example you may observe the behaviour of a magistrate in court as part of your normal advice giving role, or observation of how a magistrate interacts with colleagues, during pre and post court sessions. The most difficult of methods to use systematically, this approach provides general impressions and may require more detailed follow up before any hard and fast conclusions can be drawn. However, it can often be a successful pre-cursor to a more in depth evaluation method being used. It can also be a useful indicator of the success or otherwise of training when carried out over several occassions.

Good for

Where you wish to observe any outcomes that you may have not considered up to that point, but which are relevant to the information you require.
Where you are unsure of the outcome but need to observe it.
Identifying potential issues that may need further exploration.

Approach

Usually ad hoc, no formal documentation required but remember to qualify any reporting by reference to the informal nature of the observation. Usually used as justification of a more detailed evaluation.

Hints and tips

Be aware of the limitations of this method
Always explain to the subjects of the evaluation what you are doing and why, unless it is part of the normal working routine
Always discuss your impressions with the person who has been observed
Remember, you are evaluating the performance of the training not the performance of the person.

Analysis

Analysis of this information is problematical because of its nature
It is usually the best policy to just record your observations in a factual way as a basis for further work or discussion.

Semi-structured observations

General

Always use a brief which contains primary self questions - usually closed requiring yes/no answers, such as "Did the magistrate treat the defending solicitor in an assertive way?"
In addition always provide the observer with supplementary self questions such as - "How did the assertiveness compare with the guidelines set out in the assertiveness training packages?"
Always remember you are evaluating the training not the individual.
It is important that the observer(s) are familiar with the desired behaviour required by the training and are familiar with the brief.

Good for

This approach enables you to use different levels of self-questions to aid the observation, raising specific issues and focusing responses, in a way that is not as easy to achieve with an unstructured approach. For example, you can see whether the magistrate explained the reasons for the sentence clearly and supplement that by observing how the defendant reacted to the explanation.
This approach is particularly effective for evaluating the impact of training on magistrate behaviour.
It also allows observers to pick up non-verbal clues and assess the confidence of the subject.

Approach

On the observation sheet, use primary high-level self-questions (e.g. those requiring a "yes/no" reply).
Also use supplementary self-questions, to point the observer towards more detailed information about specific points.

Hints and tips

The preparation of self-questions on the observation sheet is crucial to effectiveness with this approach.
Be aware that the observation may cover areas, which the observer has not already thought of.
Formulate your observation structure.
Observers should be independent and must not take part in the training.

Analysis

Review observations to identify possible areas for improvement. Group similar points from observations together.
Report the results of this grouping exercise before making a separate analysis of the information.
Write up a report highlighting key points and issues raised.
Present your analysis with the appropriate qualifying statements. To include an appreciation of how the observation process may affect the information gathered.

Structured observations

General

This approach is most used where you want to make an observation of relatively large numbers, or a very specific and detailed aspect of some training, where consistency is paramount. The formulation of a tightly structured brief, allows little flexibility and restricts the observer's ability to step outside the brief and record unrelated observations.

Approach

Observation sheet uses specific, pre-prepared self-questions (similar to self-completion questionnaires). Mix of:

closed questions (e.g. with yes/no answers),
coded responses (e.g. on a rating scale) and
open questions.

Good for

This approach may be used when responses to specific areas of detail are required. For example, you may wish to investigate the impact of specific parts of the training and have detailed questions that need to be answered by observing behaviours.

Hints and tips

The preparation of self-questions on the observation sheet is crucial to effectiveness with this approach.
Be aware that the observation may cover areas, which the observer does not feel are relevant.
Observer should usually be independent but must be familiar with the desired outcomes of the training.

Analysis

Analysis of information is relatively straightforward as the data will consist of numerical data (yes/no answers), with some qualitative data from supplementary questions. With the qualitative data the process is:

Review findings/views to identify suggestions for improvement. Group similar points from observations together, by looking for common themes and phrases.
Assemble and report the results of this grouping exercise before making a separate analysis of the information. It is often a good idea to tabulate these results or use another graphical method such as a distribution chart to help you see the overall picture.
Write up a report highlighting key points and issues raised. Always include 'minority' information with this so that the whole picture can be seen.
Present your analysis with the appropriate qualifying statements relating to the sample size and make up of your interview group.
The size and composition of your sample of interviewees should usually reflect the size and makeup of the trainee group. However, in some circumstances you may wish to use a specific group of interviewees for example; new magistrates, experienced magistrates etc.

When using small samples for observation it may be difficult to draw definite conclusions or sufficiently robust information to formulate actions.

What you need to do before carrying out observations

Be sure that the observation approach that you want to use is the right one for the type of evaluation you are looking at.
Decide what you want to find out and select the approach to best collect that data. This may seem self evident but it is surprising how often an approach is decided before the information is thought about.
Identify an experienced observer.
Decide how you will select participants for the observation(s). This may involve considerations of a suitable sample size, how that sample group is made up e.g. do you want experienced magistrates, or new magistrates? It is a good idea to select a group that reflects the composition of the complete training group.
Allow plenty of time for analysis of the information - it may take longer than you think.

Using observations to evaluate whether objectives have been met.

The use of observations for establishing if objectives have been met, where training is in a classroom situation, is relatively straightforward. You will need a clear understanding of the training material objectives of the key learning points. You can observe if these have been met by observing whether delegates confirm understanding and objective.

Using observations to evaluate what has been learned by participants

Observations may be used in this context, particularly when wishing to determine exactly what people have learned from the training. This is not just about if objectives have been met but also what other learning has occurred. For example, have there been any other things that have arisen as a result of meeting learning objectives, such as, a different attitude that may have been encouraged, or greater confidence in carrying out certain functions. An example of a semi-structured observation brief appears below.

Primary Self Question (general points)	Supplementary Self Questions (detailed points)
Did you observe the magistrate display any new knowledge/skill(s) that they haven't displayed before that may be related to the training?	What new knowledge was displayed?
	What new skill(s) were displayed?
	What does the magistrate now do differently as a result of the training?
	Did the magistrate apply fully their learning from the training?
	If not, what was not applied and what were the reasons for that? For example: Was there no opportunity? Did the magistrate miss or fail to recognise the opportunity?

Using observations to evaluate the impact of training

When evaluating the impact of training using observations, it can be effective to use any of the three approaches, depending on what information you want to collect. An example of a semi-structured interview brief appears below.

Primary Question (general points)	Supplementary Questions (detailed points)
Did the magistrate now appear competent and confident in using the sentencing guidelines effectively in court?	Did the magistrate follow the new framework?
	Did the magistrate appear well prepared for their first sitting using the new guidelines?
	Did the magistrate appear confident in using the new guidelines?
	Did the magistrate give the appropriate sentence for the circumstances?
	If not, what were the things missed out?

Important - the purpose of the evaluation is to measure the impact of the training upon the work of the magistrate NOT to measure the individual magistrate's performance