Magistrate Monitoring and Evaluation

Evaluation Guidance: Evaluation Tools

Questionnaires Interviews Focus Groups
Observations self assesment Self-assessment Verbal Feedback
self reflection Self-reflection

Interviews

Using interviews to evaluate:

The Validation of the event.
Whether Objectives of the training have been met.
Learning - what people have learned from the training.
Long term Impact of training on the work of the individual or group.

Different approaches
Face to face or via telephone?
What you need to do before conducting an interview.

Different approaches

There are 3 basic approaches to interviews:

Unstructured | Semi-structured | Structured

Unstructured interviews

General

You can use an unstructured approach when you want a free-flowing discussion with few limitations around a generalised question, such as "What aspects of the training have you been able to apply so far?". An unstructured approach will often uncover information that you had not considered, but may be relevant.

Good for

Where you wish to draw out information that you may not have considered up to that point, but which is relevant to the information you require.
Where you are unsure of the outcome but require participant's views on it.
Where the participants are fairly sophisticated or experienced and share a common agenda.
Where you feel that the information you require is complex and multi-layered.
Where the diversity of the interviewees is liable to lead to multiple or variable opinions.

Approach

Use simple, open questions, that give a high level lead into the information you require such as, "Tell me your opinion of the training".
Be prepared to ask the same questions in different ways to get interviewees to provide the detail that you want.
Allow the discussion to flow and prompt when it does not.
Be prepared to ask qualifying questions such as, "You gave this response how can you illustrate that?" or "Give me an example of how that has worked for you?"

Hints and tips

Prepare the general questions which you want to be answered to allow as much discussion as possible.
Ensure interviewees are clear about the purpose of the interview and check their understanding.
Be aware that interviewees may be hard to keep on track and be prepared to intervene to bring the interview back on course.
Set a time limit on the interview so that participants are able to judge how long they will be involved.

Analysis

The information that you gather is almost always qualitative in nature and will require careful analysis. A suggested approach for this analysis is:

Review findings/views to identify suggestions for improvement. Group similar points from interviews together, by looking for common themes and phrases.
Assemble and report the results of this grouping exercise before making a separate analysis of the information. It is often a good idea to tabulate these results or use another graphical method such as a distribution chart to help you see the overall picture.
Write up a report highlighting key points and issues raised. Always include 'minority' information with this so that the whole picture can be seen.
Present your analysis with the appropriate qualifying statement relating to the sample size and make up of your interview group.
The size and composition of your sample of interviewees should usually reflect the size and makeup of the trainee group. However, in some circumstances you may wish to use a specific group of interviewees for example; new magistrates, experienced magistrates etc.

Semi-structured interviews

General

You should use a semi-structured approach where you want a more focused discussion about more detailed aspects of the training, e.g. the specific knowledge learned. It can also be used where it is important to gather information about a specific aspect of the training, e.g. the skills learned from a role play exercise, but you want to retain some flexibility to explore any areas that crop up in discussion. The semi-structured approach involves the formulation of a semi-structured brief for the interviewer, which consists of general questions such as "Do you now feel able to meet the objective for session two which was...? Supported by a series of supplementary questions relating to the detail of the objective.

Good for

The semi-structured approach is good for circumstances where you have a good idea of the information you want to find out, but want to retain some flexibility so that discussions may go into areas that you have not considered. This approach gives the interviewer more control over the discussions than with an unstructured approach, as the area to be covered has been formulated in the brief. The range of divergence from the brief is down to the interviewers who may find themselves moving into areas which they may find interesting and relevant, but which they had not anticipated.

Approach

Use two levels of question in the brief:

Primary questions that may be closed such as "Were you able to meet this objective?"

Supported by

Supplementary questions to elicit further information such as "If not, why not?" or "Can you give me an example to illustrate what you mean by that?"

The interviewer can exert control over the discussions by the way they use the supplementary questions.

Hints and tips

The semi-structured approach requires careful formulation of the brief to cover all the areas that you want to gather information on. The brief should include:

Introduction - explain purpose, cover confidentiality, gain permission to tape/make notes.
Warm up - easy non-threatening, non-controversial questions.
Main body - cover the main purpose in a logicial progression.
Cool off - a few straightforward questions at the end to defuse any tension.
Closure - thank you and goodbye, remembering that often points are made just as you leave, the hand on the door syndrome.
Set a time limit on the interview so that participants are able to judge how long they will be involved.

Interviewers must be sufficiently experienced and confident to step outside the brief on occasion if a response seems to require further discussion.

Analysis

The same approach as is suggested for unstructured interviews is recommended. However, because of the more structured approach it is usually easier to group the findings around the primary questions. In addition, analysts should be aware that although quantitative data may emerge from the discussions, one should always qualify this with reference to the sample group, on what basis it was selected, and including any minority information. For example, if a group of twelve people all have the same view it could be erroneous to extrapolate that view to all of the people who had undergone the training.

Review findings/views to identify suggestions for improvement. Group similar points from interviews together, by looking for common themes and phrases around the primary questions.
Assemble and report the results of this grouping exercise before making a separate analysis of the information. It is often a good idea to tabulate these results or use another graphical method such as a distributive chart to help you see the overall picture.
Write up a report, highlighting key points and issues raised. Always include 'minority' information with this so that the whole picture can be seen.
Present your analysis with the appropriate qualifying statements relating to the sample size and make up of your interview group
The size and composition of your sample of interviewees should usually reflect the size and makeup of the trainee group. However, in some circumstances you may wish to use a specific group of interviewees for example; new magistrates, experienced magistrates etc.

Structured interviews

General

You may use a structured interview approach where you need information about specific issues and areas of detail. For example, you may wish to follow up on a specific set of activities within a training programme, or to check that an objective or objectives relating to a specific topic such as sentencing provisions have been met. You are unlikely to get the richness of information that comes from semi-structured approach but in some circumstances this is appropriate.

Approach

As with all types of interviews proper preparation of the facilitator's brief is important. In a structured approach the brief is constructed using specific targeted questions aimed at eliciting the closely defined information that is required. The brief may be a mixture of question types including closed questions, coded responses, classification question types etc. It is sometimes possible to ask the same question more than once, using different types to validate the answers. This is often done by professional researchers, but may be beyond the scope of all but very experienced interviewers.

Hints and tips

You may formulate your brief in the same way that you would a semi-structured interview brief, with primary questions supported by supplementary questions to drill down to the level of detail you require.
The question types may be varied to allow the interviewer to explore issues in depth. For example, you may wish to ask a direct question with a yes/no answer, but follow that up with a question that asks the interviewee to rate a response by using a discrimination scale.
Interviewers must be aware that a consistent view on any question may not be reached and that it is important to record all views in order to arrive at a full picture.
Set a time limit on the interview so that participants are able to judge how long they will be involved.

Analysis

The information that you gather will often be a mixture of qualitative and quantitative data. As with unstructured and semi-structured interviews the suggested approach would be as follows:

Review findings/views to identify suggestions for improvement. Group similar points from interviews together, by looking for common themes and phrases.
Assemble and report the results of this grouping exercise before making a separate analysis of the information. It is often a good idea to tabulate these results or use another graphical method such as a distribution chart to help you see the overall picture.
Write up a report highlighting key points and issues raised. Always include 'minority' information with this so that the whole picture can be seen.
Present your analysis with due reference to the sample size, the selection of the group etc.

Face to Face interviews

Good for

Allowing flexibility if questions require follow up.
Allows the interviewer to pick up non-verbal clues.
Enables the interviewers to strike up a rapport with the interviewee.
Where complex responses are expected.
May be resource intensive.

Approach

Can be used in all formats.

Analysis

Requires detailed note taking.
Mixture of qualitative and quantitative data together with non-verbal "impressions" can require careful and detailed analysis.

Telephone interviews

Good for

Large numbers of interviewees.
Where resources are an issue.
Where non-verbal clues are not needed.
Where limited flexibility is needed.

Approach

Can be used in all formats.
However, more often used in a structured approach.
Information can be recorded on paper or on tape (with the appropriate equipment and safeguards).

Analysis

Requires detailed note taking.
Mixture of qualitative and quantitative data can require careful and detailed anaylsis.

What you need to do before conducting an interview

Be sure that an interview approach is the right one for the level of evaluation you are looking at.
Decide what you want to find out and select the approach to best collect that data.
Identify an experienced interviewer.
Decide who and how the information is to be recorded.
Decide how you will select participants for the interviews. This may involve consideration of a suitable sample size, how that sample group is made up e.g. do you want experienced magistrates, or new magistrates? It is a good idea to select a group that reflects the composition of the complete training group.
Allow plenty of time for analysis of the information - it may take longer than you think.

Using interviews for validation

While it is possible to use interviews for validation, it is more usual to use questionnaires. There may be instances when the use of an interview may be appropriate, for example when a large amount of detailed information is required. In this scenario the use of a standard or semi-structured approach may be the approach to use. However, you would need to be convinced that the extra time and effort involved would give you sufficient added value to use these approaches rather than questionnaires. An example of a section of a semi-structured interview brief for validation appears below:

Primary Question (general points)	Supplementary Questions (detailed points)
Was the design of the training effective and easy to follow?	Do you feel that there was a clear introduction to the training?
	Do you feel that the training objectives were clear?
	Do you feel that the structure of the training was logical and easy to follow?
	Do you feel that the content of the training was relevant?
	Do you feel that the training met equal treatment and diversity guidelines?
	Do you feel that clear language was used in the training materials?
	Do you feel that there was appropriate interaction in the training?

Using interviews to evaluate whether objectives have been met

It is relatively common to use an interview to evaluate whether objectives have been met even though questionnaires are the most common way to evaluate this.
An interview may give you more detailed information and allow a more flexible approach than a questionnaire.
As with all evaluation using the pragmatic approach, some form of risk analysis should be carred out in order to make sure that an interview is appropriate in the given circumstances.
It may be possible to use any of the three approaches, however, the example of a section of semi-structured interview brief is given below:

Primary Question (general points)	Supplementary Questions (detailed points)
Was the design of the training objectives effective and easy to understand?	Do you feel that the training objectives were clear?
	Do you feel that the structure of the training objective was logical and easy to follow?
	Do you feel that the content of the training was relevant to the training objectives?
	Do you feel that the training objectives met equal treatment and diversity guidelines?
	Do you feel that the exercises enabled you to meet the training objectives?
Was the delivery of the training objectives effective and easy to understand?	Do you feel that the trainer/facilitator explained the training objectives clearly?
	Do you feel that the trainer/facilitator presented and explained the information, exercises and demonstrations effectively?
	Do you feel that the trainer/facilitator made clear the link between the training objectives and the exercises?
Was the training environment suitable for meeting the objectives?	Do you feel the training room helped or hindered your achievement of the training objectives?
	Do you feel that the equipment helped or hindered your achievements of the training objectives?

Using interviews to evaluate what has been learned by participants

Interviews may be used in this context, particularly when wishing to determine exactly what people have learned from the training. This is not just about whether objectives have been met but also what other learning has occurred. For example, has there been any additional learning as a result of meeting the learning objectives, such as, a different attitude/skill or greater confidence about a certain area of work? It is also important that the interview doesn't become an oral test of knowledge. This can be achieved if desired by Self Assessment testing or the use of learning checks. In this context an unstructured or semi-structured approach is often deployed. An example of a semi-structured brief appears below.

Primary Question (general points)	Supplementary Questions (detailed points)
Did you learn any new knowledge/skill(s) as a result of the training?	What new knowledge have you learned?
	What new skill(s) have you learned?
	What do you now do differently as a result of the training?
Do you feel competent in applying the new knowledge/skill(s)?	Did the training fully equip you with the new knowledge/skill(s) required?
	Are there any gaps in the training?
	What improvements would you make to the training to make it more effective?
	Are there any barriers to you applying fully the new knowledge/skill(s)?
Have you applied the new knowledge/skill(s)?	Have you been able to apply fully your learning from the training?
	How often have you applied the new knowledge/skill(s)?

Using interviews to evaluate the impact of training

When evaluating the impact of training using interviews, it can be effective to use any of the three approaches, depending on what information you want to collect.
An example of a semi-structured interview brief appears below.

Primary Questions (general points)	Supplementary Questions (detailed points)
Do you consider yourself now competent and confident in using the sentencing guidelines effectively in court?	Did the training prepare you for actually carrying out sentencing under the new framework?
	Were you aware of the likely impact of the new sentencing framework on how you would carry out your sentencing functions before your first sitting?
	Was the guidance in the training effective? Did it cover the scenarios that you have come across in court?
	Have there been any scenarios that you have come across that are not covered in the guidance or the training?
	If so, how were you able to deal with them?
Do you think that the supporting materials supplied to you (forms, handouts etc) were of use now that you have had chance to apply the training?	Specifically, do you think that the sentencing reasons form is of use to you?
	If not, what changes do you think need to be made in light of your experience?
	Do you think that the process itself transfers well to the court environment?
	It not, what changes would you like to see?
Now that you have had chance to apply the training in the courtroom, what is your opinion of the success of the training programme?	Do you feel that the training needs any additional material or exercises?
	If so, what would you recommend?