Magistrate Monitoring and Evaluation

Evaluation Guidance: Evaluation Tools

Questionnaires Interviews Focus Groups
Observations self assesment Self-assessment Verbal Feedback
self reflection Self-reflection

Focus Groups

Using focus groups to evaluate:

The Validation of an event
Whether Objectives of the training have been met
Learning - what people have learned from the training
Long term Impact of training on the work of the individual or group

General advice.
Different approaches.
What you need to do before running a focus group.
Focus group protocol.

General advice

The most popular use of focus groups is to evaluate the impact of training. It can also be used to establish if learning has taken place/been applied. It is less effective, from a cost point of view, for validation and objectives.

Different approaches

Focus groups are essentially group interviews where a representative sample of learners take part in a discussion to explore a number of questions or issues relating to a particular training event or events. All focus groups require a facilitator to run them so that all participants have a chance to air their views, and to make sure that no one person or group within the focus group is allowed to dominate the discussion. There are a number of rules and advice for the successful running of focus groups which are drawn together in the focus group protocol, which is attached to this document. There are 3 basic approaches to focus groups:

Unstructured | Semi-structured | Structured

Unstructured focus groups

General

You should use an unstructured approach where you want a free-flowing discussion with few limitations around a generalised question, such as "What did you learn from the training?" An unstructured approach will often uncover information that you had not considered and facilitators will need to prepare for this.

Good for

Where you wish to draw out information that you may not have considered up to that point, but which is relevant to the information you require.
Where you are unsure of the outcome but require participant's views or perceptions of it.
Where the participants are fairly sophisticated or experienced and share a common agenda.
Where you feel that the information you require is complex and multi-layered and therefore not suited to a more structured approach.
Where the diversity of the interviewees is liable to lead to multiple or varied opinions.

Approach

Use simple, open questions, that give a high level lead into the information you require such as "tell me your opinion of the training".
Be prepared to ask the same question in a number of different ways to promote discussion.
Allow the discussion to flow and prompt when it does not.
Be prepared to ask qualifying questions such as "You gave this response how can you illustrate that", or "Give me an example of how that has worked for you?"

Hints and tips

Prepare the general question which you want to be answered carefully to allow as much discussion as possible.
Ensure interviewees are clear about the purpose of the focus group and check their understanding.
Be aware that interviewees may be hard to keep on track. Be prepared to intervene to bring the interview back on course should the discussion wander too far off track or begin to contravene the focus group protocol.
Set a time limit on the interview so that participants are able to judge how long they will be involved.

Analysis

The information that you gather is almost always qualitative in nature and will require careful analysis. A suggested approach for this analysis is:

Review findings/views to identify suggestions for improvement. Group similar points from interviews together, by looking for common themes and phrases.
Assemble and report the results of this grouping exercise before making a separate analysis of the information. It is often a good idea to tabulate these results or use another graphical method such as a distribution chart to help you see the overall picture.
Write up a report highlighting key points and issues raised. Always include 'minority' information with this so the whole picture can be seen.
Present your analysis with the appropriate qualifying statements.

Semi-structured focus groups

General

You should use a semi-structured approach where you want a more focused discussion about more detailed aspects of the training, for example, you are looking at detailed aspects of training such as knowledge learned and where it is important to gather info about a specific aspect of the training such as the skills learned from a roleplay but want to retain some flexibility to allow you to explore any areas that crop up in the discussion. The semi-structured approach involves the formulation of a semi-structured brief for the facilitator, which consists of general questions such as "Do you now feel able to meet the objectives for session two, which was..?" Supported by a series of supplementary questions relating to the detail of the objective. The role of the facilitator is crucial to the running of this type of focus group.

Good for

The semi-structured approach is good for circumstances where you have a good idea of the information you want to find out, but want to retain some flexibility so that discussions may go into areas that you have not considered. This approach gives the facilitator more control over the discussions than with an unstructured approach as the areas to be covered have been formulated in the brief. The range of divergence from the brief is down to the facilitators who may find themselves moving into areas which they may find interesting and relevant.

Approach

Use two levels of question in the brief:

Primary questions that may be closed such as "Were you able to meet this objective?"

Supported by

Supplementary questions to elicit further information such as "If not, why not?" or "Can you give me an example to illustrate what you mean by that?"

The facilitator can exert control over the discussions by the way they use the supplementary questions.

Hints and tips

The semi-structured approach requires careful formulation of the brief to cover all the areas that you want to gather information on. The brief should include:

Introduction - explain purpose, cover confidentiality, gain permission to tape/make notes.
Warm up - easy, non-threatening, non-controversial questions.
Main body - cover the main purpose in a logicial progression.
Cool off - a few straightforward questions at the end to defuse any tension.
Closure - thank you and goodbye, remembering that often points are made just as you leave, the hand on the door syndrome.
Set a time limit on the interview so that participants are able to judge how long they will be involved.

Facilitators must be

Sufficiently experienced and confident to step outside the brief on occasion if a response seems to require further discussion.
Experienced and confident enough to bring discussions back on track to the brief where circumstances demand.

Analysis

The same approach as is suggested for unstructured focus groups is recommended. However, because of the more structured approach it is usually easier to group the findings around the primary questions. In addition, analysts should be aware that although quantitative data may emerge from the discussions, one should always qualify this with reference to the sample group, on what basis it was selected, and include any minority information. For example, if a group of twelve people all have the same view it could be erroneous to extrapolate that view to the whole cohort that had undergone the training.

Review findings/views to identify suggestions for improvement. Group similar points from interviews together, by looking for common themes and phrases around the primary questions.
Assemble and report the results of this grouping exercise before making a separate analysis of the information. It is often a good idea to tabulate these results or use another graphical method such as a distributive chart to help you see the overall picture.
Write up a report highlighting key points and issues raised. Always include 'minority' information with this so that the whole picture can be seen.
Present your analysis with the appropriate qualifying statements.

Structured focus groups

General

You may use a structured focus group approach where you need information about specific issues and areas of detail. For example, you may wish to follow up on a specific set of activities within a training programme, or to check that an objective or objectives relating to a specific topic such as sentencing provisions have been met. You are unlikely to get the richness of information that comes from a semi-structured approach but in some circumstances this is appropriate.

Approach

As with all types of focus groups, proper preparation of the facilitator's brief is important. In a structured approach the brief is constructed using specific targeted questions aimed at eliciting the closely defined information that is required. The brief may be a mixture of question types including closed questions, coded responses, classification questions etc. It is sometimes possible to ask the same question more than once, using different types to validate the answers. This is often done by professional researchers, but may be beyond the scope of inexperienced facilitators. The facilitator will need to exert firm control of the group to keep it on track.

Hints and tips

You may formulate your brief in the same way that you would a one to one structured interview brief.
The question types may be varied to allow the facilitator to explore issues in depth. For example, you may wish to ask a direct question with a yes/no answer, but follow that up with a question that asks the group to rate a response by using a discrimination scale.
Facilitators must be aware that it will not always be possible to arrive at a consensus view and prepare for that eventuality by recording the full spectrum of views across the group.
It may be necessary for the facilitator to exercise tight control over the group in order to keep the discussion on track.
Set a time limit on the interview so that participants are able to judge how long they will be involved.

Analysis

The information that you gather will often be a mixture of qualitative and quantitative data. As with unstructured and semi-structured focus groups the suggested approach would be as follows:

Review findings/views to identify suggestions for improvement. Group similar points from focus groups together, by looking for common themes and phrases.
Assemble and report the results of this grouping exercise before making a separate analysis of the information. It is often a good idea to tabulate these results or use another graphical method such as a distribution chart to help you see the overall picture.
Write up a report highlighting key points and issues raised. Always include 'minority' information with this so that the whole picture can be seen.
Present your analysis with due reference to the sample size, the selection of the group etc.

What you need to do before running a focus group

Be sure that a focus group approach is the right one for the level of evaluation you are looking at.
Decide what you want to find out and select the approach to best collect that data.
Identify an experienced facilitator to run the group(s).
Decide how you will select participants for the group(s). This may involve consideration of a suitable sample size, how that sample group is made up e.g. do you want experienced magistrates, or new magistrates? It is a good idea to select a group that mimics the composition of the complete training cohort.
Be aware that an ideal group size is between 6 and 12 people. You may wish to run more than one group to cross validate your findings.
Make yourself familiar with the focus group protocol
Identify a competent note taker to record the proceedings.

Focus group protocol

The focus group approach requires several things to be successful, these are:

A competent facilitator to run the group.
A recorder who takes part in the discussion but who accurately and fully records comments.
The participants' group itself which should be no more than 10 people - if the pilot group is large then take a sample group to limit numbers.
Agreed rules for the exercise.

Rules for focus groups

When conducting an evaluation in the form of a focus group, it is important that participants are aware of some ground rules so that the evaluation can be carried out effectively and that all group members have the opportunity to contribute. It may be helpful for the facilitator to discuss these rules before the focus group starts the evaluation process.

The facilitator:

Is there to facilitate the process, they will not question the group's or individual's comments, unless they breach the rules, or unless they need clarification.
Will ensure that everyone, who wants to, has chance to air their views.
Will ensure that no one person or group dominates the process to the detriment of others.
Will ensure that participants are aware that the process is confidential and risk free.
Will ensure that the group rules are adhered to.

Participants

Will not disparage other's views - all views are to be recorded as all have validity.
Will exhibit the normal respect and courtesy for colleagues taking part.
Will not indulge in personal criticism, but confine their comments to the design and delivery of the training under discussion.
May request an anonomised copy of the notes of the meeting for their own use.

Using focus groups for validation

While it is possible to use focus groups for validation, it is more usual to use questionnaires. A focus group approach to validation may be applicable when you have a small cohort of trainees and, or, the event is going to be run over an extended period with long intervals in between. In these scenarios the use of a structured or semi-structured approach would be the suggested approach to use. However, you would need to be convinced that the extra time and effort involved would give you sufficient added value to use these approaches rather than questionnaires. An example of a section of a semi-structured focus group brief for validation appears below:

Primary Question (general points)	Supplementary Questions (detailed points)
Was the design of the training effective and easy to follow?	Do you feel that there was a clear introduction to the training?
	Do you feel that the training objectives were clear?
	Do you feel that the structure of the training was logical and easy to follow?
	Do you feel that the content of the training was relevant?
	Do you feel that the training met equal treatment and diversity guidelines?
	Do you feel that clear language was used in the training materials?
	Do you feel that there was appropriate interaction in the training?
Was the delivery of the training effective and easy to follow?	Do you feel that the trainer/facilitator prepared for the training?
	Do you feel that the trainer/facilitator established a rapport with all delegates?
	Do you feel that the trainer/facilitator promoted anti-discriminatory practice?
	Do you feel that the trainer/facilitator presented and explained information, exercises and demonstrations effectively?
	Do you feel that the trainer/facilitator facilitated group learning?

Using focus groups to evaluate whether objectives have been met

As with validation it is more usual to use questionnaires or interviews to evaluate whether objectives have been met. However, it is possible in certain circumstances to use focus groups effectively to evaluate at this level. One example of this may be when a specific high risk training intervention has been run for the first time and trainers want reassurance that learning objectives are being met. This may occur when the use of a pilot is not possible or when using an expert group of participants to evaluate the training. As with all evaluation using the pragmatic approach, some form of risk analysis should be carried out in order to make sure that a focus group is appropriate. It may be possible to use any of the three approaches in these circumstances, however, the example of a section of semi-structured brief is given below.

Primary Question (general points)	Supplementary Questions (detailed points)
Was the design of the training objectives effective and easy to understand?	Do you feel that the training objectives were clear?
	Do you feel that the structure of the training objectives was easy to follow?
	Do you feel that the content of the training was relevant to the training objectives?
	Do you feel that the training objectives met equal treatment and diversity guidelines?
	Do you feel that the exercises enabled you to meet the training objectives?
Was the delivery of the training objectives effective and easy to understand?	Do you feel that the trainer/facilitator explained the training objectives clearly?
	Do you feel that the trainer/facilitator presented and explained the information, exercises and demonstrations effectively?
	Do you feel that the trainer/facilitator made clear the link between the training objectives and the exercises?
Was the training environment suitable for meeting the objectives?	Do you feel that the training room helped or hindered your achievement of the training objectives?
	Do you feel that the equipment helped or hindered your achievement of the training objectives?

Using focus groups to evaluate what has been learned by participants

Focus groups may be used in this context, particularly when wishing to determine exactly what people have learned from the training. This is not just about if objectives have been met but also what other learning has occurred. For example, have there been any other things that have arisen as a result of meeting learning objectives, such as, a different attitude that may have been encouraged, or greater confidence about a certain area of work? In this context an unstructured or semi-structured approach is often deployed. An example of a semi structured brief appears below.

Primary Questions (general points)	Supplementary Questions (detailed points)
Did you learn any new knowledge/skill(s) as a result of the training?	What new knowledge have you learned?
	What new skill(s) have you learned?
	What do you now do differently as a result of the training?
Do you feel competent in applying the new knowledge/skill(s)	Did the training fully equip you with the new knowledge/skill(s) required?
	Are there any gaps in the training?
	What improvements would you make to the training to make it more effective?
	Are there any barriers to you applying fully the new knowledge/skills(s)
Have you applied the new knowledge/skill(s)	Have you been able to apply fully your learning from the training?
	How often have you applied the new knowledge/skill(s)

Using focus groups to evaluate the impact of training

This is possibly the most popular use of focus groups. When evaluating the impact of training it can be effective to use any of the three approaches, depending on what information you want to collect. An example of a semi-structured brief appears below.

Primary Questions (general points)	Supplementary Questions (detailed points)
Do you consider yourself now competent and confident in using the sentencing guidelines effectively in court?	Did the training prepare you for actually carrying out sentencing under the new framework?
	Were you aware of the likely impact of the new sentencing framework on how you would carry out your sentencing functions before your first sitting?
	Was the guidance in the training effective? Did it cover the scenarios that you have come across in court?
	Have there been any scenarios that you have come across that are not covered in the guidance or the training?
	If so, how were you able to deal with them?
Do you think that the supporting materials supplied to you (forms, handouts etc) were of use now that you have had chance to apply the training?	Specifically, do you think that the sentencing reasons form is of use to you?
	If not, what changes do you think need to be made in light of your experience?
	Do you think that the process itself transfers well to the court environment?
	If not, what changes would you like to see?
Now that you have had chance to apply the training in the courtroom, what is your opinion of the success of the training programme?	Do you feel that the training needs any additional material or exercises?
	If so, what would you recommend?