Magistrate Monitoring and Evaluation

Evaluation Guidance: Evaluation Tools

Questionnaires Interviews Focus Groups
Observations self assesment Self-assessment Verbal Feedback
self reflection Self-reflection

Questionnaires

May be used to evaluate:

Validation | Objectives |Learning | Impact |
General advice | Different question types (general) | Question types (specific).

Questionnaires for validation

It is common for questionnaires to be used for validating a training event. You may wish to design your questionnaire to cover a range of validation data such as:

Satisfaction with the learning experience.
Immediate feelings on meeting learning needs.
Competency of facilitator(s).
Achievement of learning objectives.
Effectiveness of learning methods used.
Facilities, e.g. venue, catering, accommodation.
Quality/usefulness of learning materials, e.g. handouts, PowerPoint.
Pre - event briefings.
Administration - before and during the event.

In order to do this successfully you should use a mixture of question types.

Questionnaires to test whether objectives have been met

Again it is quite usual to use questionaires to check that objectives have been met.

You should ensure that the question type is appropriate or is part of a mix of appropriate types.
The objectives to be used should be clear, that is, contain the three elements of performance, criteria and condition.
You should use a balanced and non biased scale when using semantic/differential and Likert type questions.
You should consider when the most appropriate time is to issue the questionnaire, to ensure objectives have had time to be met.

Questionnaires to measure learning

When using questionnaires to measure whether learning has been achieved there are a number of things to consider:

Don't be tempted to make the questionnaire an exam paper. You can use a learning check approach within the event itself to do this more effectively.
You will need to allow time for the learning to take place before issuing a questionnaire this may be some time after the learning event.
You can use questionnaires to assess respondent's perceptions of what they have learned by asking suitably open questions, and follow up questions such as "How would you improve the learning?", or "What reasons do you think hindered you from learning effectively?"

Questionnaires for measuring impact of the training

Although it is not usual to use questionnaires for measure impact, it can be an appropriate method in some circumstances, for example when you wish to get information from a very diverse large group and a sampling method such as interviewing or focus groups is not feasible, or where the geographical location of the potential respondents makes interviews or focus groups unrealistic. You will need to consider:

The best time to issue the questionnaire to allow the impact to take place.
If you are using a questionnaire to measure perceptions, how you might wish to validate those perceptions.

General advice

Questionnaires are one of the most commonly used evaluation tools for evaluating Magistrate training. Compiling a good questionnaire will provide an effective tool for gathering informative and useful information. The main decision to be made is the question type. It is important to keep focused on the purpose of the evaluation, so that you can use the appropriate format and question type to draw out this information.

Different question types

Questionnaires can be constructed using a number of different question types. The mix of types is dependent on what you want to know and how you want that information expressed e.g. a numerical format using questions that record reponses in a quantitative way such as Likert type scales (see guidance below), or in a narrative format such as an open format. It is perfectly feasible to mix and match different types in a questionnaire. However, you should be aware that information from questionnaires is, by its nature, more limited than information from other methods of evaluation because of the distance between the questioner and the respondent. There are a number of different question types including:

Open | Coded/Multi-choice | Classification | Semantic Differential | Likert type scale

Open questions

Approach

The use of open questions is good where you need to collect views in a flexible way, for example where a yes/no answer doesn't give enough detail, or where you are unsure of the scope of the information that you want to collect.
Open questions are easy to design because of their general nature, for example:
- What issues arose as a result of implementing this training?
- How do you plan to implement these aspects of the training?

Open questions often throw up ideas and responses that you may not have expected and you must be able to deal with these.

Good for

Testing knowledge usually of a more complex nature. The nature of open questions allows flexibility of response and encourages the respondent to give a fuller response than a straight yes/no answer
Measuring attitudes
Exploring views

Hints and tips

Can lead to a diversity of responses - be prepared for this.
Although easy to design and can provide non-bias responses, this type of question should be used with caution as it often elicits unfocused information.
Important not to give any hints to the answers in order to give an untainted response.

Analysis

Need to consider, assess and validate each individuals response to obtain relevant outcomes - this takes time.
Requires a well defined analysis framework in order to break down and group the responses.

Examples

What were the most useful areas of learning that you will take away from this event?
What were the least useful areas of learning and why was this?
What other comments would you like to make?

Coded/multi choice questions

Approach

Design is important because of the need to cover as many options as possible.
Information gathered is often superficial because of the limited options but can be useful when used with other question types, or where options are of a limited nature due to the content of the question

Good for

Establishing facts.
Testing knowledge.
Measuring attitudes - the respondent is given a choice of specific answers.
Analysing information easily, especially where large groups of respondents are being questioned.

Hints and tips

Try and use one type consistently, e.g. tick box, underline statement etc.
Design questions to give maximum choice.
Consider using narrative options to supplement replies, such as "other, please describe".

Analysis

Usually straightforward and numerical, except where there are a number of narrative options.
Easy to analyse as allows for a structured range of responses.

Example MS Word [24KB]

Classification questions

Approach

Can be used with other question types in a questionnaire when it is important to reflect the views of segments of the target population.

Good for

Classifying respondents into a particular category or group, e.g. males, females or ethnic origin.
It is used to check how representative the sample is of the whole population against specified criteria.
Analysing data is relatively easy because of the nature of the categorisation.

Hints and tips

Often best to ask these questions at the end of the questionniare as asking them at the beginning may put people off.
If the questionnaire is being completed anonymously, it is important that the combination of classification questions do not identify an individual.
Explain why you want this information.
Ensure that a full range of options is included.

Analysis

Straightforward - Usually numerical based or relating other information to one or more classifications.

Example

Click here to see an example in Microsoft Word Word Icon

[24KB]

Semantic differential questions

Approach

Questions should be constructive to illustrate the extreme ends of a rating scale, e.g. none...extensive, frequent... not at all etc.
Discrimination scales should be consistent in length.
Sometimes used in combination with narrative comments.

Good for

Assessing skills.
Measuring attitudes and opinions. Respondents give subjective judgements on a rated scale.

Hints and tips

Response ranges are usually 4 to 10 scale points in length.
Carefully consider the range as the larger the scale the more difficult it is to interpret.
When considering the range, even numbers (i.e. 1-4; 1-6 etc.) avoid respondents' tendency to go for the middle option.

Analysis

Easy to analyse as allows for a structured range of responses and enables segmentation of data to be carried out easily.
Basic numerical analysis will give outcome data relatively easily.

Examples

Click here [27KB]to see an example in Microsoft Word
Click here [32KB]to see an example of scoring methods in Microsoft Word

Likert type scale questions

Approach

Questions should be constructed to allow the respondent a balanced set of options, that is the same number of positive and negative statements.
It is optional whether to have a neutral option for the same reasons as stated in the guidance for the semantic differential type question.

Good for

Assessing skills, measuring attitudes and opinions.
Indicating respondent's views against a specified rating.

Hints and tips

To avoid a perception of bias or constraint it is important to have a balanced set of response options, i.e. of favourable/unfavourable.
Need to consider whether to have a middle option (for the same reasons as listed for semantic differential type questions).
Always keep the positive and negative ends of the scales on the same side of the questionnaire.

Analysis

Easy to analyse as allows for a structured range of responses.
Numerical analysis will give outcomes.

Example

Click here Word Icon [26KB]to see an example in Microsoft Word

Using questionnaires for validation

It is common for questionnaires to be used for validating a training event. You may wish to design your questionnaire to cover a range of validation data such as:

Satisfaction with the learning experience.
Immediate feelings on meeting learning needs.
Competency of facilitator(s).
Achievement of learning objectives.
Effectiveness of learning methods used.
Facilities, e.g. venue, catering, accommodation.
Quality/usefulness of learning materials, e.g. handouts, PowerPoint.
Pre - event briefings.
Administration - before and during the event.

In order to do this successfully you should use a mixture of question types.

Using questionnaires to test whether objectives have been met

Again it is quite usual to use questionaires to check that objectives have been met.

You should ensure that the question type is appropriate or is part of a mix of appropriate types.
The objectives to be used should be clear, that is, contain the three elements of performance, criteria and condition.
You should use a balanced and non biased scale when using semantic/differential and Likert type questions.
You should consider when the most appropriate time is to issue the questionnaire, to ensure objectives have had time to be met.

Using questionnaires to measure learning

When using questionnaires to measure whether learning has been achieved there are a number of things to consider:

Don't be tempted to make the questionnaire an exam paper. You can use a learning check approach within the event itself to do this more effectively.
You will need to allow time for the learning to take place before issuing a questionnaire this may be some time after the learning event.
You can use questionnaires to assess respondent's perceptions of what they have learned by asking suitably open questions, and follow up questions such as "How would you improve the learning?", or "What reasons do you think hindered you from learning effectively?"

Using questionnaires for measuring impact impact of the training

The best time to issue the questionnaire to allow the impact to take place.
If you are using a questionnaire to measure perceptions, how you might wish to validate those perceptions.