Evaluation Design

Evaluation research is widely used within the fields of Psychology, Education and Criminology in order to consider the ‘effectiveness’ of a treatment, programme or intervention, to assess whether or not there is an impact on the participant and to measure what that impact might be. In the case of the From Boys to Men project a six week educational intervention programme called Relationships without Fear (RwF) was the focus of the evaluation. In order to carry out this type of evaluation the study needs to be constructed as an experiment enabling the investigator to manipulate one or more variables whilst ideally controlling for or ruling out other variables that might impart an effect upon the participant.

Pre-test Post-test Design

There are many different experimental design models that fall under the heading of evaluation research, but perhaps most commonly an objectivist approach is adopted whereby the researcher seeks measureable verification as evidence for effectiveness or impact. Models of this type often seek to find causal relationships between variables, to identify a ’cause’ and an ‘effect’. Perhaps the simplest experimental design which might be used to investigate the effect of a single variable (treatment or intervention) by using a before and after comparison, is the pre-test post-test design. The From Boys to Men project implemented the Attitudes to Domestic Violence (ADV) questionnaire before and after pupils received the RwF intervention programme and at three-month follow-up. The research team were able to measure a change in attitude following the intervention and were also able to show that the intervention had a lasting impact three months later. The questionnaires used in the From Boys to Men project are available in Research Materials.

Static Group Comparison

The pre-test post-test model can measure changes in participants’ attitude or behaviour following implementation of a treatment, programme or intervention, but does not enable the researcher to rule out the influence of other contributing factors (confounding variables) that might have influenced that change. If the treatment or intervention is applied to a single group of participants, it is impossible to say with any certainty whether an effect can be solely attributed to the intervention. For example, the recipients of the RwF intervention might change their attitude towards domestic violence due to some influence other than the intervention, such as a high profile domestic violence incident reported in the national media perhaps. The experimental design can be strengthened by the inclusion of a comparison group that will be tested pretest and post-test but who will not receive the treatment or intervention. However, in this simple static group comparison design it still cannot be assumed that any measured impact can be attributed solely to the treatment or intervention as there may be other differences between the groups that might be affecting the results.

Control Groups

The quasi experimental designs discussed do not always reveal satisfactory evidence as no attempt has been made to control other variables that might have an effect on the results. It is not possible to conclude that a measured impact is directly due to the treatment or whether it might be due to some other difference between the groups. The comparison groups therefore need to be as similar as possible. For example, in a school-based model, as represented in Phase 1 of the From Boys to Men project, it would be ideal for each group of pupils to have a similar proportion of boys and girls to eliminate any gender bias and to represent similar age ranges, ethnicity and social economic status, so that any impact or effect from the intervention or treatment cannot be attributed to any other variable and the researcher can conclude with more confidence that any effect must be due to whether or not the pupils participated in the intervention programme.

In the From Boys to Men project 1,203 Year 9 pupils (aged 13-14) were recruited from 13 schools in Staffordshire and were divided into two groups:

  • ‘intervention schools’ – 619 pupils from 27 classes in seven schools
  • ‘control schools’ – 584 pupils from 27 classes in six schools

572 participants were male and 596 were female (gender was not recorded for 35 participants). The majority of young people who took part described themselves as White (89.5%) and as British (96.8%). Participation was voluntary.

As long as the two participant groups are comparable, it is possible to see what would happen if the group did not receive the treatment or intervention and to report whether or not it had a differential effect on the groups.

Randomised Control Trial (RCT)

In order for researchers to claim that a change measured in a group of participants is due solely to a treatment or intervention, the control group must be similar to the treatment group on all criteria apart from the treatment itself. The most efficient method of trying to achieve this is to randomly allocate subjects to the control and treatment groups. Randomisation ensures that the two groups are comparable on all factors, so that the difference between the groups can be considered as a valid measure of the treatment effect. The assumption is therefore that the only difference between the groups is that one has received the treatment.  If this is the case, the researcher can claim a causal connection between the treatment and the outcome. Whilst this design model is an improvement over the static group comparison, it is still not perfect, as it is still not possible to establish with absolute certainty that the two groups were comparable. In order to achieve greater comparability, the groups of participants must be much larger to increase the likelihood that their external variables will be similar or representative of one another.

Quasi-Experimental Design

In research studies, it is not always possible or even desirable to obtain a truly random sample of participants. In order for a sample to be representative of the population it needs to be large enough to reflect the parent population, therefore studies of this magnitude are relatively uncommon. In order to overcome the need for a large numbers of participants researchers sometimes choose to match their samples. The non-random allocation of subjects to groups is often referred to as quasi-experimental design. In this method, the researcher may decide to arrange that the groups need to be similar according to certain specified variables such as participation in anger management treatment, rate of school attendance, socio-economic status i.e. dictating the choice of participants to some extent. However, members of each group could still be randomly chosen from the school to reflect other variables evenly such as gender, age and ethnicity.