We suggest that you copy and save the URL from your browser’s toolbar. This will allow you return to this page of personalised guidance and share the information with colleagues in future.
Before you start
Before starting work:
- Make sure you have a defined purpose for your evaluation.
- Ensure your key stakeholders understand what is to be produced (eg a business case or a report to a funder).
- Draft a logic model for your evaluation. This is like a road map for your project that shows your initiative’s different parts and how they combine to achieve your goal.
- For help, check out the logic model template from Evaluation Support Scotland and the guidance for filling it in.
- View stage one of the step-by-step guide for more tips on how to get started.
Defining your initiative
Before you choose how you are going to design your evaluation, you first need to understand the core aspects of your initiative. To do this, use the PICO framework to help you answer the following questions:
- What is your Population? (The group of people who receive your service)
- What is your Intervention/initiativeand what parts need evaluating (especially with a complex service)?
- What is your control or Comparator group? (If there is no group of people not receiving your initiative, can you at least measure changes in the same group of people before and after they receive your initiative?)
- What are your key Outcomes? (See next section)
For more info on the PICO framework itself, check out the PICO explainer here.
Choosing the right design
Producing evidence that shows your initiative is effective is crucial to your evaluation’s success. To help you achieve this, you need to select an evaluation design model. This is a practical plan for working out the real-world effectiveness of your initiative. For example:
Before and after design (with control group)
- This involves comparing how things change before and after your initiative for your group of patients/staff/users compared to a group that did not receive it.
- Always ensure the two groups are comparable, i.e. similar demographically.
- Head here for more details.
Statistical Process Control
- This is a method commonly used in the NHS to work out how the metrics for your initiative are changing over time compared to the average metrics over a particular period.
- This design is specifically used for quantitative data (see below) and needs multiple sets of data to work.
- Head here for more details.
Process evaluation
- This is often used in healthcare as we are not evaluating efficacy – knowing whether an initiative works – but effectiveness ie focusing on why and where an initiative works in real life settings.
- Process evaluation is ideal for identifying and understanding factors such as implementation and context.
- Head here for more details.
View stage two of the step-by-step guide for more resources on selecting an evaluation design.
Data explained: Qualitative
Qualitative data is non-numerical data that is observed and described. It can be sourced by conducting interviews at the start and end of your initiative.
This helps you ascertain how people felt about your initiative as they progressed through it. Examples of qualitative methods you can use are:
Interviews
You can hold interviews with select staff delivering your initiative or with the users who received it:
- If you are interested in sourcing comparable views from your interviewees, use structured or semi-structured interviews (i.e. keep questions consistent).
- To uncover more complex insights – and all possible challenges – conduct in-depth interviews instead to elicit richer feedback.
Head here for more details on the different types of interview available and watch this video for more information on designing and conducting semi-structured interviews. This YouTube channel also has information on other types of interviews, as well as tips on how to analyse qualitative data.
Focus groups
These are discussions with a group of participants (typically 6-10 people) and are useful for:
- Gathering diverse views and perspectives.
- Understanding group dynamics and how opinions can converge or differ.
- Exploring complex issues in depth and identifying common themes.
- Focus groups are particularly useful for evaluating initiatives where group interaction is key. Think community-based programmes or team-based healthcare services.
Head here for more details.
Process or pathway maps
These involve creating visual representations of the steps and processes involved in an initiative. This method is helpful for:
- Identifying the key stages and touch points of your initiative as well as potential bottlenecks to delivering it.
- Understanding the flow of activities from the perspective of different stakeholders.
- Highlighting areas for improvement and ensuring that all aspects of the initiative are considered.
- Pathway maps are especially useful for complex initiatives where multiple steps and stakeholders are involved, helping to provide a clear overview of the entire process.
Head here for more details.
Patient stories
This process involves participants recording their experiences, thoughts and feelings from their own perspective to understand their experience of care. This approach is great for:
- Collecting detailed insights into the day-to-day impact of an initiative.
- Allowing patients to express their views in their own words and offer a more personal perspective.
Head here for more details.
Data explained: Quantitative data
Quantitative data is numerical data that is measured, counted or compared. It allows you to monitor how key numerical metrics have changed because of your initiative.
Quantitative methods to consider are:
Analysing routinely collected data
This is quantitative data that may be available in your service ie electronic health records data or submissions to national datasets. For example:
- You can look at how data for the metric you’re interested in (such as specific questions in the NHS staff survey) has changed compared to before your initiative was introduced.
- You will also need to consider which design to use ie Statistical Process Control or before and after (see above).
Head here for more details.
Designing and running a custom survey
This allows you to collect data on key outcomes. Surveys should ideally include both quantitative questions (for example, rating scales) and qualitative open-text questions (such as suggestions for improvement):
- To gather comprehensive feedback and create detailed summary metrics, design a survey to be conducted at multiple time points. This approach allows you to track changes over time.
- When designing your survey, also consider how it will be distributed (either paper or online) and the time required for data analysis.
- Online tools like Google Forms or Microsoft Forms offer built-in data summarisation tools that can save time and resources.
Head here for more details.
Metrics to use
What metrics you should collect will depend on the key questions you are trying to answer and what the output of your evaluation is. When using the evaluation design tool, select your initiative type and size to source the right list of metrics:
- Initiative size relates to the budget for your initiative ie the total spend listed in your business case.
- Initiative type relates to the type of initiative you are evaluating. Select the type that best matches your service (if yours has no match to any available types, select ‘Other’).
Make sure to capture the following minimum set of metrics:
- How many people (staff or patients/members of the local population) you are reaching with your initiative.
- Satisfaction with the initiative.
- Both total costs and costs per user.
- Basic set of demographic metrics (user age/gender/ethnicity) to understand inequalities in your initiative delivery.
For more information, view our guidance on determining what outcomes to measure.
Key takeaways
Here’s a quick overview of the different tasks that you’ll need to carry out to ensure evaluation success.
Planning and design
- Define the purpose of your evaluation and ensure all stakeholders are aligned with your goals.
- Use a logic model to visualise all the elements that make up your initiative and how they contribute to your goals.
- Answer the questions in the PICO framework (Population, Intervention, Control, Outcomes) to guide your evaluation design.
Evaluation designs
- Select a suitable design (eg before and after, Statistical Process Control or process evaluation).
- Consider using both quantitative and qualitative data to create a more comprehensive understanding of your initiative’s effectiveness.
Data collection
- For qualitative data, use interviews to gather insights and feedback.
- For quantitative data, consider surveys for larger-scale data collection.
- Use existing data sources like electronic health records.
Metrics
- Choose metrics based on your evaluation questions and project size.
- Include metrics for reach, satisfaction, costs and demographics.