We suggest that you copy and save the URL from your browser’s toolbar. This will allow you return to this page of personalised guidance and share the information with colleagues in future.
Before you start
Before starting work:
- Make sure you have a defined purpose for your evaluation.
- Ensure your key stakeholders understand what is to be produced (e.g. a business case or a report to a funder).
- Draft a logic model for your evaluation. This is like a road map for your initiative that shows its different parts and how they combine to achieve your goal.
- For help, check out the logic model template from Evaluation Support Scotland and the guidance for filling it in.
- View stage one of our step-by-step guide for more tips on how to get started.
Once you have completed these steps, you’re ready to book your evaluation surgery:
- These are one-to-one advice sessions with a specialist in evaluation processes sourced from a NW London organisation.
- Fill in the request form on the evaluation surgeries page.
Defining your initiative
Before you choose how you are going to design your evaluation, you first need to understand the core aspects of your initiative. To do this, use the PICO framework to help you answer the following questions:
- What is your Population? (The group of people who receive your initiative)
- What is your Intervention/initiative and what parts need evaluating (especially with a complex service)?
- What is your control or Comparator group? (If there is no group of people not receiving your initiative, can you at least measure changes in the same group of people before and after they receive your initiative?)
- What are your key Outcomes? (See next section)
- For more info on the PICO framework itself, check out the PICO explainer here.
Be sure to gather all your answers to these questions before your evaluation surgery.
Choosing the right design
We’ve recommended an evaluation surgery based on your answers in the evaluation design tool questionnaire. This is either because you submitted many ‘Not sure’ answers – suggesting you need advice – or because some of your answers indicated that your initiative is potentially complex and you will benefit from the surgery.
At the surgery, our expert will walk you through one of the following evaluation designs for your initiative:
Retrospective study
- This is used to carry out an evaluation after delivering your initiative and where you cannot easily collect data at multiple time points or from different groups:
- You may be able to complete an evaluation retrospectively by conducting a clinical audit of the data you already have.
- However, bear in mind that collecting data from a single time point with no comparison available is a very weak design (see below) and stops you from making definitive statements about whether your initiative is actually working.
- Ask yourself if you could collect data from different time points eg by using routinely collected data and looking at a different year of data to act as a comparison.
Before and after design
- This involves collecting data for your key metrics at the start and end of your initiative (or its study period) and comparing how results have changed:
- For example, how has a person’s wellbeing changed after they used your initiative compared to their wellbeing at the start?
- However, this is considered a weak design because there is no control group to compare the people who didn’t receive your initiative to those who did.
- Head here for more details.
Before and after design (with control group)
- This involves comparing how things change before and after your initiative for your group of patients/staff/users compared to a group that did not receive your initiative.
- Always ensure the two groups are comparable ie similar demographically.
- Head here for more details.
Statistical process control
- This is a method commonly used in the NHS to work out how the metrics for your initiative are changing over time compared to the average metrics over a particular period.
- This design is specifically used for quantitative data (see below) and needs multiple sets of data to work.
- Head here for more details.
Difference in differences
- This is similar to the before and after (with control group) design but is more sophisticated:
- It involves calculating the difference in outcomes before and after the initiative for each group individually before you move on to calculate the variations between the differences of the two groups.
- This design takes into account any external factors that may influence outcomes, providing a clearer picture of your initiative’s impact.
- Head here for more details.
Process evaluation
- This is often used in healthcare as we are not evaluating efficacy – knowing whether an initiative works – but effectiveness i.e. focusing on why and where an initiative works in real life settings.
- Process evaluation is ideal for identifying and understanding factors such as implementation and context.
- Head here for more details.
View stage two in the step-by-step guide for more resources on selecting an evaluation design.
Data explained: Qualitative data
Qualitative data is non-numerical data that is observed and described. It can be sourced by conducting interviews at the start and end of your initiative.
This helps you ascertain how people felt about your initiative as they progressed through it. Examples of qualitative methods you can use are:
Interviews
These are interviews with select staff delivering your service or with the users who received it:
- If you are interested in sourcing comparable views from your interviewees, use structured or semi-structured interviews (i.e. keep questions consistent).
- To uncover more complex insights – and all possible challenges – conduct in-depth interviews instead to elicit richer feedback.
- Head here for more details on the different types of interview available and watch this video for more information on designing and conducting semi-structured interviews. This YouTube channel also has information on other types of interviews too as well as tips on how to analyse qualitative data.
Focus groups
These are discussions with a group of participants (typically 6-10 people) and are useful for:
- Gathering diverse views and perspectives.
- Understanding group dynamics and how opinions can converge or differ.
- Exploring complex issues in depth and identifying common themes.
- Focus groups are particularly useful for evaluating initiatives where group interaction is key. Think community-based programmes or team-based healthcare services.
- Head here for more details.
Process or pathway maps
These involve creating visual representations of the steps and processes involved in an initiative. This method is helpful for:
- Identifying key stages and touch points of your initiative as well as potential bottlenecks for delivering it.
- Understanding the flow of activities from the perspective of different stakeholders.
- Highlighting areas for improvement and ensuring that all aspects of the initiative are considered.
- Pathway maps are especially useful for complex initiatives where multiple steps and stakeholders are involved, helping to provide a clear overview of the entire process.
- Head here for more details.
Ethnographic methods
This involves observing and interacting with participants in their real-life environment:
- This approach helps you to understand the context of your initiative and uncover factors that may not emerge through interviews or surveys alone.
- Head here for more details.
Quantitative data
Quantitative data is numerical data that is measured, counted or compared. It allows you to monitor how key numerical metrics have changed because of your initiative.
Discuss how to approach to quantitative data during your evaluation surgery. The specialist may advise you to use routinely collected data or to collect custom data, for instance, by designing your own survey. Some quantitative methods to consider are:
Conducting a power calculation
If you want to know what sample size you need (i.e. how many people you need to reach with your initiative to show a significant effect), consider a power calculation.
- A power calculation helps you to detect the ‘effect’ of your initiative. The higher the power, the greater the chance you can detect what is ‘true’ and lower the chance of bias. This is highly influenced by sample size. For example, if your initiative reaches and measures impact on 1,000 people, you are more likely to be able to tell if it has had a real effect on their outcomes compared to if the same initiative only reaches and measures the impact on 10 people.
- Use these online calculation tools to get started.
Analysing routinely collected data
This is quantitative data that may be available in your initiative ie electronic health records data or submissions to national datasets. For example:
- You can look at how data for the metric you’re interested in (such as specific questions in the NHS staff survey) has changed compared to before your initiative was introduced.
- You will also need to consider which design to use ie Statistical Process Control or before and after (see above).
- Head here for more details.
Designing and running a custom survey
This allows you to collect data on key outcomes and conduct power calculations. These help determine the appropriate sample size needed to detect meaningful differences or changes:
- Surveys should ideally include both quantitative questions (for example, rating scales) and qualitative open-text questions (such as suggestions for improvement).
- To gather comprehensive feedback and create detailed summary metrics, design a survey to be conducted at multiple time points. This approach allows you to track changes over time.
- When designing your survey, also consider how it will be distributed (either paper or online) and the time required for data analysis.
- Online tools like Google Forms or Microsoft Forms offer built-in data visualisation tools that can save time and resources.
- Head here for more details.
Metrics to use
What metrics you should collect will depend on the key questions you are trying to answer and what the output of your evaluation is:
- When using the evaluation design tool, select your initiative type and size to source the right list of metrics.
- Initiative size relates to the budget for your initiative ie the total spend listed in your business case.
- Initiative type relates to the type of initiative you are evaluating. Select the type that best matches your initiative (if yours has no match to any available types, select ‘Other’).
Make sure to capture the following minimum set of metrics:
- How many people (staff or patients/members of the local population) you are reaching with your initiative.
- Satisfaction with the initiative.
- Both total costs and costs per user.
- Basic set of demographic metrics (user age/gender/ethnicity) to understand inequalities in your initiative delivery.
For more information, view our guidance on determining what you want to measure.
Key takeaways
Here’s a quick overview of the different tasks that you’ll need to carry out to ensure evaluation success:
Planning and design
- Define the purpose of your evaluation and ensure all stakeholders are aligned with your goals.
- Use a logic model to visualise all the elements that make up your initiative and how they contribute to your goals.
- Answer the questions in the PICO framework (Population, Intervention, Control, Outcomes) to guide your evaluation design.
- Once this info has been gathered, request an evaluation surgery appointment for personalised advice on your potential evaluation design and more.
Evaluation designs
- Select a suitable design (e.g. before and after, Statistical Process Control or process evaluation).
- Consider using both quantitative and qualitative data to create a more comprehensive understanding of your initiative’s effectiveness.
Data collection
- For qualitative data, use interviews to gather insights and feedback.
- Also consider a power calculation to calculate an appropriate sample size.
- For quantitative data, consider surveys for larger-scale data collection.
- Use existing data sources like electronic health records.
Metrics
- Choose metrics based on your evaluation questions and project size.
- Include metrics for reach, satisfaction, costs and demographics.