Photo by Agence Olloweb on Unsplash
People who are closest to front line program operations often get a “feel” for whether the program is working as intended. When this sense is not supported by documented information, it can be perceived by oneself and by others as being informed by intuition rather than by evidence. Usually a sense that something is working is grounded in some information. In How to Measure Anything: Finding the Value of Intangibles in Business, Douglas Hubbard says “Measurements are almost always performed to test the truth of some idea, and those ideas don’t just come from a vacuum.” This post presents some ideas about how to measure to test the truth of seemingly “intuitive” program knowledge. It takes some common reasons for thinking a program is effective and talks about ways to extend existing information to understand the program better, address potential bias, and substantiate program effectiveness in a more robust way—in other words, ways to move from intuition to evidence.
I think the program is a success because there is demand for the program. People show up.
Great! People value the program. Can you get information on why they value it, what they value about it, how it helps them?
- Ask (surveys; interviews; focus groups): How does the program help them achieve their goals?
- Assess: Do their goals align with the program goals?
- Ask (surveys; interviews; focus groups): what impact does the program have on them?
- Assess: Can any of that impact be quantified?
I think the program is a success because of the stories/anecdotal evidence I get—people are telling me it’s made a difference.
Great! You know it makes a difference for some people. Can you get information to tell you WHAT SHARE of people using the program are affected by it?
- Ask (surveys; interviews): Participant questions on outcomes
- Decide: how often would it need to achieve certain outcomes to be to be considered successful?
- Decide: Is the program successful if it has an effect for some groups and not all?
- Assess: Might differences be related to factors outside program scope?
- Assess: is affecting only some groups consistent with the program mission
I think the program is a success because I can see that participants are engaged.
Great! You have observed indicators of participant engagement, such as exhibiting interaction with the activities, or showing signs of enjoyment. Can you document those observations in a systematic way? Can you identify program components with higher and lower levels of engagement? Can you find out how participants are affected?
- Ask (in-program rapid feedback; end of program survey): participant questions on interest in or satisfaction with each program component.
- Ask (in-program discussions; end of program survey): participant questions on how they have been affected by the program (knowledge, attitude, skills, awareness)
I think it’s a success because we are achieving our output goals.
Great! You are operating as planned. Can you get evidence to show that your outputs are leading to desired outcomes? Is there some evidence you can get that would convince you and others more?
- Ask (in-program discussions; end of program survey): participant questions to determine whether desired outcomes are being met, e.g. how they have been affected by the program (knowledge, attitude, skills, awareness)
- Research: Look for existing studies showing that these outputs lead to desired outcomes, and/or that outcomes lead to target goals and impacts.
I think the program is making a difference, but I can’t show a long-term impact.
- Assess: Can the “big” or long-term outcome be broken down into smaller components, and can the program be shown to have an impact on some of those? The long-term “grand challenge” outcome may be influenced by too many other factors to ever be shown to be affected by a single program. For example, reducing the number of people experiencing poverty is an ambitious goal influenced by many non-program factors. What are “smaller” or more narrowly defined goals that also have an impact on peoples’ lives?
- Review: revisit your theory of change or logic model to make sure you have short and medium term outcomes identified; specify additional ones if needed.
- Change your measured outcome to something more specific and medium-term. For example, instead of trying to show that poverty is reduced, might the program measure (1) increased diversity of income sources; (2) changes in spending or savings habits; (3) increase in funds available for emergencies; (4) increase in income opportunities/choices in local area.
