Photo by Emily Whorter on Unsplash

1. Start with high level, fundamental questions.
2. Drill down into more specific measurement and evaluation questions.
3. Identify a purpose for the information through “why/so what” questions.
4. Put it together–link each question to a use in a statement to help prioritize.

an approximate answer to the right question is worth a great deal more than a precise answer to the wrong question–John Tukey, Mathematician/Statistician

The fundamental questions

• What do you want to understand better about your programs?
• What do you want others to understand better about your programs?

The point of evaluating a program or tracking information within or across programs is to gather information to address a defined need.  So it’s essential to explicitly define those needs.  These two fundamental questions provide a good starting point to identify measurement and evaluation needs.

Measurement and evaluation questions

Specific questions aligned with each of the M&E purposes outlined on this website are suggested below.  The questions or information needs you define will point to specific data that need to be collected.

Program improvement

• Is the program/portfolio operating as planned?
• Is the program/portfolio achieving its goals?
• Does effectiveness vary with different groups/settings/program elements?
• Are there outside factors that are contributing to effectiveness?
• Is the program/portfolio successfully engaging with partner communities?

Research/knowledge development

• Does this model/program component/intervention work, and in what circumstances?
• What contributes to achieving target outcomes?
• What can we learn that about program delivery can be applied in other settings?
• What can we and others learn about how public engagement strategies work?

Marketing

• What impact has the program (or portfolio of programs) had?  What value or benefits has it provided?
• How does it contribute to global grand challenges?
• How are we contributing to the public interest?

Reporting and ­­Performance Monitoring

• Is the program or organization meeting standards or goals set by an external organization?
• Are the staff/programs meeting internal institutional standards?

Public access and external coordination

• What existing programs or capacities could help address societal, community, partner organization, or individual needs?
• What programs are available to meet specific community or resident needs?
• Where can someone go to access the resource they need?

Internal coordination

• Are there programs that could work together for greater effect?
• Are there/have there been programs with similar approaches or working with similar groups that can learn from each other?

Strategic planning

• Is the institution as a whole meeting its targets/goals and being effective in public engagement (students/faculty/institution/ community impacts)?

Allocation

• How much is each unit doing towards this function of the larger organization?
• How should central resources in engagement be distributed?

The why/so what questions

It’s easy to make a wish list of “nice to know” information, but given the time and resources needed to collect it, there should be a substantive reason to justify the effort.  Thinking through some “why” questions helps prioritize the information to collect.

• Why do you want to know it?
• How do you expect to use the information? How do you expect others will use it?
• Will the information help you or others make decisions? Which decisions?  What specific information will you need in order to make those decisions with confidence?

Putting them together

Your answers to the questions above can be put together in a series of statements.  Going through this exercise confirms whether you really need the information, and the statements can help communicate the purpose of the evaluation efforts to others.

I/we want to know [information area] in order to (so that)  [action or decision].

• I want to know if my program/portfolio is operating as planned in order to make operational adjustments if needed
• I want to know if this intervention works, and in what circumstances, so that others can develop effective programs or build on the research further.
• I want to know the benefits and value the program provides in order to present the information to legislators in order to assure continued support.
• We want to know if a staff member’s programs are effective in order to make a decision about promotion.
• We want to know if specific policies, resources, or actions have had an impact in order to decide whether to continue them.
• We want others (the public) to know the topics covered by each program so that they can find resources and programs that interest them
• We want others (people operating programs) to know which audiences each program works with so that similar programs can coordinate, share best practices, and cross-market where applicable.
• We want to know all the programs engaging on a topic so that when grant opportunities come in we can coordinate and apply our full capacity to present a strong proposal.

Further reading on defining questions that matter

Patton, M.Q. (2012), Essentials of Utilization-Focused Evaluation.  Sage.