Use what you learn from each round of evaluation to drive toward greater impact.
Intro to Program Development
Many teams enter the evaluation process intending to use what they learn to make their program better. But it can be challenging to follow through on that intention.
It takes time to gather and analyze data, and it’s easy to lose sight of the questions you started with—people get busy, and often by the time the results of an evaluation come in, everyone has already moved on and is occupied with other things. Also, team members with evaluation expertise may take the lead with data analysis and interpretation while other members of the team may be less involved or left out entirely and miss out on the opportunity to help interpret the data and learn from the results.
The IDEAS approach to fast-cycle iteration helps overcome the challenges stated above and ensures that your team gets the most out of the evaluation process. In this section, you’ll review three questions to guide the fast-cycle iteration process:
- What does the data say?
- So, what does this mean?
- Now what do we do?
These reflection questions are included in the fast-cycle iteration worksheet, so you can use the worksheet to prepare for an evaluation, make sense of the data once it’s been collected, and identify next steps.
What does the data say?
The first question in the fast-cycle iteration process is, “What does the data say?”
To answer this question, begin by revisiting the Theory of Change-based questions you identified when you developed your evaluation plan:
- Which domains of your ToC did you set out to explore?
- What did you hope to learn?
- What information or data did you gather and what does the data say?
Consider all relevant data sources, including quantitative, qualitative, and practice-based observations. Your aim is to describe what the data says in straightforward, factual terms. Depending on the study design you employed and the tools you used to gather information, you might include average or mean scores, statistical significance values, effect sizes, percentages, or themes. You might find it helpful to graph average change over time and individuals’ change trajectories—this can help you examine variability in impact.
So, what does this mean?
The next question in the process is, “So, what does this mean?” Here you’re trying to make sense of the data. How would you explain the findings in plain language, and what are the implications? For targets and outcomes, weigh the strength of the evidence considering the study design, sample size, and type of data you collected. It might be helpful to envision a continuum, with “weak or no evidence of change” on one end and “strong evidence of change” on the other. Then, select the language that best summarizes the evidence’s strength when all sources of information are considered.
You might find it helpful to map the findings onto your ToC by placing a colored dot next to each construct to represent the strength of the evidence of change, with red corresponding to no evidence, yellow corresponding to some evidence of change, and green corresponding to strong evidence of change.
Consider the magnitude of change you hope to see on each target and outcome:
- What amount of change would be practically or clinically significant?
- What amount of change have other comparable programs been able to achieve, and with what resources?
The final question in the fast-cycle iteration process is, “Now what?” Take the whole picture into account in identifying what you might do differently going forward.
Consider implications for the ToC, ongoing program development, and evaluation plan:
- Regarding your ToC, are there constructs that need to be added, removed, or refined?
- What are the implications for your program and program materials?
- How might you improve or adapt your approach to evaluation in the future?
On the second tab of the fast-cycle iteration worksheet, you’ll find a simple action plan template that can be used to identify your top priorities and make a plan for moving that work forward.
The El Busesito team engages in continual learning and improvement.
We hope that you’ve found this toolkit helpful and we wish you the best on your journey toward greater impact at scale.
In the words of Ann Mei Chang, author of Lean Impact
“While invention is the spark of a new idea, innovation is the process of deploying that initial breakthrough to a constructive use.”