Program Development
Use what you learn from each round of evaluation to drive toward greater impact.
Intro to Program Development
Many teams enter the evaluation process intending to use what they learn to make their program better. But it can be challenging to follow through on that intention.
It takes time to gather and analyze data, and it’s easy to lose sight of the questions you started with—people get busy, and often by the time the results of an evaluation come in, everyone has already moved on and is occupied with other things. Also, team members with evaluation expertise may take the lead with data analysis and interpretation while other members of the team may be less involved or left out entirely and miss out on the opportunity to help interpret the data and learn from the results.
The IDEAS approach to fast-cycle iteration helps overcome the challenges stated above and ensures that your team gets the most out of the evaluation process. In this section, you’ll review three questions to guide the fast-cycle iteration process:
- What does the data say?
- So, what does this mean?
- Now what do we do?
These reflection questions are included in the fast-cycle iteration worksheet, so you can use the worksheet to prepare for an evaluation, make sense of the data once it’s been collected, and identify next steps.
What does the data say?
The first question in the fast-cycle iteration process is, “What does the data say?”
To answer this question, begin by revisiting the Theory of Change-based questions you identified when you developed your evaluation plan:
- Which domains of your ToC did you set out to explore?
- What did you hope to learn?
- What information or data did you gather and what does the data say?
Consider all relevant data sources, including quantitative, qualitative, and practice-based observations. Your aim is to describe what the data says in straightforward, factual terms. Depending on the study design you employed and the tools you used to gather information, you might include average or mean scores, statistical significance values, effect sizes, percentages, or themes. You might find it helpful to graph average change over time and individuals’ change trajectories—this can help you examine variability in impact.
So, what does this mean?
The next question in the process is, “So, what does this mean?” Here you’re trying to make sense of the data. How would you explain the findings in plain language, and what are the implications? For targets and outcomes, weigh the strength of the evidence considering the study design, sample size, and type of data you collected. It might be helpful to envision a continuum, with “weak or no evidence of change” on one end and “strong evidence of change” on the other. Then, select the language that best summarizes the evidence’s strength when all sources of information are considered.
You might find it helpful to map the findings onto your ToC by placing a colored dot next to each construct to represent the strength of the evidence of change, with red corresponding to no evidence, yellow corresponding to some evidence of change, and green corresponding to strong evidence of change.
Consider the magnitude of change you hope to see on each target and outcome:
- What amount of change would be practically or clinically significant?
- What amount of change have other comparable programs been able to achieve, and with what resources?
Now what?
The final question in the fast-cycle iteration process is, “Now what?” Take the whole picture into account in identifying what you might do differently going forward.
Consider implications for the ToC, ongoing program development, and evaluation plan:
- Regarding your ToC, are there constructs that need to be added, removed, or refined?
- What are the implications for your program and program materials?
- How might you improve or adapt your approach to evaluation in the future?
On the second tab of the fast-cycle iteration worksheet, you’ll find a simple action plan template that can be used to identify your top priorities and make a plan for moving that work forward.
The El Busesito team engages in continual learning and improvement.
In the summer of 2022, the El Busesito team gathered for an evaluation retreat to review the data from the past year. Using the reflection questions above, the team worked together to interpret the data and identify next steps. The following is an example of something they learned and how they’re planning to act on that knowledge. See the team’s full fast-cycle iteration worksheet below for more detail.
What? Participants’ letter recognition increased from 17% in the fall to 30% in the spring.
So what? Although participants increased their literacy skills over the course of the year, the level of their skills is still relatively low.
Now what? The team identified several potential solutions, which included modifying math and literacy activities in the lesson plan and incorporating a new curriculum. These changes will be reflected in the program materials and ToC. In the next round of evaluation, the team will gather information on the impact of those strategies.
The El Busesito team engaged in shared learning by presenting their findings to the rest of the Valley Settlement staff as part of their annual evaluation retreat. During this event, each of the organization’s six program teams presented their learning from the past year and discussed lessons learned and next steps. Valley Settlement’s organization-wide commitment to ongoing learning and improvement has made them a trusted and impactful organization in their community.
We hope that you’ve found this toolkit helpful and we wish you the best on your journey toward greater impact at scale.
In the words of Ann Mei Chang, author of Lean Impact
“While invention is the spark of a new idea, innovation is the process of deploying that initial breakthrough to a constructive use.”