Developmental evaluation does not always follow a standard sequence of activities as is common in formative and summative evaluations. Specifically, while formative and summative evaluations often begin with a careful exploration of the context and program, followed by the creation and execution of an evaluation plan, DE might follow a less predictable path.
However, if you’re new to DE and trying to understand how to begin the work, an outline of a process that can work in many settings may be helpful. In the sections of the toolbox to follow, specific tools and approaches for all of these initial steps are explored, along with sample tools and protocols that you can adapt to your setting.
Developmental evaluation does not have to be conducted by a single person. In fact, the larger and more complex the initiative, the more likely it is that you’ll need a team to be effective. Some of the benefits of a multi-person team include:
- Greater variety of perspectives to shape the evaluation’s learning.
- Greater on-the-ground availability to observe as things unfold.
- Decreased risk of individual bias driving data collection, analysis, and findings.
Roles on developmental evaluation teams can be broken up in many ways. Multi-site evaluations may have a different evaluator embedded across different sites, with a lead evaluator working across the sites. An evaluation team may include the lead evaluator who is actively engaged in the meetings and interactions with all of the partners, but other evaluators who help with data collection, analysis, and preparing results to share.
- For an example of how team members can be distributed across sites, see the OMG Center for Collaborative Learning’s story about a developmental evaluation for The Bill & Melinda Gates Foundation’s Community Partnerships Portfolio.
- The FSG blog, What Does Developmental Evaluation Look Like? explores how to engage in DE using a team.
As you enter into a new developmental evaluation, take the time to build an understanding of the participants, their strategy, and the context. This includes:
- Building familiarity with the context and strategy while building relationships with key stakeholders.
- Assessing the readiness for learning and adaptation.
- Building an understanding of developmental evaluation among key stakeholders.
For more about these first steps, see the section on Staying on Top of the Action.
Once the foundational work is underway, you can begin the learning process and create the ongoing structures for engaging in learning with the key stakeholders:
- Building the learning muscles of the key stakeholders by engaging them in learning activities (see the section on Learning Activities under Structure for the Learning Process), such as before and after action reviews.
- Generating a Theory of Change (see the section on Theory of Change under Structure for the Learning Process), either as your own exploratory tool or in partnership with key stakeholders.
- Generating the initial set of evaluation questions (see the section on Evaluation Questions under Structure for the Learning Process) collaboratively with the key stakeholders.
Some DE practitioners develop a learning framework or plan at this point in the process, while others use an iterative process of evaluation question generation and answering without an overarching learning framework.
For more about this stage of the work, visit the Structure for the Learning Process.
The ongoing implementation of DE includes a steady flow of data collection, analysis, shared interpretation of the findings with the key stakeholders, supporting the use of the findings, and generating new questions. Often new questions are being generated even as the current questions are being answered.
For more about this stage of the work, visit From Collecting Data to Generating Findings.
Unlike many other forms of evaluation, you will not be collecting longitudinal data, focusing on the same set of questions over a long period of time, or in general implementing a set of pre-defined steps. Rather, you are finding out what the most crucial questions are right now that could help your partners develop their strategy. Then you help answer them, and before you know it, you’re moving on to a different set of questions.
Tip 1: Never underestimate the power of a happy hour
Personal connection matters a lot in DE, more so than in traditional evaluation. The evaluator needs many different individuals who are willing to share their unique perspectives, sometimes in private and sometimes in public. Taking time to engage in a more personal and less formal way can build the trust that leads to frank conversations. It doesn’t have to be happy hour, but engaging with key stakeholders in more informal, relaxed, and personal settings can help build those relationships and shift the evaluator from an outside observer to a trusted colleague.
Tip 2: Your ideas may be right, but sometimes they aren’t as important as you think
When you’re the only person in the room who is charged with really looking at how the process is functioning, who is engaging in what ways, what issues are under the surface, what is happening externally that might matter, whether it’s achieving the intended outcomes, how the outcomes relate to the problem, etc., it can be easy to feel like you have unique insights. You probably do. But just because you have these insights doesn’t make them important.
In fact, sometimes what feels critical to the developmental evaluator doesn’t feel at all critical to the participants, even after a thoughtful dialogue led by the evaluator. Be prepared to give up your insights. Not all will be equally important, relevant, or even wanted by participants. It is not necessarily a sign that the group is avoiding an issue if they don’t agree with you that it’s an issue. It may, in fact, be a sign that they know their issues better than you. If you’re having trouble letting go, put the issue in a private “parking lot” of things to continue to monitor over time or revisit if needed.
Tip 3: You have biases and assumptions too
Part of your role is to surface assumptions made by the key stakeholders, from assumptions about causal relationships to assumptions about the external environment with its opportunities and threats. You also may help surface how biases are contributing to decisions, interpersonal dynamics, or overall initiative dynamics.
You will bring your own biases and assumptions to this work, and they will have a direct impact on what you see as noteworthy or in need of being tested. The first and most important step to addressing your own biases and assumptions is to assume they are influencing your choices and to try to identify them along the way. Concrete actions you can take to surface your biases and assumptions and decrease their influences include:
Using frameworks generated by others to help you think about the situation – this may prevent you from using your own past experience as a measure of what things ought to be.
Capturing multiple perspectives to inform your findings, and not dismissing the conflicting or outlier viewpoints, particularly when they don’t resonate with your views.
Collecting data systematically, not just informally or without structure. Even with informally gathered information, be systematic in how you keep and analyze your notes.
Analyzing your data. This sounds obvious, but in the heat of the moment you may be tempted to generate a theme from a set of check-in conversations because it resonated strongly with you. Take the time to analyze your data using systematic procedures to decrease the bias you introduce into how you use the information.
Talking with a colleague who is outside the project can help bring perspective to how you’re interpreting what is happening.
Pay attention to how people react to what you share. If your findings don’t resonate, take time to understand why, including asking them if they see any biases or assumptions in the findings. Remember that your data does not represent the “facts” of a situation, but rather represents one version of reality.
Tip 4: Provide value
As the developmental evaluator, you are taking up time – individual time and group processing time. If you cannot demonstrate value relatively early in your process, you will quickly find that less and less time is available. Fortunately, hidden dynamics and an overwhelming sense of uncertainty are often present at the beginning of an effort, so it is possible to show value quickly when you walk into an emerging effort. However, you also need to show value regularly, not just at the beginning.
One way to demonstrate that you are a true partner is by occasionally stepping outside the DE role to share some resources on decision-making models, facilitate a focus group for other purposes, help identify a speaker for an event, etc. Just like the other members of the effort are wearing multiple hats and ready to pitch in, so too should you be. Just be careful not to cross the line into an ongoing member of the strategy team who is contributing primarily in ways other than developmental evaluation.
Tip 5: Don’t make decisions based on your scope of work
Just as we wish more programs would focus on achieving their outcomes rather than merely delivering their services, you need to do the same. Your scope of work, even if overtly prescriptive, cannot be the driver for your developmental evaluation efforts. Focus on what is needed and then check in on whether it can happen within your budget, rather than focusing on what you promised to deliver and checking in on whether it meets the needs.
If you find yourself wanting to say, “I can’t do that, it’s out of scope,” and the ask has anything to do with DE, pause to assess whether it’s outside your scope or outside DE. If only the former, this is a great moment to practice what you preach and be thoughtful and open to adaptation in order to stay focused on the outcomes of the work.
Tip 6: Write your contracts differently
Want to avoid the problems highlighted in Tip 5? Developing a scope of work for a DE can and should look different than other evaluation scopes of work. Typically you do not have a pre-determined set of deliverables and timelines. Instead, the group decides together at the start and throughout how to prioritize the resources set aside for evaluation. That said, you do need to work with your partners to think about what the check-in points and processes will be for reviewing whether the developmental evaluation is adding value. When your deliverables are less concrete, alternative approaches can be helpful in the contracting process. For example, you may have a yearly debrief with the partners on the contributions of the DE work and an assessment of whether and how it should continue in the future.