Once you’ve developed relationships and have begun to understand the strategy, context, and the vision that partners have for the change they want to cause in the world, the foundation is laid to begin the learning process that is so critical for DE to be successful. That process includes building the learning muscles of the key stakeholders, generating evaluation questions together, and sometimes creating Theories of Change.
Some DE practitioners develop a learning framework or plan at this point in the process, while others use an iterative process of evaluation question generation and answering without an overarching learning framework.
A learning framework is a tool for articulating the potential challenges and opportunities the partners need to learn about to be successful, including identifying what both the partners and the evaluator will pay attention to along the way. Developmental evaluators who choose to use this approach may develop it initially after doing one-on-one interviews and observing key meetings and activities. It is a living document and will guide evaluation questions, data collection, and learning discussions ongoing.
- An example learning framework is included in the appendix of DE 201: A Practitioner’s Guide to Developmental Evaluation, by Elizabeth Dozois, Marc Langlois, and Natasha Blanchet-Cohen, published by the J.W. McConnell Family Foundation.
Developmental evaluation questions sometimes focus on the same issues as traditional evaluation, but they can also explore many other issues and through many different frames. Questions can vary from a focus on causal connections to emergent patterns, an examination of different perspectives to building an understanding of drivers and solutions.
The worksheet below identifies different types of questions, provides examples and gives you a place to craft your own questions. You can use this tool to further your own thinking or in partnership with key stakeholders to help them explore the questions they want answered.
Some evaluation questions are about experimenting: What did we do? What happened as a result? What could we do differently? One of the roles of DE is to help with rapid experimentation on different strategies that help tackle pieces of the larger issue. Not only can you collect data and facilitate learning about the specific experiment, you can also facilitate the dialogue about how the experiment fits into the broader context.
Sometimes a Theory of Change is a powerful tool in the process of developmental evaluation, but it functions differently than in a typical evaluation. Due to the developmental context, the theory of change cannot be designed to describe the road ahead as it should be, but rather how it is currently articulated. Below are three ways that a theory of change can be helpful:
- As an independent, exploratory activity. A Theory of Change can be a tool to bring together the different messages you are hearing about why the work is important and how the strategies relate to outcomes. It can help you explore where there are assumptions being made, gaps in the flow of outcomes, or conflicting beliefs among stakeholders.
- As a planning activity led by stakeholders. Sometimes stakeholders have already decided that a Theory of Change is a good tool for their process and will ask for help generating it. In these cases, the focus is on generating a document that can help with their developmental process, rather than the evaluation, including focusing on the strategies they need to explore, the outcomes they care about, and where uncertainty exists.
- As an evaluation tool developed with stakeholders. There are times when a theory of change is a shared tool for evaluation. By articulating with stakeholders their assumptions about how the work they are doing will contribute to outcomes, it can help surface evaluation questions and contribute to the stakeholders actively testing their assumptions. The Theory of Change must be adaptive, changing as new learning is generated. Often it has less emphasis on strategies and instead focuses on chains of outcomes that might help achieve the ultimate end goals.
Before you go too far into using Theories of Change in the context of a DE, recognize that they pose a risk: by visually documenting the current beliefs about the problem and its solutions, you can give your partners a false sense of certainty or predictability, which may limit their interest or even ability to look at the problem through other lenses. It is better to use Theories of Change only when there is a specific value that is worth this risk, rather than as a default tool.
The protocol below offers one process for creating a Theory of Change in the context of developmental evaluation.
- For an example of using a Theory of Change with stakeholders, see the OMG Center for Collaborative Learning’s story about a developmental evaluation for The Bill & Melinda Gates Foundation’s Community Partnerships Portfolio. See the OMB Center for Collaborative Learning: Community Partnerships Portfolio under the Community Pages
- There are many great resources for developing theories of change, but few explore them in the context of DE. For an introduction to Theories of Change, please see the resources on www.BetterEvaluation.org.
There are a variety of ways to engage with your partners as part of DE. There is no agreement on which of these ways appropriately belong with DE and which should stay in the realm of strategy coaching. A developmental evaluator intervenes in the development of a strategy or innovation when he or she brings evaluative questions, observations, or data to the partners, particularly leading up to decision-making points. This may look like:
- Asking questions at key moments to create space for participants to explore something that would otherwise be left unsaid, or to pause the process when there is a risk of making a decision too quickly.
- Providing new information, both at pre-planned moments in time and as relevant to the dialogue underway, drawing on the ongoing data collection and analysis occurring.
- Facilitating learning dialogues around evaluation questions, including moving from exploration of findings to decisions on actions to take.
- Reminding participants to consider previous issues, learning, or decisions before moving into a new decision or taking action.
- Mapping and modeling, including Theories of Change, systems mapping and other ways of helping participants describe their understanding of how change will occur and why.
Learning when and how to engage in any of these actions takes time, not just in learning the skill of DE, but also in any new initiative where you are deploying DE skills. Every group has its own style and processes. What may be a sign of ignoring a key issue in one group could simply be an indication of a different way of working through issues in another group.
- For an example of types of engagements a developmental evaluator has used, see the OMG Center for Collaborative Learning’s work with The Bill & Melinda Gates Foundation’s Community Partnerships Portfolio.
- For a thoughtful list of the types of DE interventions, see page 50 in DE 201: A Practitioner’s Guide to Developmental Evaluation by Elizabeth Dozois, Marc Langlois, and Natasha Blanchet-Cohen, published by the J.W. McConnell Family Foundation.
Developmental evaluation is only successful if the learning generated through the evaluation is used to inform decisions about strategy. For this to happen, results need to be shared in different ways from the traditional evaluation.
There are many powerful techniques for engaging in learning. Below are just a few to explore. Each provides a way to introduce findings, include intuitive learning from participants, and generate actions to take in response. It is important to begin using learning techniques early in the developmental evaluation, before any of the very challenging issues have been surfaced. By the time the really tough conversations are happening, it’s too late to develop the learning muscles and you risk straining what capacity for learning does exist.
Note that, with all these techniques, the results of the evaluation are not delivered through a formal report. In developmental evaluation, the findings need to be useful and timely, rather than documented in a more structured manner. Expect to share findings through formal and informal conversations along with memos, emails, presentations, etc.
Emergent learning is a package of tools that collectively create a systematic approach to developing more effective strategies through purposeful reflection on past experience (including through data), the current context, and the desired outcomes of the work. It provides a steady process for building learning over time, creating a track record of what works, while also helping to understand how to improve.
The emergent learning toolbox includes three key tools, each of which is a facilitated process that you, as the evaluator, can lead, or that you can engage a facilitator to lead:
- Before Action Reviews: Thoughtful discussions prior to a strategy being implemented that use past experience, including evaluation findings to craft a strategy to achieve the outcomes desired. These can be used for many strategies, from events or key meetings to upcoming components of the overall strategy.
- After Action Reviews: Reflective dialogues after a strategy was implemented that break down what happened and why, in relationship to the desired outcomes of a strategy.
- Emergent Learning Tables: Structured conversations that seek to answer a key framing question about how to achieve something important to the participants, using a mix of reflecting on past experience and evaluation findings, culling key insights, generating ideas about how to be effective in the future and making commitments to implement the ideas.
- For an example of using Emergent Learning in DE, see the Missouri Foundation for Health’s Infant Mortality Initiative in the Community Pages.
- Fourth Quadrant Partners’ website on Emergent Learning: http://www.emergentlearning.com/ introduces the concepts and links to resources for more information.
- For an understanding of the emergent learning table specifically, Growing Knowledge Together: Using emergent Learning and EL Maps for Better Results by Marilyn Darling and Charles Perry provides a wonderful introduction and examples.
- For an understanding of before and after action reviews, Learning in the Thick of It by Marilyn Darling, Charles Parry and Joseph Moore provides a great preview of how to use them. It is for subscribers of the Harvard Business Review only, but you can also access a copy through Fourth Quadrant Partners.
A Strategic Learning Debrief is a specific process for helping participants to collectively understand and interpret the findings from various data sources, considering them within the context of their theory of change, and identifying changes to their strategies. The Strategic Learning Debrief is an opportunity to purposefully and collectively:
- Present and explore different types of learning in the context of the intended and unintended outcomes of the strategy, including systematic data collection and analysis as well as intuitive and experiential learning by participants.
- Reflect on the learning through a process of collectively interpreting what it means and building a shared understanding of the implications.
- Apply the learning directly to the strategy and determine actions that need to be taken, whether changes in strategy or changes in learning.
Critical features of a successful Strategic Learning Debrief include:
- All participants are equal. The evaluators in the room have no more authority over how evaluation findings are interpreted than any other participant. In fact, the developmental evaluator is there to ensure participants interpret the results, not to interpret them for the participants.
- All information can be included. Both intuitive learning (not based in formal evaluative data) and learning from data collection tools should be included.
- All information is not equal. While intuitive learning is important, it should not drive a change in strategies without other evidence to support it.
- The facilitator and note-taker are not participants. Their role is to ensure a successful dialogue and documentation of decisions. If they actively participate in the content or direction of the interpretation, the overall success of the debrief will be harmed.
The protocol below describes the process of a strategic learning debrief and provides a sample protocol.
- For a description of how to implement along with a case study, see Strategic Learning in Practice: Tools to Create the Space and Structure for Learning by Jewlya Lynn, published by the Center for Evaluation Innovation.
- For examples of learning resulting from a Strategic Learning Debrief, see If You Build It They Will Come: Creating the Space and Support for Strategic Learning by Jewlya Lynn, Phillip Chung, Scott Downes, and Rebecca Kahn, published by the Foundation Review.
Submitted by Carolyn Cohen of Cohen Research & Evaluation
Learning Circles are a facilitated process for engaging an organization’s leadership, project staff, and key partners in collectively examining and reflecting on real-time data. The group jointly agrees on the focus of each session, thereby ensuring that the discussion is salient to its near-term information needs. Learning Circles are designed to deepen understandings of emergent outcomes, and inform adaptation and decision-making.
Learning Circles use an appreciative inquiry approach and, where appropriate, leverage Theories of Change, two tools that can be helpful on their own in DE.