While there is an “art” to DE, it also requires the same technical skills as other fields of evaluation. Methods skills are important, including having a mix of qualitative and quantitative techniques to deploy. Developmental evaluation isn’t just reflection and learning – it depends on having systematically collected and analyzed information.
It helps to have a large toolbox of methods to reduce the risk of using a familiar tool just because it is familiar, when that tool might not be right for the moment. For example, rather than seeing your method to answer a question as a key informant interview, remember that a key informant interview is just a data collection technique – how you design that protocol and analyze the data is the “method.” Below are a couple of methods to spur your thinking, very briefly described and sited to sources that can give you more insight. Please also visit www.BetterEvaluation.com to learn about other methods you might be able to use.
There will be times where your toolbox does not have the right method available. In the context of DE, you rarely have time to learn an entirely new method, so it is very beneficial to have a network of researchers available that you can tap into as needed.
Story Formulation Methods
|Most significant change
||On-the-ground or the most affected participants generate stories about the most significant change observed during critical periods. Panel of designated stakeholders (i.e., steering committee) systematically filter those stories to choose the most significant of them, while documenting the reasons for their choices.
||…in capturing unexpected change and when you need to assess impact on participants’ lives.Can provide highly customized information to stakeholders across a diversity of implementation plans or outcomes.
||The Most Significant Change MSC Technique: A Guide to Its Use, by Rick Davies and Jess Dart
|Intense period debriefs
||Focus group-style discussions with key stakeholders immediately follow periods of intense activity.
||… when information is needed from multiple perspectives involving strategies, processes, decisions, outcomes, and roles and relationships.
||Data Collection for Advocacy Evaluation: The Intense Period Debrief, from the Innovation Network
||A form of action research that takes a positive- rather than problem-focused lens. Participants are asked to describe good moments when they felt they organizations were working and motivated. They reflect on these moments to identify what success looks like and what they can achieve.
||…when the effectiveness of developmental evaluation depends on overcoming barriers to trust or biases or assumptions (often negative) are at risk of preventing progress.
||Appreciative Inquiry Commons, a website through Case Western Reserve University
|Rapid reconnaissance (or Sondeos)
||Involves a team of evaluators who collect various types of quick, but intensive data, including semi-structured interviews with and observation of participants.
||… when something key has changed, time and resources are limited, you need to establish quick baseline conditions, or the problem space cannot be identified prior to project start.
||The “Sondeo”: A Rapid Reconnaissance Approach for Situational Assessment
|Systems mapping/ systems diagrams
||A structured method for representing a system and all its complexity, focusing on visually mapping how people and organizations relate.
||…when you need to surface beliefs about how change will occur, identify new avenues for change, or evaluate the expected and unexpected changes resulting from an intervention.
||Spotlight: Systems Mapping for Advocacy Planning and Evaluation by Andy Stamp and Julia Coffman from Innovation Network
|Social network analysis
||The mapping and measurement of relationships between people or organizations.
||… when there are uncertainties or concerns about how the network of people or organizations will function, such as how information flows or where trust needs to be built.
||Social Network Analysis, A Brief Introduction on Orgnet.com
||A methodology for planning and assessing development programming that is oriented towards change and social transformation.
||… when there is uncertainty or different perceptions among stakeholders about the potential outcomes emerging from the initiative.
||Outcome Mapping on BetterEvaluation.org
||A tool to visualize relationships among people and organizations and to assess the relative impact of these pathways.
||… when there is uncertainty around the best avenues of potential influence over specific decision-makers.
||Power Mapping: Charting Strategic Relationships from DFA Training Academy
||A method for understanding the stake that different people have in an effort and its potential impact.
||…when you need to build an understanding of the setting up front and there is uncertainty about the stakes and stakeholders.
||Stakeholder Mapping and Analysis on BetterEvaluation.org
Methods for Exploring the Future
||Scenario development is a technique for identifying the variety of future environments that may unfold and the drivers of those futures. They prepare the strategists for many potentials, decreasing the risk of paralysis when the unexpected happens.
||… when the strategy being implemented is very context-dependent and is being implemented in a highly variable environment.
||The Use and Abuse of Scenarios by Charles Roxburgh from McKinsey & Company
||Simulations are the imitation of a real-world process, beginning with developing a model that represents the behaviors/functions that one would expect to see.
||… when you want to assess the effects of a potential change in process, stakeholders, priorities, capacities, etc.
||No resource currently available. Know of a good resource? Please let us know!
||The “hypothetical opposite of a postmortem” where stakeholders generate plausible reasons why a project might fail at the beginning of the process.
||…. when you sense stakeholders are hesitant to speak up about potential problems or overly optimistic in their outlooks, which could decrease their ability to anticipate issues.
||Performing a Project Premortem by Gary Klein in the Harvard Business Review
Methods for Uncovering Underlying Themes, Causes and Relationships
|Root cause analysis
||A tool to identify what, how, and why an event or circumstance happened. A list of causal factors is generated (e.g., the “Five Whys”) and logically mapped to identify the highest-level underlying cause.
||…when dealing with system(s)-level change and the symptoms tend to be addressed rather than the underlying causes.
||Root Cause Analysis for Beginners by James J. Rooney and Lee N. Vanden Heuvel, published in Quality Progress
||A process to brainstorm and organize a large number of ideas into common themes or categories. Participants generate ideas in groups, then work iteratively to categorize them, highlighting relationships and themes.
||…when you need to identify common themes, causes, or solutions. Also helpful when you need to dissect a complex issue and underlying relationships or influences.
||Affinity Diagrams on Mind Tools
|Causal Loop Diagrams
||A tool to map cause and effect relationships and to separate drivers from outcomes. Often used in conjunction with affinity mapping.
||…when there are multiple sources contributing to the problem, and you need to disentangle those sources.
||Systems Behavior and Causal Loop Diagrams from Arizona State University
|Force field analysis
||A tool that emphasizes the balance between restraining forces, barriers to change, and driving forces, factors supporting change. Stakeholders assess the number and strength of both restraining and driving forces to determine how to limit restraining forces while encouraging driving forces.
||…when decisions are difficult to make with many reasons for and against each choice and when stakeholders need to strengthen the forces supporting the change and weaken those against it.
||Force Field Analysis from Mind Tools
|Complexity aware monitoring
||A method to separate complex aspects of a project from simple and complicated aspects. Complex aspects are characterized by interrelationships, non-linear causality, and emergence.
||…when you want to track the unpredictable, there is low certainty and agreement over solutions, cause and effect relationships are not well understood, and/or situations are highly dynamic and strategies must be adaptive
||Complexity-Aware Monitoring from USAID’s Monitoring and Evaluation Series
|Soft systems methodology
||A seven step problem-solving technique incorporating a more holistic or humanistic approach to systems analysis.
||…when political, social, or cultural contexts complicate the problem space.
||Soft System Methodology from Action Research and Evaluation On-Line