If you do nothing else as a developmental evaluator, at the very least, you are there to help ask evaluation questions, offer systematic data and observations and, ultimately, help your partners to foresee and manage potential problems and opportunities. Developmental evaluation can take you much deeper, helping to untangle issues that are already known but difficult for participants to manage, helping identify pathways to success, and helping resolve conflicts with new information.
To do any of these, you need to stay on top of the action, including building an understanding of the context, the strategy, the reason the work matters to the stakeholders, the barriers, and the opportunity. There are many ways to do this. Below are three examples that can be effective in many different settings.
Nothing can replace the value of the one-on-one conversations. A critical first step is often to meet with each key partner to build the relationship and begin to understand their unique perspectives.
Note that these are not traditional interviews, but rather conversations that collect information to be systematically analyzed. The goal is to follow the person where they go and dig into the issues they care about while surfacing insights about some issues you have identified as important to investigate further.
It can be useful to repeat one-on-ones regularly, such as every quarter in a fast moving initiative or twice a year in a slower moving effort. As the trust is built, more and more of the “real story” will be revealed through these conversations. It’s not uncommon to have a participant thank the evaluator for the conversation in the end, saying how great it felt to talk about the things they feel are important, but are left out of the current conversation.
It’s important to analyze the data, particularly if you have many stakeholders, rather than rely on your gut instincts about what issues emerged and how. You can limit your analysis to coding the major themes of thinking, making it easier to link together how one person is talking about an issue with how another is talking about it. You can then use the results to flag issues to observe, do additional data collection around, or help the group to surface and discuss.
The protocol below describes the design, implementation, and analysis of one-on-one conversations. The sample protocol is a mix of key questions and prompts to dig deeper into the issues that are surfaced.
Developmental evaluation assumes that the work is in development, ever adapting and changing. As the developmental evaluator, part of your role is to be present, observing the changes as they happen. You can observe many different things, from the relationships and decision-making dynamics, to the underlying assumptions driving the work, the relationship between the strategy and the external environment, how participants define success and failure, the outcomes emerging, and more.
Observation in the context of DE does not have a pre-defined protocol that stays the same, and it may never result in a need to compile and analyze notes across a long period of time. At the same time, observations should not lack the rigor of evaluation. Rigor in observation helps to increase the quality of the data you’re collecting by guarding against your own biases and creates a base of knowledge that you can draw upon in the future when an evaluation question suggests a need to look back and to identify patterns that are driving successes or failures as the strategy unfolds.
To support this rigor, it can be helpful to have a protocol that flags the issues you want to pay attention to, while leaving room to observe what else emerges. Then plan to compile and analyze your notes, perhaps during a short window of time when critical meetings are occurring or to look at patterns over a longer time.
The protocol below describes the design, implementation, and analysis of an observation protocol appropriate for a DE setting and provides a sample protocol.
A simple and highly effective data collection tool for rapid learning, particularly when you have a large group of key partners, is a short survey called the “Right Now” survey. It’s never more than half a page and is handed out and completed during five minutes of an existing meeting. The questions always include some variation of:
- Right now, the greatest opportunities for success are…
- Right now, I am most concerned about…
- Right now, I most need help with…
The tool surfaces the issues that participants may not be sharing publicly, perhaps because there is no place on the agenda to talk about the issue, or they aren’t ready to disclose their concerns or ideas to everyone. Once you have established relationships with participants, you may not need this as they might come up and tell you what’s really going on. But in some settings, the number of stakeholders is large enough that without a tool like this, you will quickly lose sight of the breadth of opinions and understandings.
The protocol below describes the design, implementation, and analysis of the “Right Now” survey and provides a sample.