Now that you know the reason why you should use evaluation for your advocacy work, let’s talk about how to do it. This part of the toolkit walks you through the basic steps of how to develop an evaluation of either specific advocacy strategies or your organization overall, using a tool called the “Evaluation Focus Worksheet.” Even if there is only one small component of your work you want to evaluate, it is important to work through these initial steps to ensure your evaluation is useful to you.
The first step in any evaluation design, large or small, is being clear about who will use the results of the evaluation and for what purpose. Example audiences include:
- Program staff, organizational leaders, or board members;
- External partners, including coalition members or other stakeholders; and
- Fundraisers within the organization or the funders outside the organization.
Example purposes for evaluation include:
- Accountability and/or understanding the impact of the effort;
- Understanding the environmental context and its influence; and
- Periodic strategy improvement or real-time strategic learning.
For example, your purpose and audience might look like:
After selecting the primary audience and their purpose for evaluation, you are ready to focus the evaluation!
- For more resources on how to identify the audience and use of an evaluation, review Step 1 of the User’s Guide to Advocacy Evaluation Planning.
The next step in evaluation planning is to create a visual that represents your advocacy strategies and their relationship to the policy changes and broader impact on society you are hoping to achieve. To simplify, on one side of the visual are the things you have control over and on the other side are the things you hope to influence.
In between the day-to-day work and the policy change you’re trying to achieve are interim outcomes, or those smaller changes that occur along the way. Interim outcomes include many different things, from changes in how policymakers understand and prioritize your issue to framing in the media to recruitment of new advocates and the actions they take.
The visual map you create by documenting your strategies, identifying the interim outcomes likely from those strategies, and articulating the policy change and impact you are hoping to achieve is called a Theory of Change. You may also know the term logic model, which is a similar type of document.
Working through a Theory of Change will surface certain assumptions or gaps that exist between strategies, interim outcomes, and policy change. Visual documentation of the changes you’re hoping to cause can also help you identify any gaps in your strategies – things that may be needed to get to the policy change you are hoping to achieve. Finally, Theories of Change can help you to narrow your evaluation focus by identifying the interim outcomes most important for you to measure.
- For a quick introduction to the step involved in constructing a theory of change, check out this brief primer by ActKnowledge.
- For more information about Theories of Change, see the Introduction to Theories of Change from the Harvard Family Research Project.
- For an interactive online system specifically designed to help advocates develop Theories of Change, as well as think about the evaluation overall, see the Aspen Institute’s Advocacy Progress Planner.
Most advocacy strategies have many different activities (within the Sphere of Control) that contribute to one or more changes (within the Sphere of Influence) in your participants, your community, or the policy environment. These changes are those things outside your control, those you can only hope to influence. Below is a list of interim outcomes that are part of this Sphere of Influence and common in advocacy efforts:
- Organizational capacity and diversified funding;
- Awareness and salience of your issue;
- Partnerships and partners who are aligned and collaborating on strategies;
- New advocates and new champions;
- Public will and political will; and
- Issue framing, issue reframing, and earned media coverage.
- This User’s Guide to Advocacy Evaluation Planning guide provides a list of typical outcomes for advocacy strategies including a great worksheet to think through the relationship between your strategies, outcomes and policy goals.
- The Advocacy Strategy Framework from the Center for Evaluation Innovation offers a simple one-page tool for thinking about the theories of change that underlie policy advocacy strategies.
The next step in your evaluation planning is to review the outcomes on your Theory of Change and ask the question: Which are most important for us to better understand, including whether and how we’re achieving them?
By narrowing the focus to the interim outcomes that matter the most, you decrease the time and effort needed to complete the evaluation. A sample focus might include:
Now that you have a focus – the interim outcomes you want to learn about – it’s time to write an evaluation question.
Evaluation questions are an important way of ensuring the data you collect will be useful. By taking the time to carefully craft the question you hope to answer, you are giving yourself direction on what information will be useful and what information is interesting, but not directly related.
Evaluation questions often start with:
- How did our…
- What happened when…
- What influenced…
- What changes did we…
- What patterns do we see…
Evaluation questions often include:
- …contributed to…
- …among our audience…
- …most likely to…
- …resulted from…
- …by [a specific date]…
- …within [a specific timeframe]…
- What changes did we see in how targeted policymakers framed the policy issue over the course of last year’s legislative session?
- What patterns did we see in how our coalition partners took action after participating in our Day at the Capitol?
- How did our key message spread through earned and social media over the last six months?
To add an evaluation question to the earlier example:
It is important to consider when you plan to use information from an evaluation and design the evaluation accordingly. Do you have a short timeframe during which you can use the evaluation information to improve your advocacy strategies? If so, you want to ensure your data collection and analysis can realistically happen within that timeframe. Is your timeframe longer, with time to do more in depth evaluation work? Or does the evaluation need to generate useful information steadily, every three or six months? These questions will shape what types of data you can collect, how quickly you need to move from collecting to analyzing, and what decisions you can inform.
Including a sample timeframe, your question may look like:
Similarly, before identifying data collection strategies, you want to assess your capacity to undertake data collection, analysis, and reporting.
Consider asking questions such as:
- Do you have staff time to conduct interviews, observations, deploy a survey, analyze the data, etc.?
- Do you have staff with skills and experience with basic quantitative analysis, such as identifying means, percentages, or otherwise manipulating numbers in Excel or another program?
- Has anyone on your team analyzed qualitative data in the past? If not, is anyone ready to build this skill?
- Do you have an account with an online survey service? Is anyone familiar with how to use the service’s analysis supports?
- What data already exists and is already being summarized/used that can contribute to the evaluation?
Before selecting your data collections tools, have you:
- Clarified who will be using the results of your evaluation and for what purpose?
- Articulated the impact you are hoping to achieve with your work?
- Identified activities, outputs, and interim outcomes towards meeting your impact goal?
- Considered your timeframe and organizational capacity during that timeframe?
- Prioritized what aspects of the evaluation will demonstrate the unique contribution of your advocacy effort?