Below are descriptions of and templates for a wide variety of data collection tools often used in advocacy evaluation. The tools presented here vary in the level of intensity, time, and resources or skills needed to collect and analyze data. When adapting tools, remember to only select the tools that match your time constraints and available resources, but that still work towards answering your evaluation question.
Adapting the data collection tools may be as simple as swapping out words so the tool fits the focus of your work. There may be times, however, where these tools do not meet your specific needs. If that happens, think about the idea behind the tools and what type of information the tool is trying to capture, and how that might apply in your own work to guide your adaptation.
- For an even more comprehensive list of data collection tools for advocacy settings and information on how to use them, please visit the Handbook of Data Collection Tools from the Annie E. Casey Foundation, prepared by Organizational Research Services.
Bellwether interviews are a method for determining where your issue is positioned in the policy agenda, how influential leaders think about your issue, and how likely they are to act on it. The method was first developed by the Harvard Family Research Project.
- Bellwethers are individuals who are seen as influential people in the public and private sectors, whose jobs require them to be politically informed and to track a broad range of policy issues. These individuals are thought leaders in your state. Often, they are innovators, and they are always individuals whose opinions about policy issues carry substantial weight and predictive value.
- Bellwether interviews take a unique approach to interviewing. The interviewees are unaware what the focus of the questions will be prior to the interview, ensuring they are able to answer authentically, unprompted, and without knowledge of your expectations.
- The Harvard Family Research Project has more detailed information on the method, including example questions.
An Intense Period Debrief is a protocol for a small group to debrief what happened during the implementation of an important strategy. For example, an Intense Period Debrief may be used after the completion of an attempt to pass legislation, hosting a large community forum, or completing a set of “pilot” activities.
The primary purpose of an Intense Period Debrief is to capture the lessons learned while implementing a strategy. Often, with a complex initiative, multiple partners are involved and each is engaging in a combination of preplanned and last minute activities that have a collective impact on the outcome. By purposefully discussing the activities completed, we can learn how barriers were overcome, how problems were solved, and which activities seemed to help versus hinder the process. By documenting the discussion and using a standardized protocol, we can better use the information the next time a similar strategy is implemented.
- For instructions on how to develop your own protocol for intense period debriefs, see this resource from the Innovation Network, who developed the methodology.
An observation checklist is a tool for capturing information about both presenters and participants in a wide variety of settings. It is appropriate for tracking what topics are being discussed, who is discussing them, the response of the audience, and whether key messages are being used.
For example, you could use an observation tool to track whether key individuals are using your framing or message, or what commitments are being made by your coalition partners. Observation checklists are a very flexible tool for purposefully and systemically capturing information across multiple meetings/events and allowing for comparison.
Example settings where the checklist may be useful include public hearings, testimony, community forums, presentations to community groups, trainings, or other gatherings where policymakers, advocates, consumers, and others are discussing advocacy issues.
An intensity of partnership assessment is a qualitative assessment of changes in your partnership with key organizations and individuals. The tool is used for two purposes:
- To clearly define priority relationships and the types of changes desired over time, which should help in determining activities needed; and
- To track progress over time as relationships develop and change, allowing for an assessment of the success in meeting partnership goals.
Policymaker ratings were developed by the Harvard Family Research Project and have been used in a variety of advocacy settings. The method is designed to increase understanding of political support for your advocacy issue, focusing on a specific subset of policymakers who you identify as important to your advocacy efforts.
Policymaker ratings are designed to draw on the expert knowledge of advocates who frequently work with different policymakers. The strategy works best when advocates (or their partners) rate policymakers with whom they have regular contact or who have verifiable information. The ratings are a set of scales that include policymakers support for, and influence on, the given issue.
- Can be completed a single time as information to guide strategy;
- Can be repeated in a regular basis to monitor changes and continue to guide strategy; and
- Are a very low-effort strategy for collecting information about political will.
- For additional information about policymaker ratings including when and how to use them, visit this resource from Innovation Network.
Constituent and champion tracking are strategies for monitoring the development of key players in your advocacy strategy.
- Constituent tracking allows you to monitor how your key constituents become advocates. You can track the number of constituents you have at each stage on the scale, or you can follow individual constituents to see how they develop so you are able to see who to focus time and effort on developing.
- Policymaker champion tracking is already something most advocates do informally. This is the ongoing counting of the number of actions different policymakers take that indicate they may be champions for your issue. It requires going beyond looking at voting behavior and paying attention to a variety of other actions policymakers can take.
- For an in-depth and practical handbook on champion tracking and measurement, visit Champions and “Champion-ness” written by David Devlin Foltz and Lisa Molinaro and released by the Center for Evaluation Innovation.
It takes years, not days or months, for advocacy work to truly make a difference. Often, it is not enough to contribute to passage of a policy; it can be critical to block attempts to change it and help ensure implementation is successful. Even achieving a policy win can take many years depending on the political environment.
Given this long-term timeframe, it is critical that advocacy organizations have the capacity and resources to sustain their work over time. Advocacy capacity assessment tools help organizations assess their strengths, identify areas for growth, and communicate with their partners about their organizational capacity.
- The Alliance for Justice has a free, online advocacy capacity assessment tool. This is an excellent tool that is relevant to most advocacy organizations. We strongly encourage you to use this tool or other existing capacity tools rather than design your own.
Surveys are appropriate for many settings and uses. For example, surveys are beneficial for measuring:
- Audience changes after an event or meeting;
- Engagement of participants in an advocacy strategy;
- Skills/knowledge changes after a training;
- Knowledge and awareness of your advocacy issue;
- Information about people’s perception of a problem; and
- Willingness to support an issue.
Surveys can be implemented in many ways, such as:
- On printed documents that are handed in or mailed back by participants;
- Online or emailed to participants; or
- By phone or in person.
A higher percentage of your participants are more likely to complete a short survey, while focused surveys that ask questions highly relevant to your advocacy strategies avoid the problem of having interesting information that you don’t have a clear way to use.
Tracking/logging is a strategy to monitor information about advocacy events, actions, or behaviors that can be easily observed by staff. Tracking is a low-intensity/resource strategy and is useful for monitoring activity or changes over time. This method can be particularly useful for long-term advocacy goals that may not have an immediate impact as tracking allows you to evaluate day-to-day advocacy efforts.
Types of information you may want to track include:
- Number, type and content of communications (phone calls, emails, newsletters, etc.);
- Participation in a wide range of events and activities;
- New stakeholders engaged and level of engagement; and
- Use of messaging in documents such as newsletters, newspaper articles, and flyers.
Tracking tools can be set up or using tables in Word or spreadsheets in Excel (or other document and spreadsheet software of choice). The advantage of using tables is that they are user-friendly and can be printed out and filled in by hand. The advantage of spreadsheets is that counts and charts can be generated automatically.