Evaluating participation of nontraditional voices helps to understand whether the participation is working, both from the perspective of the participants and the organization.
Two types of evaluation are important:
- Evaluating the participation process, which you can do through a process evaluation.
- Evaluating the outcomes of having the participation of nontraditional voices informing your practice, which you can do through an outcome evaluation.
A process evaluation’s main purpose is to describe how participation is occurring. It will help measure the level of participation, consistency in participation, and activities where participation is occurring. Some process questions designed to track participation include:
- What types of nontraditional participants are part of your meetings (gender, race, age, county of origin, perspective, etc.)?
- How many meetings were held and how many nontraditional participants attended?
- How many orientation meetings were held and who participated?
- What materials were developed to support nontraditional participants and who received them?
- How many decisions were made with the involvement of nontraditional participants?
- Are nontraditional voices present through all phases of the discussion?
An outcome evaluation’s main purpose is to help the organization understand how nontraditional voices have had an impact on decisions and experiences. It may examine changes in knowledge, skills, attitudes, and organizational perspective. Outcome evaluations can track discussions leading up to the decisions made with and without participation of nontraditional voices, helping show how their involvement may have changed the decisions made. Some outcome measures a group can collect include:
- How have organizational policies, procedures toward nontraditional participants changed over time?
- Are nontraditional voices satisfied with the level of participation?
- How do the decisions made with nontraditional participants differ from those made without?
- What are the issues that nontraditional participants bring to the forefront prior to decision-making?
The next step is for the group to identify how it will know the goal is met. Identify the measure of success: do you need to ensure that each decision made by your group has a vote by a member of an underrepresented group? Or is the measure of success that the group took time to discuss and consider the issues brought up by these voices before making the decision?
Measures might include:
- How many decisions were made with input from nontraditional participants?
- How many decisions made mirrored the recommendations of the consumer focus group?
Other evaluation questions might include:
- Are other participants satisfied with the level and type of nontraditional participation?
- How have goals and outreach efforts incorporated input from the community?
The next step in the evaluation process is to identify milestones or timelines to help gauge success. Milestones need to be SMART: Specific, Measurable, Achievable, Relevant, and Time-bound. Achievable milestones give your group room to grow. Relevant milestones help you measure your progress on the evaluation questions.
- Recruit 5 nontraditional participants from target populations to participate in key informant interviews by the next quarter.
- Develop coalition orientation materials by next quarter.
- Connect all nontraditional participants to a liaison by next quarter.
Example outcome milestones include:
- 80% of participating nontraditional members will report developing new leadership skills.
- 80% of participating nontraditional members will report satisfaction with their opportunities to provide input to planning and decision-making.
- 80% of participating organizational members will report satisfaction with the level of nontraditional involvement in planning and implementation.
Once your group identifies milestones, the next step is to identify the types of information you need to collect to measure the milestones. You can start to identify activities to “count” on a tracking sheet. Easy milestones to track might include: the number of nontraditional voices participating regularly, the number of orientation sessions, etc. More complicated milestones to measure may include satisfaction of nontraditional voices with their participation and the leadership skills developed by nontraditional voices.
Since some data collection methods take more time and effort than others, it is a good idea to consider when the evaluation results are needed and what resources are available to collect the information.
Set aside time at a meeting to present, discuss and act on evaluation findings: what did you learn? How will you use the information?
- Create a short presentation of the process and outcome evaluation findings;
- Facilitate a discussion of what the findings indicate about the organizations efforts and the community feedback;
- Promote a discussion of strengths and how to improve in the future; and
- Mutually identify concrete steps to take to improve.
Findings are an opportunity for improvement, not a judgment of efforts. The language you use when you present evaluation findings can set this tone. Presenting evaluation results does not need to be one person’s responsibility. Nontraditional voices, organizational leadership, members who have taken on mentorship positions or an external evaluator may all help in the presentation of results.