Outcomes, indicators and measures are often confusing as well as intimidating to non-evaluators, but this need not be the case. Key to a successful evaluation is determining upfront how to select and prioritize outcomes, how to determine what is realistic to measure, and how to make sure you are measuring outcomes that accurately reflect your program’s impact. Logic models are a tool that will help you in this task. It is also important to pay attention to the purpose of your evaluation as that is an important determinate of what you’ll want to measure.
We certainly can measure a lot of things that do matter, but there’s a lot out there that gets measured that does not matter. Keeping this advice in mind will keep you focused on what does matter and what can be measured. Let’s dig deeper by considering two key concepts: The Sphere of Control and the Sphere of Influence.
The Sphere of Control is, as the name suggests, those things we have control over (or at least we think we do!). It is the activities of our program, the deliverables to our funders, the timeline of milestones reached, and the number of people involved in our program. These are the things we DO. These are typically the things that we measure. They tell you about the effort you put in but they don’t tell you about the “why bother” of the program.
The Sphere of Influence is the change we hope to see occur through the farm to school programs we implement. These can be categorized into short, intermediate, and long term outcomes. For example, in a school garden program, a short term outcome might be for students to have increased awareness and knowledge about how plants grow, or an appreciation that where one lives determines what types of food can be grown and that being a locavore means prioritizing foods that are grown locally.
All of these examples are things we would hope a student would learn by participating in a garden program that included lesson plans about plant biology and local food systems. But, while you have control over creating the garden, having students participate in the garden, developing and teaching the curriculum you DO NOT have control over whether those outputs of your farm to school program will actually result in student learning. Learning falls within the Sphere of Influence.
The Sphere of Control leads to the Sphere of Influence.
As you move forward in planning your evaluation, these two concepts will help you identify which outputs (Sphere of Control) are most closely linked to which outcomes (Sphere of Influence). The more distance there is between control and influence the less successful your evaluation will be. The Guiding Criteria: Measure what matters (what you want to influence) AND what can be measured (what you can directly connect to the activities that you control).
One way to organize those things you control and those things you want to influence is to lay it out visually using a logic model. Logic models use words and/or pictures to describe the sequence of activities thought to bring about change and how these activities are linked to the results the program is expected to achieve.
The picture below shows the structure and content type found in a typical logic model. This graphic from University Extension in Wisconsin, is a simplified pictorial depiction of the relationship between your program’s activities and its intended effects.
The process for thinking through change includes three steps:
Identify the problem(s)
On the far left side depicted by arrows is information about what the problem is that you are addressing. Here, this is called the “situation” and within the situation are priorities that are linked to why this situation or problem is an issue. For example, you may work for a community garden organization. Within your organization’s mission, vision, values or mandates may be statements about the importance of increasing food system knowledge of youth and adults. A problem identified by your organization is the lack of youth knowing where their food comes from. Within the situation you may want to articulate why youth do not know – for example, a lack of agricultural/food system education within the schools, urbanization and the lack of an agricultural economy/way of life, and so on.
Name the desired results: what is that vision you have the future?
In a logic model, the desired results are your goals, which are called outcomes and impacts. There are three types of outcomes:
- Short-term, also known as the first outcomes you can expect your program to achieve if it is well-implemented. These are basically LEARNING outcomes: awareness, knowledge, attitudes, skills, opinions, aspirations, and motivations.
- Next come medium-term outcomes, those that you would expect to occur after the short term outcomes are achieved. These are ACTION outcomes: changes in behavior, practices, decision-making, policies, and social action.
- Finally, long-term outcomes also called impacts, follow upon the achievement of short and medium term outcomes. These are SYSTEM CHANGES, be they social, economic, civic, or environmental.
Develop your strategy
Between the PROBLEM and the VISION is the STRATEGY you will have developed to address the problem and reach your goal or future vision. The STRATEGY is shown on your logic model under the category of OUTPUTS. There are two components.
- The work/action/strategies of your program, such as building educational school gardens, conducting workshops for teacher, developing and implementing educational curriculum for students, teaching garden skills, and so on.
- Then there are the target groups of your program, or who you intend to reach, be they individuals or organizations.
When you put it all together, you have a graphic representation of your program, the participants you are targeting with your programming, and the outcomes you hope to achieve. The logic model is a great tool to help you plan your evaluation as it keeps you focused on your activities, target audience, and intended outcomes – each is information that is necessary in order for you to design your evaluation.
The Farm to School Evaluation Toolkit identifies seven target audiences that can be affected by a farm to school program. The seven audiences are:
- Food Service Staff and Operations
- School Leadership
- Community Members
Typically, farm to school programming focuses on students through garden activities, nutrition or food systems education, and local foods offered in the cafeteria. In practice, though, many other groups are affected by farm to school activities. For example, if teachers are using the garden to educate their students about science, do they perceive it as successful? What does success mean to them? Do they value the experiential learning that occurs through hands-on work? What are the challenges they have experienced? What about teachers not using the gardens as a “classroom” – do they see value in having a school garden? Are they interested in using the garden for educational purposes in the future? If not, why not?
There are short, intermediate and long-term outcomes for each of these audiences that are associated with farm to school activities. The Toolkit’s seven attachments – one for each target audience – provides guidance on how to select appropriate outcomes based on the activities in your program.
There are short-, intermediate-, and long-term outcomes for each of audience associated with farm to school activities. The Toolkit’s seven attachments – one for each target audience – provides guidance on how to select appropriate outcomes based on the activities in your program. The attachments accompanying this toolkit will help you identify outcomes that you want to measure as part of your evaluation, related to those audiences you identified in the previous step. Below is an example of how the attachments are laid out with respect to matching activities to outcomes. Look down the column “This may be a fit for your program if…” to identify activities you are doing, then look at the Outcome column, which identifies the type of outcome you can expect to see based on the activity.
Suggestions for selecting outcomes:
- Select only a few outcomes to measure. While many outcomes are likely to be achievable through the program components that are part of your farm to school model, you don’t want to measure all outcomes. Select those that are most important for achieving the evaluation purpose you identified.
- Make sure you select outcomes that are realistically achievable from your program. You want to evaluate yourself against those changes you are most likely to be able to cause.
- Make sure you select outcomes that your program, or the programs implemented by others who are helping to partner with you on this evaluation, can directly influence. Avoid trying to measure outcomes that are more related to activities happening outside of your control.
- Short term outcomes (learning outcomes) and intermediate term outcomes (behavior changes) are more easily linked to your activities than are long-term outcomes (systems changes).
- Remember you want to know what changes your program is changing, not what you hope it will change. Be realistic about what your farm to school activities can achieve.
The Target Audience Outcome Matrix will help you explore the audiences of your farm to school programs, select the appropriate target audiences, and identify specific outcomes you believe your program can accomplish related to those audiences.
Webinar #2: Outcomes, Indicators and Measures focuses on Step 2 in the Overview & Steps Guide and covers evaluation outcomes, indicators and measures. This webinar also provides an overview of how a logic model can help in the development of an evaluation plan. Several handouts are used during this training: the Overview & Steps and the Evaluation Plan Template and at least one of the Toolkit Attachments. This is an interactive training. By the end of the webinar, you will be able to continue filling out your evaluation plan (Step 2), specifically identifying outcomes you want to evaluate, and linking the activities of your program to the outcomes.