Welcome to the Community Pages of the Strategic Learning Toolkit. The pages in this section have been created by evaluators and partners of strategic learning who want to share their experiences. Please contact us if you’re interested in contributing to the Community Pages.
In 2010 The Colorado Trust launched a three-year, statewide strategy to build public will to help achieve access to health for Coloradans. The strategy, Project Health Colorado (PHC), was designed to empower individuals, organizations and networks to engage audiences around a set of shared values, ultimately moving them to take action to build awareness around access to health themselves. The strategy included multiple components working independently and in concert to build public will, including: a common message framework; funding for 14 grantee organizations; a paid media and mobilization strategy; grantee convenings designed to build grantee skills and networks of partners; and an evaluation that included real-time, data-driven strategic learning to assist grantees and The Trust in testing and adapting their strategies to improve outcomes.
In the context of strategic learning, theories of change (TOCs) provide specific, achievable interim outcomes on the way to longer term, more significant changes in the world. Ideally, these outcomes are achievable in a few months, allowing for steady assessment of progress. The Trust developed an initial theory of change focused on a 10,000-foot view of Project Health Colorado with five key outcomes and a variety of strategies. Each grantee developed their more detailed TOCs with a chain of interim outcomes leading up to The Trust’s outcomes.
Although the Trust has a long history of using evaluation for learning, this approach to real-time, data-centered strategic learning was new to the organization. Strategic learning coaches helped implement the learning process with grantees and the Trust. Data was generated from a variety of sources and included (but not limited to):
- Each grantees’ data collection strategies, such as meeting observations, commitment to action cards, meeting reflections, surveys of participants, etc.
- Twice-yearly learning reports from each grantee describing their strategies, how they learned about their related outcomes, and how they adapted in response;
- Surveys from grantee staff who participated in quarterly convenings;
- Tracking information from the paid media and mobilization campaign;
- Evaluation findings from an interim and final report;
- Polling data and message-testing focus groups; and
- Data collected in response to emergent needs, such as a choice modeling survey to help with message testing.
Frequent strategy management meetings with learning coaches, technical assistance providers and communications consultants provided opportunities to interpret and use the information collected. Twice-yearly formal debriefs brought learning coaches together with Trust staff and leadership to interpret data collected during the preceding six months. The strategy management meetings and debriefs led to decisions on how to adapt the strategy, as well as questions to bring to grantees at convenings.
The quarterly grantee convenings allowed both for the strategic gathering of learning information and an opportunity to put those lessons to use. The grantees discussed the questions identified during The Trust’s debrief and provided recommendations, resulting in further adaptations to the strategy.
In addition, ongoing feedback loops were implemented, facilitated by the learning coaches. Relevant learning from The Trust’s process was brought to the grantees for exploration through targeted outreach or during regular meetings with the grantee organizations. Similarly, learnings of value to the Trust generated by grantees were brought back to the Trust. Importantly, these feedback loops helped facilitate timely use of the information, supporting the real-time learning component of the process.
As a result of the evaluation’s efforts, individual grantees shifted their strategies in real time, increasing their ability to influence their audiences’ awareness of access-to-health issues, understanding of their impact on communities, and engagement in actions to help build will. The Trust itself shifted parts of their strategy, such as allowing for more personalized coaching for grantees and training for volunteers, changes to grantee convenings (including increasing the focus on best practices from the grantees themselves) and shifts in messaging. The strategic learning model built the capacity of both the Trust and the grantees to use information effectively in real time to shift their strategies as needed, allowing for increased effectiveness of the strategy as a whole.
To learn more about the strategy and the strategic learning model, consult these resources:
- If You Build It, They Will Come: Creating the Space and Support for Real-Time Strategic Learning (log-in required) from the Foundation Review looks at real-time strategic learning and how organizations and evaluators can foster learning to improve effectiveness.
- Beyond the Talking Points: Engaging the Public in Conversations on Health Reform explores the experiences of Project Health Colorado messengers, highlighting the interpersonal skills needed to engage the public in difficult discussions.
- Storytelling: A Tool for Health Advocacy focuses on the internal elements used to persuade different audiences and identifies key guidelines to help advocates choose which stories to share.
- Building Public Will: One Faith Leader at a Time explores the experience of one organization, Together Colorado, and their efforts to engage and support faith leaders in advocating for access to health care, including how they worked with leaders, tips for building public will and mobilizing new advocates around a particular issue.
- Social Media Strategies: Starting Conversations about Health Care Reform provides tips and guidance learned from Project Health Colorado and introduces five steps to develop a social media strategy that can engage audiences.
Colorado HealthStory (HealthStory) was a grantee of The Colorado Trust’s Project Health Colorado public will building strategy. HealthStory’s particular strategy was implemented through a partnership of three organizations: Colorado Rural Health Center, Colorado Coalition for the Medically Underserved, and ClinicNET, and had two main components:
- Collecting individual health stories, with a focus on empowering individuals to share their stories, helping them become more aware of access to health issues, and
- Sharing stories and local community health profiles through a variety of mechanisms, including community forums, partner organizations, toolkits others could use to share the stories, websites, and newsletters, all of which were intended to share information about access to health, leading to increased understanding and empathy, as well as causing some individuals to act on a personal conviction.
The HealthStory Theory of Change process focused on articulating the outcomes of the strategy and tying them to the broader will building framework, balancing the need for people to leave with specific steps to take with the desire to not have the forums become health resource fairs.
HealthStory and its learning coach used a combination of intense period debriefs shortly after piloting specific strategies – for example, the first three community forums – as well as strategic learning debriefs every six months to comprehensively explore the strategy’s implementation and impact.
At the beginning of the strategic learning process, many of the questions in the storyteller survey and meeting “reflection” forms helped determine whether the process was working and whether the content was high quality and meaningful to the audience. To evaluate these questions, HealthStory instituted a variety of data-collection strategies at the first three community forums, intended to provide in-depth information to improve the strategy, including:
- A feedback survey from storytellers, which included demographics to track diversity and types of stories,
- A meeting “reflection” for participants to complete at the end of the forums,
- Structured observation logs for staff to use during the forums, and
- Website and social media analytics.
Early learning quickly revealed that the story-collection process was positive for participants and resulted in rich stories. Staff worked together to improve and make their story-collection process more systematic as they learned what worked best. Early learning also identified that the sharing of health stories at the forums followed by small-group dialogues was powerful and meaningful to forum participants.
The learning after the first three forums identified a need to shift how people were recruited into the strategy, switching from broad, open invitations (including using press releases) to leveraging organizational partnerships and using targeted and personal ways to engage both new storytellers and forum audiences. With time, this further adapted to a model that used existing community meetings for sharing HealthStory content, rather than investing significant resources in hosting forums and recruiting audiences.
The learning also revealed a need to revise some of the data collection strategies and parts of the overall strategy. These changes included a shift from using “reflection” for participants to using calls with key community leaders to understand the impact of the forum on community dialogues. Further, feedback on the implementation of the story collection strategy was replaced with follow-up phone calls to a sample of storytellers to learn how they took action or talked with others after telling their story.
In addition to the important strategy changes made by HealthStory based on their strategic learning efforts, over time, HealthStory staff largely took over their strategic learning, facilitating their own debriefs and beginning to develop and implement learning tools with minimal requests for support or feedback from the coaches. The process of working with coaches built HealthStory’s capacity not only to collect data to inform their strategies, but also to plan how and where to collect data and how to use that data.
The Colorado Daylight Partnership (Daylight) is a collaborative effort led by the Mental Health Center of Denver and the Colorado Commission for the Deaf and Hard of Hearing. Its purpose is to provide assistance to publicly funded behavioral healthcare providers in Colorado who want to improve access to behavioral health services to deaf and hard of hearing Coloradans.
Daylight has four primary components, all centered around the value of consumer choice for communication access:
- A Learning Collaborative with behavioral healthcare providers across Colorado to leverage their expertise in working with deaf and hard of hearing people and create opportunities for cross-training and resource sharing;
- The development of statewide standards for the provision of behavioral healthcare to deaf and hard of hearing persons and support for behavioral health providers to implement those standards;
- The promotion of service delivery and telebehavioral health strategies that minimize cost while improving outcomes; and
- Promoting consumer choice around communication preference by distributing hearing assistive technology and equipment and ADA kits to Learning Collaborative agencies, and advocating for and supporting behavioral health agencies to hire ASL proficient providers and/or qualified sign language interpreters.
Daylight’s process started with the development of an action plan in 2008 for increasing access to high quality mental health and substance abuse services for deaf and hard of hearing individuals.
When implementation of the action plan began in 2009, Spark Policy was invited to be part of the Core Team and facilitate the Implementation Team (a larger group of key providers and consumer advocates who guided the work). One of Spark’s roles was to help identify strategic moments when existing data could be leveraged or new data collected to help inform critical decisions. In the first couple months, the need to understand the current state of practice among providers became clear as workgroups on standards and consumer engagement were formed. Spark released an assessment survey to providers throughout the state. The aggregate results were used for decision-making by both the Core and Implementation Teams.
Another critical strategic learning moment occurred when it was time to invite provider organizations throughout the state to join the learning collaborative. Initially the Daylight participants planned to open an invitation to everyone, but they realized that it was critical to have highly engaged participants who were ready to take action to create a dynamic environment that inspired action among all organizations.
To identify which partners would bring this energy and commitment to the table, Spark created one-page write-ups based on the readiness data for each organization and develop a set of interview questions. One deaf and one hearing interviewer talked with each provider organization, capturing their insights and responses to the questions. Additionally, data was gathered on the number of deaf/hard of hearing clients served by each organization. Altogether, this helped in developing a cohort of “early-adopter” organizations that were serving deaf/hard of hearing consumers, had areas for growth and other areas of readiness, and were committed to the work.
This approach of identifying a critical decision, leveraging existing data, gathering new data, and using it, along with data from the program evaluation, to inform the decision continued throughout the first few years of the Colorado Daylight Project, helping the project to be strategic in its ongoing decisions and driving it toward success.
From the standards, the Colorado Daylight Project moved on to establishing the Learning Collaborative and supporting the implementation of the standards. Eight provider organizations from around Colorado agreed to serve as pilot sites for the implementation of the standards, with support from Daylight through membership in the Learning Collaborative.