Everyone Talks About Systems Change—But How Do We Measure It?

(Illustration by Melinda Beck) 

The work of government systems change is complex. It requires understanding historical systems built on an unjust foundation that have been repeatedly broken and put back together again through reforms to policies, practices, and funding. Our systems are also shaped by shifting cultural norms and how people think about the role of government, both inside and outside its institutions.

How do we know if our systems change work makes a difference? At Third Sector, we are committed to empowering government to deliver responsive services and achieve better outcomes for everyone, regardless of race, background, or circumstance. We do this by providing technical assistance on issues ranging from diversion and reentry to behavioral health, early childhood, postsecondary opportunities, and workforce development, together with our government partners at the state and county levels.

We work with government agencies to improve outcomes for their communities via six systems levers: data, funding, policy, services, external relationships, and internal culture. In our projects, we reimagine how an agency and its partners engage with these levers to change how systems operate.

We partnered with the Connecticut Office of Early Childhood, for example, to build and launch a parent and family cabinet that advises the department on critical decisions. We have worked with nine counties across California to remake their Full Service Partnership behavioral health programs to provide vital wraparound services for people most at risk of homelessness, hospitalization, and recidivism. In Oregon, we reduced rearrest rates by supporting collaborative partnerships between a county housing agency, the parole and probation department, and local service providers to increase access to permanent supportive housing and comprehensive services.

A common thread across our projects is an unwavering commitment to improving the lives of all people and reducing disparities. But we don’t always know to what extent our efforts have the impact we desire.

In 2024, we experimented with our first systems change evaluation. Commissioning a series of case studies, we sought to understand how our technical assistance to government partners contributes to systems change and improves community outcomes. We partnered with Cause IMPACTS, a woman-owned social impact consulting agency, to lead the evaluation as an external neutral partner and work with us to translate insights into guidance for our organization and the field.

Unlike programmatic or process evaluation, systems change evaluation is an emerging practice (although we did discover some shining examples of new approaches, including at Rotary Charities). We had to create our own approach to assess our impact and understand whether the projects we contributed to delivered not only the technical changes we promised in our contracts but also deeper, sustained changes within systems and communities.

To accomplish this task, we rooted our evaluation approach in FSG’s “Water of Systems Change” framework by John Kania (who serves on Third Sector’s board), Mark Kramer, and Peter Senge to uncover the contributions of our technical assistance to systems change and to be able to describe systems-level impacts.

We translated this framework into the following guiding questions to organize the case studies:

• Structural: How are current resources and practices different from before?
• Relational: How do norms, relationships, and power dynamics differ from before?
• Transformative: How are key actors in the system thinking and acting in different ways than before?
• Community: How has the project improved outcomes for people accessing services? Have we improved the lives of people facing the greatest barriers to resources and opportunity?

In addition to this framework, we partnered with the equitable evaluation advisor and LEEAD scholar David Hanson to embed culturally responsive and equitable evaluation practices in our approach, from the questions we ask to how we engage stakeholders and talk about evaluation. Expanding the Bench, an initiative dedicated to expanding diversity in research and evaluation, defines such practices this way:

Culturally responsive and equitable evaluation (CREE) requires the integration of diversity, inclusion, and equity in all phases of evaluation. CREE incorporates cultural, structural, and contextual factors (e.g., historical, social, economic, racial, ethnic, gender) using a participatory process that shifts power to individuals most impacted. CREE is not just one method of evaluation; it is an approach that should be infused into all evaluation methodologies. CREE advances equity by informing strategy, program improvement, decision-making, policy formation, and change.

Our approach to evaluation and case studies embodies these core values of equity, inclusion, accountability, and shared learning. Case studies focus on authentic stakeholder engagement and emphasize lived experiences and storytelling over quantitative data and metrics alone. Evaluation uses stakeholder accounts to assess and understand systems change impacts. It also prioritizes feedback from program participants to help us understand community-level impacts. Capturing stories of change from stakeholders involved in or affected by each project represents the crux of our approach to systems change evaluation.

It is important to note that as one actor in complex systems, Third Sector cannot claim attribution for systems- or community-level impacts. We did not attempt to prove causation or attribution. Instead, our case studies focused on exploring how projects supported by Third Sector contributed to systems and community change.

We found that focusing on how Third Sector, alongside other stakeholders, contributed to the project’s outcomes worked well. The evaluators determined that when engaging various stakeholders, it was impossible to separate Third Sector’s impact from the project’s impact. They were one and the same to the participants.

With these frameworks in place, Cause IMPACTS set out to uncover stories of change inspired by four Third Sector projects. Cause IMPACTS was able to identify shifts at the structural, relational, transformative, and community levels and highlight lessons that guide Third Sector’s current and future strategies.

If you want to try out our approach, we have four recommendations for measuring systems change:

1. Apply a systems lens view. Use the Water of Systems Change framework to explore the project’s story, including what happened at the structural, relational, and transformative levels, and to shape your research questions and analysis. You can also use the framework as a probe to go deeper in conversations with stakeholders. Look for shifts across levels to understand how systems changed over the course of the project and how change was sustained once it ended.
2. Embed equity. Use a culturally responsive and equitable evaluation approach. Do this by recognizing the value of lived experiences and storytelling as a powerful form of data. Include the voices and expertise of system stakeholders and community members throughout the research process, not just during data collection. Partner with them to define research questions, make sense of data, and confirm insights. Remember to always compensate participants for their time and contributions. We used MIT’s Living Wage Calculator to estimate how much to pay participants, seeking to pay at or above the living wage. For each hour, we provided gift cards valued at $20-$25. We also recognized participants by listing their names in the case study reports. Consult Expanding the Bench for resources on CREE.
3. Focus on your contribution. With systems change, it is nearly impossible to claim attribution for projects outside a lab. Instead, tell the story of what happened, the players involved, and what changed over time because of your organization’s role. Listen closely to stakeholders to understand what changes emerged and how your work may have contributed to those shifts.
4. Begin in the world of your stakeholder. Remember that the person you speak with does not have the same view of the project as you do. Understand that person’s entry point and ask questions from there. If stakeholders do not know your organization and they know the project by a particular name or through a process with a caseworker, be sure you’re using scaffolding that makes sense in their world. Don’t ask, “What was your experience with Third Sector or the Full Service Partnership?” Instead ask, “Have you noticed changes in your experience with your casework or at the clinic? What have those changes been like?”

This year, Third Sector is conducting five case studies and will continue to build on the findings. As we experiment with measuring systems change and gain insight into how to transform systems to make them more equitable and outcomes-focused, we must share best practices and frameworks. Systems change is long-arc work. We can accelerate the pace of change by sharing and learning together as a field.

Read more stories by Annie Neimand & Anum Ali Mohammed.