Does Collective Impact Really Make an Impact?

Collective impact tends to generate strong reactions. Mention it, and the person across from you will likely either extol its virtues or roll their eyes. Introduced in a 2011 SSIR article, the approach to social change came onto the scene and quickly became a buzzword and philanthropy darling. For some, it offered structure and clarity about how to operate complex, multi-sector social change processes that can otherwise feel opaque and intangible. For others, not so much.

If you Google “collective impact critique,” you’ll find many articles describing collective impact’s limitations: It’s too top down; it’s not focused enough on community, social justice, and equity; and it’s too simplistic for entrenched social problems. Critics argue it doesn‘t adequately acknowledge the work that preceded and informed it; it wasn’t backed by adequate research and evaluation; and it branded and “consult-ified” work that communities, organizers, and other researchers had been engaging in for decades.

Over the last seven years, FSG, the Collective Impact Forum, and the Aspen Institute have tried to address some of these critiques. The Collective Impact Forum’s “Principles of Practice” resource guide, released in 2016, sought to bring in issues of equity, community engagement, shared leadership, data use, and local context. And, in early 2017, with the support of several prominent foundations, the forum partnered with our organizations—ORS Impact and Spark Policy Institute—to conduct a rigorous study of collective impact’s effect on institutions, such as school systems, human services organizations, and nonprofits, and target populations or environments. The study examined 25 initiatives in the United States and Canada. While the initiatives targeted a range of issues and had different geographic reaches, all of them had been in operation for more than three years and could present evidence of implementing the collective impact approach.

We entered into the study in 2017 with a healthy skepticism about the approach and a belief that a rigorous exploration would help the social sector better understand collective impact’s potential utility. The study results have helped deepen our thinking and altered our mindsets.

Here are eight important findings:

1. Collective impact undoubtedly contributed to changes in target populations or places. From the original set of 25 initiatives involved in the study, we selected the eight initiatives we felt could help us most deeply explore the question about the connection between collective impact and changes among populations and places. These initiatives demonstrated strong implementation of the collective impact approach, and had documentation of meaningful changes among the targeted population in things like improved river health, decreased rates of homelessness, and lower rates of teen births. In our study, target populations may be specific people within specific systems, geographic areas, or with special needs. We collected data from a range of stakeholders to understand their “contribution story,” or how they believed their efforts or other factors led to change. Analysts then used a methodology called process tracing to test the hypotheses about the connection between initiative activities and outcomes, assessing both the degree to which the data supported the proposed linkages and the degree to which the relationship uniquely explained the result. We also assessed the overall strength of each initiative’s complete “contribution story” for the ultimate question: to what degree did all the prioritized activities and outcomes add up to contribute to the ultimate change sought?

   How researchers used “process tracing” across initiatives in the study. Click image to enlarge. (Graphic courtesy of ORS Impact)

   As noted above, we determined that collective impact undoubtedly contributed to  changes at scale within initiatives’ targeted people or places. In many cases, stakeholders achieved these changes through the development or expansion of programs and services that reached targeted populations. Others achieved outcomes through policy changes at the state or organizational level. Many initiatives were able to collectively leverage resources to advance important components of the work. In these eight initiatives, we are confident that the initiative’s work to cultivate a shared vision, engage in mutually reinforcing activities, and support effective multi-sector partnerships—all tenets of the collective impact approach—played an important role in achieving at least one of their population-level goals. This was true across different issues areas—including juvenile justice, education, and health—and across statewide, county, and regional efforts.

2. Contribution to population-level change doesn’t always look the same. In some cases, the use of collective impact on its own could not explain the outcomes achieved. For example, one collective impact initiative supported a school district’s strategic plan. But while the initiative’s work likely contributed to academic gains, external factors such as federal funds to support school reform and the strength of the school reform model being implemented were equally important in explaining success. These results illustrate how collective impact can play a role in supporting and sometimes catalyzing other efforts versus driving the change directly.

3.  Quality of implementation matters. Collective impact is defined by a set of five conditions: backbone support, common agenda, mutually reinforcing activities, shared measurement system, and continuous communication. Several findings in the study suggest that more complete implementation of these conditions results in greater impact. Reflecting on their success, initiatives that achieved change in their target populations frequently prioritized two collective impact conditions: backbone support and a common agenda. Backbones often play an important role in convening and facilitating collaboration between partners. Likewise, sites described the importance of identifying a common agenda to move change forward. Those with the highest levels of implementation of common agendas were more likely to see policy changes and changes to existing practices among institutions, along with systems changes that required multiple organizations to coordinate shifts in practice. Shared measurement systems were also important, and although they were one of the least frequently implemented conditions, when they were present, stakeholders described them as an important driver of change. On the other hand, the five initiatives that had not achieved population-level change described weaker levels of implementation of the five conditions.

4. Implementation of an equity approach doesn’t happen by accident, and a strong equity focus can lead to more equitable outcomes.

   
   For the purposes of this research, we defined equity as:

   Fairness achieved through systematically assessing disparities in opportunities and outcomes caused by structures and systems, as well as addressing these disparities through meaningful inclusion and representation of affected communities, targeted actions, and changes in institutional structures, and systems to remove barriers and increase pathways to success.

   Using a rubric, we explored three facets of equity:

   • Capacity to engage in equity, including developing shared language, creating internal capacity, and building credibility with the community

   • Equity-focused actions, including using locally relevant and disaggregated data to identify strategies and targeting actions to the populations of greatest need

   • Representation and meaningful inclusion, including engaging those with lived experience and creating opportunities for the community to lead

   While most of the study initiatives saw the value of infusing equity into their work, only a few have been able to develop their capacity, target their actions, and authentically engage and shift power to communities. Equity takes intentionality, and most initiatives did not begin their work with an equity lens. Initiatives identified several challenges, including identifying ways to effectively engage communities, accessing the necessary data to conduct root cause analysis, and developing shared understanding and buy-in across partners for an equity approach. However, when initiatives did focus on equity, there was evidence of positive outcomes. Initiatives with stronger capacity to engage in equity and stronger implementation of equity-focused actions were more likely to help narrow gaps in outcomes compared to initiatives with more-limited implementation of these elements. For example, one initiative with a strong equity focus helped to narrow the gap in high school graduation rates between students of color and white students. However, a strong equity focus did not guarantee that a initiative would achieve its desired outcomes. A number of initiatives that began to focus on equity more recently had not yet seen the same kind of equity impact on systems or target populations.

5. Understanding relationships between the five conditions could support stronger implementation. The collective impact approach presents the five conditions as a set of discrete elements to address. Our study provides a more-nuanced way to conceptualize them. Results suggest that the backbone organization and common agenda are foundational and worth investing in early to create a strong base for the other conditions. When initiatives effectively implemented their backbone organization and common agenda conditions, they had stronger implementation of mutually reinforcing activities and were able to foster changes in institutions. A strong common agenda and set of mutually reinforcing activities helped shape the shared measurement system. Finally, rather than as a stand-alone condition, stakeholders often described continuous communication as a role of the backbone organization.

   Understanding the inter-relationships between the collective impact conditions based on study results. Click image to enlarge. (Graphic courtesy of ORS Impact)

    

6. There are many ways to engage in systems change. The study results showed that initiatives had a strong focus on changing the systems within which they operate and that many different types of systems changes can support population-level ones. The types of systems change more frequently identified as supporting population-change included those focused on providing new or expanded services, improving or aligning organizational practices (such as changing the way organizations operate), enhancing the capacity of the workforce, and changing organizational or legislative policies. We also found that systems change took a variety of forms, including change within a single organization, across similar types of organizations, and across diverse types of organizations. Change also varied in its level of formality. All of them added value to the work, but it was important for initiatives to match the scope and scale of the change to their aims. For example, informal change, such a piloting a new process or implementing a time-limited campaign, sometimes provided organizations with an opportunity to practice new ways of working, while formal change, such as changes to beverage policies in child care settings, allowed them to institutionalize and sustain change.

7. It takes time to create real change. While our study found that collective impact can make a strong contribution to population-level change in certain places and under certain conditions, it can take time. Among the eight study initiatives we examined most deeply, the time between inception and impact ranged from 4 to 24 years. The initiatives that did not see a demonstrated population change ranged in lifespan from 4 to 7 years, suggesting that collective impact is a long-term play, not a shortcut to social change.

8. We have a lot more to learn. While our results shed light on several important issues, more questions remain. To test the relationship between implementation and outcomes, our study examined initiatives that were effectively implementing the conditions and had achieved outcomes. It would be useful to learn more about situations when initiatives do not successfully implement collective impact, and when initiatives implement the conditions well but do not achieve change. It would also be beneficial to understand the potential value of collective impact in relation to other approaches for cross-sector collaboration, and to determine if and when collective impact is the best approach. With regard to the approach itself, there is more to unpack around the conditions—for example, the value of different types of backbones, the ways in which data can support learning, and when it is critical to involve different sectors. There is also more room to explore the principles of practice, especially equity.

Because of our experiences designing, implementing, and analyzing the results of our study, we have a more-nuanced sense of what it looks like to foster alignment between sectors, create a shared sense of accountability, and bake equity into the work. We also have a better sense of how the components of collective impact can add up to more than their individual parts. Like any framework or approach, it can create challenges; for example, participating organizations can take a completely hierarchical approach, create too much gatekeeping, or reinforce the status quo of institutionalized racism built into existing systems. Moreover, collective impact is not always the solution to entrenched social problems. This study does, however, suggest that it can indeed make an impact under some conditions. We hope it contributes to the ongoing interrogation of how to improve conditions for people and environments for whom society is not doing enough.

Read more stories by Sarah Stachowiak & Lauren Gase.