(Illustration by Hugo Herrera)
One of the five conditions of collective impact, “shared measurement systems,” calls upon initiatives to identify and share key metrics of success that align partners toward a common vision. While the premise that data should guide shared decision-making is not unique to collective impact, its articulation 10 years ago as a necessary condition for collective impact catalyzed a focus on data use across the social sector. In the original article on collective impact in Stanford Social Innovation Review, the authors describe the benefits of using consistent metrics to identify patterns, make comparisons, promote learning, and hold actors accountable for success. While this vision for data collection remains relevant today, the field has developed a more nuanced understanding of how to make it a reality.
The data we collect—and the data we don’t—reflect our values and what we think is important. We track data on our finances. We monitor our blood pressure. And if you’re like me, you pay attention to your fantasy football team for five weeks and then forget about it. When we need to, we change our behavior—our spending, our exercise, our starting running backs. We make these adjustments because we care about the outcomes and have information that tells us we should do something differently.
At Equal Measure, we’ve worked with more than 100 place-based collective impact initiatives, and while their models have varied, data remains an area of focus and frustration. Collective impact initiatives need partners to come together to work toward a common goal, which means that folks with different perspectives and priorities must collaborate in new ways. Getting partners to agree on what matters—including definitions of success and how to get there—is therefore critical. The ability to use and act on data is a central competency of effective collective impact initiatives, yet many engaged in this work struggle to collect and use data to its full potential.
Here are four lessons from our work to help collective impact initiatives and their funders use data more effectively for social change.
1. Prioritize the Learning, Not the Data System
Those of us who are “data people” have espoused the benefits of shared data systems and common metrics too many times to recount. But a shared measurement system is only a means to an end, not an end in itself. Too often, new collective impact initiatives focus on creating the mythical, all-knowing data system—spending weeks, months, and even years researching or developing the perfect software that captures, aggregates, and computes data from multiple sectors. They let the perfect become the enemy of the good, as the pursuit of perfect data and technical precision inhibits meaningful action. And communities pay the price.
Using data to solve complex social problems requires more than a technical solution. Many communities in the US have more data than they know what to do with, yet they rarely spend time thinking about the data they actually need. Before building a data system, partners must focus on how they hope to use data in their work and identify the sources and types of data that can help them achieve their goals. Once those data are identified and collected, partners, residents, students, and others can work together to develop a shared understanding of what the data mean and move forward. In Connecticut, the Hartford Data Collaborative helps community agencies and leaders do just this. For example, it has matched programmatic data against Hartford Public Schools data and National Student Clearinghouse data to get a clear picture of postsecondary enrollment patterns across the community. The data also capture services provided to residents across multiple agencies and can be disaggregated by gender, race, and ethnicity to identify and address service gaps.
2. Be Clear About Whose Lives You Hope to Improve
Throughout the US, communities are designed to advance opportunities for some residents—likely white and affluent—while failing to serve others. In the oft-quoted words of Peter Drucker, “What gets measured, gets managed.” If data masks disparities by race, ethnicity, gender, or other identities in the communities we serve, we fail to address inequity and miss the opportunity to influence change that really matters. Tailoring data collection for context—and collecting, analyzing, and sharing data that allow us to understand communities—are essential actions to achieve equitable outcomes.
Collective impact initiatives must use data that highlight, not conceal, the experiences of the people they aim to support. Dr. john a. powell has helped communities think about equity through the approach of targeted universalism, which focuses on “setting universal goals pursued by targeted processes to achieve those goals.” Organizations, systems, and funders will—often inadvertently—work around the populations they should prioritize. Communities must use data to acknowledge where disparities exist and then address the structural inequities experienced by these groups.
This process requires measures that provide actionable insight into residents’ lived experiences. However, many data systems are designed to aggregate, not differentiate, experiences. For example, during Equal Measure’s recent work with Aspen Institute’s Opportunity Youth Forum to develop new measures for understanding trends among opportunity youth, the US Census’ use of PUMAS (Public Use Microdata Areas – geographic units of 100,000 or more residents that are used to report data collected through the American Community Survey) highlighted this importance. The geographic distribution of tribal communities across these PUMAs, along with their relatively small populations, means their data are consumed within these larger areas, making it nearly impossible to understand the unique experiences of youth living in tribal communities. Andrew Lewis, with the Hopi Foundation, described the severity of these measurement limitations, noting, “The way we count, in effect, impedes our ability to accurately count the native populations. And if you can’t be counted, in effect, you don’t count. … It results in a kind of systemic discrimination wherein the system itself makes it difficult for a people to achieve equity.” Collective impact initiatives must use, and advocate for, data that can be disaggregated to illuminate the experiences of population subgroups.
3. Use Qualitative Data
Many in the collective impact field—particularly funders—gravitate toward quantitative data. A review of requests for proposals from public and private funding sources in the social sector will reveal a desire to “demonstrate impact,” “build evidence,” and “scale results.” The field defaults to what it can quantify and compare, believing that metrics provide the only means to understand impact and progress. But folks living and working in communities where collective impact happens know the stories behind the numbers. Attendance data may show that children living in a particular zip code frequently miss school, but only the community members can tell us why. Collective impact is designed to bring these folks—parents, teachers, students, social workers, religious leaders, and police—together to understand what’s behind the metrics and to co-construct solutions. An exclusive focus on quantitative measurement is incomplete; collective impact must be guided by the data of human experience.
During a recent evaluation, one partner in a collective impact initiative in Maine recounted the state’s history of bringing adults and youth together to review and discuss data. The partner described how the Youth Leadership Advisory Team, a statewide effort that elevates the voice of young people in Maine who have experienced the foster care system, serves as a resource for agencies to call on youth to vet data and provide input on new projects, policies, and initiatives. In 2014, youth testimony led to a state law that extended financial assistance and other supports for youth formerly in foster care to complete postsecondary education. Collective impact work needs more of this.
4. Keep the Short and Long Games in View
Collective impact is, by definition, focused on changing systems and structures to improve community outcomes at scale. The approach moves beyond discrete programmatic interventions to “change the conditions that hold a problem in place.” Changing systems takes time—and seeing changed systems improve the lives of community members takes even longer. We need to capture achievement of big societal goals—kindergarten readiness, high school graduation, and employment—and we need to capture progress along the way. Collective impact initiatives, particularly their funders, tend to focus exclusively on measuring long-term population-level changes and miss opportunities to capture incremental change, make mid-course corrections, and continuously improve. Selecting a few key measures to monitor incremental progress can help keep initiatives on track while avoiding data overload.
Balancing attention to short- and long-term progress doesn’t just apply to individual outcomes, but also to systems outcomes. We engage in collective impact because systems need to change, and we need to monitor the status of those systems as well. While changes in these structures can be hard to measure, identifying key levers of change and understanding how systems shift is just as important as tracking outcomes among individuals.
The Aspen Institute Forum for Community Solutions’ annual grantee report models attention to individual and system-level changes in the short- and long-term well. The report asks collaboratives in the Opportunity Youth Forum network to share progress toward a number of systems changes (e.g., new policies and practices, funding changes, narrative shifts) annually, while simultaneously requesting youth-level education and workforce outcomes. This dual approach reinforces the importance of working toward sustained systems change that addresses root causes of inequity at scale and focusing on meaningful outcomes for the young people in their communities.
What Funders Can Do to Strengthen Data Use
Funders are well-positioned to support partnerships in their efforts to improve data use. Three opportunities exist to advance a more impactful approach to data use in collective impact initiatives:
- Invest in data infrastructure and capacity. Collective impact initiatives often lack the knowledge and technology to use data effectively. Making resources available to help initiatives understand their data landscapes and establish learning priorities can build a culture of learning and strategic decision-making. And providing partnerships with the technical support they need—software, hardware, training, and technical assistance—will help them translate data knowledge into action.
- Apply qualitative data to funding strategies. Quantitative metrics are critical for understanding progress at scale and over time, but community, practitioner, parent, student, and client voices are equally essential to setting the stage for new investments—and designing a rigorous evaluation process. Before launching a new funding initiative, create space for community input. And if grant reporting is necessary, ensure voices from the community are shaping an understanding of how work is carried out and its perceived effects on residents.
- Promote a focus on systems change and individual outcomes. Funder reporting demands leave many collective impact initiatives torn between demonstrating results for community residents or making progress on large-scale policy or systems changes. When communities are incentivized to focus on immediate outcomes, they often prioritize activities that result in quick wins at the expense of larger scale structural change. And when they focus solely on systems changes, they risk losing sight of the urgency in their own communities. Funders can provide the time, financial resources, and technical assistance needed to make sure collective impact initiatives drive their own agendas and don’t have to choose between the two.
A Vision for the Future
Data can play a transformative role in setting, refining, and evaluating collective impact strategies. Shared measurement—grounded in a culture of learning and focused on equitable processes and outcomes—may be our greatest hope for shared understanding of the societal challenges communities face. It’s time to align our data use with our values, and to make sure that collective impact initiatives have the knowledge, time, and resources necessary to use data to pay attention to what matters most.
Support SSIR’s coverage of cross-sector solutions to global challenges.
Help us further the reach of innovative ideas. Donate today.
Read more stories by Justin Piff.