In recent years, the idea of “collective impact” has become a staple of conversation within the social sector. Collective impact initiatives have been relatively successful at the “collective” part—at getting stakeholders around a table to build trust and to identify common goals. But many initiatives have struggled when it comes to the “impact” part. How can the leaders of a collective impact effort determine whether they are truly helping to solve the large-scale problem that they have targeted?
The answer to that question lies partly in another concept that has garnered a lot of attention in the nonprofit sector: “big data.” Too often, organizations in that sector have been unable to collect and analyze data at a large enough scale to assess the true impact of their work. Indeed, some of us who work in the sector jokingly refer to our tendency to rely on “small data.” With regard to collective impact initiatives, this challenge is especially acute. For that reason, those who lead ambitious, multi-partner initiatives are beginning to invest in big data—data sets that are granular enough to facilitate program decisions, broad enough to address complex issues, and comprehensive enough to enable an accurate estimate of social impact.
To explore how leaders can incorporate the use of big data into a collective project, I want to discuss the School Zone (TSZ), a cradle-to-career initiative that provides support to families that are trying to exit poverty. My firm, Measuring Success, is working closely with TSZ leaders on datarelated aspects of the initiative.
Launched in 2011, TSZ serves children from birth to the age of 18 who live in West Dallas, Texas, an urban community that has a history of low academic achievement rates. The initiative brings together 16 public, charter, and parochial schools, along with 29 nonprofit organizations that offer resources such as in-school and out-of-school programming, substance abuse intervention, housing and food assistance, and medical care. Leading the initiative is the Budd Center at the Annette Caldwell Simmons School of Education and Human Development at Southern Methodist University (SMU). Additional partners include the Dallas Independent School District (Dallas ISD) and the Center on Research and Evaluation (CORE) at SMU.
TSZ began by posing questions such as these: Which after-school programs provide the most successful interventions for which kinds of students? How do music lessons or athletic activities affect students’ math scores? Do students with behavioral challenges respond better to a mentorship program or to a peer-to-peer learning program? To ensure that data collection processes and systems were in place to answer those questions, TSZ turned to CORE, and CORE then partnered with Measuring Success. Together, we are creating a prototype system that will link otherwise discrete bodies of student data from participating schools and partner organizations.
It’s hard to overstate the value of incorporating data from many different sources into one system. Ultimately, the TSZ system will encompass details on students’ academic achievement and on their participation in interventions conducted by multiple providers. By collecting and aggregating such data, TSZ leaders will be able to track the students’ academic and socio-emotional development longitudinally against various indicators of community poverty. Doing so will also allow for effective case management across the TSZ ecosystem.
The work of applying a big data approach to TSZ is at an early stage, but already we have made notable progress. We have completed research and data-sharing agreements that enable access to a broad array of student data. In a related effort, we have formed a “data cohort” that brings together 11 of the 29 nonprofit partners, and that group has developed a common language for data sharing. Drawing on newly available data, some partner organizations have begun to make shifts in their program offerings—by moving, for example, to provide free tutoring for struggling students. Best of all, the academic achievement curve for students in West Dallas is beginning to bend upward. In 2014, for example, the math readiness scores of TSZ-affiliated students in grades 3 through 12 actually doubled.
Many factors have contributed to our initial successes. But certainly one critical factor has been the deep commitment by all TSZ partners to developing a capacity for effective data use. Here are several lessons that we have learned as we have sought to build that capacity.
Treat data use as a journey, not as a destination | Data sharing across an entire ecosystem requires trust between participating organizations. The Collective Impact Forum estimates that it takes on average three years for participants in a new collective impact initiative to gain enough trust to begin collaborating effectively. At the outset, many of the partner organizations in TSZ had not previously worked together. So the Budd Center worked with staff members from those organizations to help them understand the fundamentals of data-driven decision-making. The center worked with the Dallas ISD and SMU to put in place a legal infrastructure to support data sharing, and it also installed data collection software. In meetings with partners, meanwhile, we have helped them to adopt a “data vocabulary” and to cultivate a data-friendly mindset (“We are an organization that uses data”). The process has involved a great deal of trial and error. But through persistent, long-term effort, TSZ has begun to create a data-sharing infrastructure and a datasharing culture.
Handle privacy issues forthrightly and constructively | Only if both the beneficiaries and the partners in a collective impact initiative are willing to share information can such a project develop a robust data system. In the case of TSZ, we had to persuade parents to release data about their children, and we did so by explaining how the initiative would enhance their child’s educational and socio-emotional well-being. The Budd Center sent request letters to more than 8,000 West Dallas parents, and roughly 2,000 of them granted TSZ permission to access their children’s data. That step provided an important early breakthrough for data collection.
Encouraging TSZ partner organizations to share proprietary information is equally challenging—and equally crucial. To motivate them to sign data-sharing agreements with TSZ, we had to show how they would benefit from this arrangement. Funder incentives tend to push organizations to argue that they offer the only or the best solution to a given problem—not just one piece of a larger puzzle. As a result, they treat data about program participants like a rare asset that some other organization might steal away. To counter that tendency, we show organizations how they can use shared data to make their grant applications more compelling to funders: An organization that offers an after-school program, for example, could use TSZ’s data system to call up a report that shows an increase in school attendance from 75 percent to 90 percent among students in the program.
Allow for database autonomy | People often assume that all participants in a collective impact effort must use the same database system. Our experience contradicts that received wisdom. Initially, the Budd Center asked TSZ partners to adopt a universal database so that data aggregation would be easier. But it quickly became clear how costly and how labor-intensive that approach would be. Most TSZ partner organizations use their databases for purposes that go beyond what’s relevant to the data-sharing needs of the initiative. So TSZ has developed data-sharing practices that apply only to data that relate to its collective impact goals. Filtering out nonessential information has proved to be an important aspect of the new data-sharing infrastructure.
Measuring Success is building a system that will allow each school system and each nonprofit partner to export its data to a central repository without having to give up its incumbent data system. Likewise, TSZ leaders will receive the information they need without imposing unnecessary work and undue costs on partner organizations.
Build your data system iteratively | Any collective impact initiative will face a big temptation: Should it seek funding to acquire a “gold standard” data system? For TSZ, that would not have been an effective strategy. Existing off-the-shelf systems were too complex for the evolving needs of the initiative. A custom system would have cost millions of dollars to automate fully, and TSZ leaders would not have been able to adapt it to changing needs. (Software code, once it has been written, can be difficult to modify.) So we built a “minimally viable product” (MVP)—a relatively inexpensive prototype of a full-fledged data system—that proved to be a far better fit for the initiative. The MVP approach has allowed TSZ to build a data system iteratively, in response to feedback. By creating a scale model first, moreover, TSZ has enabled participants and funders to experience what the system can do for them. Now those stakeholders are more likely to develop a commitment to the system.
Linking Islands of Data
Our next step will be to turn the data system prototype into a platform that allows for both automated data uploading and direct data entry. As a result, TSZ participants will have access to a central data architecture from which they can pull richly textured information—including longitudinal data from multiple schools and nonprofits—on each TSZ-affiliated child. Eventually, we intend to add a predictive analytics engine that will enable TSZ evaluators to assess which interventions are most likely to work for a given child.
The ability to link otherwise disconnected islands of data helps Netflix to offer personalized movie choices. It helps Amazon to recommend books for customers to buy. It helped the Obama for President campaign to micro-target voters in 2008 and 2012. And, we are discovering, it can help social sector organizations to develop carefully designed solutions for underserved families and children. By incorporating big data as a strategic tool, collective impact initiatives can achieve real, measurable impact.