There’s been a lot of interest recently in collective impact, for good reason; it’s an exciting approach, and it’s demonstrating a growing track record of positive results. But while collective impact partners are increasingly interested in sharing data to identify best practices, learn from mistakes, and implement changes to improve outcomes, they often don’t know how to use shared data programs effectively, or even safely.
To learn more about data sharing in collaborative efforts, we at the Network for Nonprofit and Social Impact at Northwestern University interviewed 20 individuals representing partners in the Chicago Benchmarking Collaborative (CBC), a seven-organization community of practice that uses a shared database to track outcomes for its early childhood and adult learning programs.
The CBC has its roots in Christopher House, a nonprofit dedicated to supporting families through innovative schooling and education programs. When Christopher House first applied to the Chicago Community Trust in 2008 for funding to improve its evaluation practices, its request was met with another: Can you collaborate with other organizations on this project? The Chicago Community Trust was interested in Christopher House, but felt sure that the organization could have a great impact if it could cross-fertilize ideas and approaches, and share results with other nonprofits seeking the same big-picture outcomes, learning, and improving over time.
Christopher House agreed and found not one, but several interested agencies—Chicago Commons, Erie Neighborhood House, and the Chinese American Service League—and the organizations received funding for shared evaluation software. Over time, other nonprofits and agencies—including Chicago Youth Centers, Children’s Home + Aid, and Gads Hill Center—joined the group, and the collaborative became known as the CBC.
The group faced a steep learning curve when it came to identifying best practices for data sharing, from implementing software to interpreting the data itself. Today, however, the collaborative is an exemplar in the art and science of data sharing. The partners’ generous input during our interviews suggests three lessons for other collective impact efforts aiming to improve their data-sharing processes and results:
1. Protecting client identity is essential—and possible!
Nonprofits understandably have a fear of sharing client data, but the technology does exist to make such data sharing safe, and it’s constantly improving. So the solution lies in combining the right technologies with purposeful efforts to establish and sustain a culture of trust.
Together, members of the CBC serve approximately 12,000 clients and collect sensitive client data, including student grades. Privacy concerns are paramount, but the CBC found a software package that met its particular criteria for client security in Social Solutions’ Efforts to Outcomes (ETO). Although many platforms for secure data-sharing exist, the CBC chose this software, because it was customizable and would accommodate the range of programming offered by CBC partners. ETO allows individual organizations to track their own clients while also providing a shared reporting tool that allows all partners to see anonymized data.
When CBC partners began using the tool, they also created a memorandum of understanding (MOU) articulating a policy that specifies which data users can share and who has access. The MOU also states unequivocally that users can use such data only for the purposes of program improvement. To confirm their commitment to the MOU, all partners signed it.
2. Organizational use leads to collaborative use.
CBC’s members essentially fall into two categories: active and passive software users. From the earliest days of implementation, active users have taken it upon themselves to become familiar with the software and create their own reports. These organizations are quick to contact other CBC members, or even the software’s makers, if they encounter problems, and ultimately, they find ETO useful for reporting their own organizational outcomes in addition to the comparative results.
Passive users, on the other hand, initially used the software only to enter data as required by their membership in the CBC and relied on the CBC manager to prepare summary reports. These organizations didn’t invest in learning how to use the software, and in fact, they were often more critical of the system than their more-active peers, because they did not fully understand the software’s uses. One passive user told us that since his organization rarely uses the software beyond what is required, he never knows what to make of the reports that result from the data itself. Over time, it has proven difficult to transition from being a passive user to an active one.
The findings here suggest that the simple presence of technology is not enough to encourage people to use it. Leaders must see an investment in shared technology as a commitment to both their collaborating partners and their respective organizations. Organizational leaders can demonstrate this commitment by identifying staff to manage the software and investing in training for those employees; likewise, funders can provide support not just for shared software, but also for its implementation and evaluation.
3. Data requires interpretation.
CBC members emphasized that the presence of a shared data system—even if it’s actively used—isn’t enough to improve programs and realize system-level benefits. Partners have to take deliberate action to translate data into lessons and then into action. To that end, CBC members meet formally three or four times a year to make sense of the data and to generate meaningful comparisons. By discussing the data, partners can also identify problems that could be keeping them from making the progress they seek. (For example, in one of their meetings, CBC members found that they had been using different criteria for program success, which resulted in reporting differences.) Through discussion, they can also recognize potential, as partners explore how other nonprofits might implement successful programs and home in on best practices.
It can be difficult to find common ground across organizations that serve different populations and have slightly different missions. The software produces part of the story, but as one member told us, there is “a narrative that goes beyond the data” that partners can only understand through conversation.
For nonprofit organizations, simply collecting data can be difficult. There’s no minimizing the effort involved in that first step. Even so, data collection represents only the first step. The work of the CBC demonstrates that to engage in meaningful benchmarking, partners must have a clear sense of the data they collect and what that data represents, an understanding of how their partners measure their work, and a commitment to shared evaluation and interpretation.
The rise of collective impact and other joint efforts suggest that nonprofit leaders must rely on new technologies that enable them to share sensitive data efficiently and securely. However, as demonstrated by the CBC, it takes time to build confidence in the software, in the data, and in the partnership itself. The effectiveness of collaborative efforts now depends in part on a collective dedication to getting the most out of shared technology.