Creating a Data Culture

During a five-year research project I conducted about what makes organizations successful, I learned a shocking statistic: While 75 percent of nonprofits collect data, only 6 percent feel they are using it effectively. To me, this means that while there is a data feeding frenzy happening in the nonprofit sector, the vast majority of nonprofits have failed to develop a data culture—that is, a deep, organization-wide comfort level with using metrics to maximize social impact. 

 

Although many organizations don’t feel like their organizations are making good use of their data, creating a data culture is critical to their success. Actively and consistently using data to inform decisions allows nonprofits to track whether their programs are resulting in the outcomes they intend. In fact, in my survey of 250 social entrepreneurs, organizations that began measuring their impact from the start tended to have a faster path to scale. 

These organizations were scaling faster because they had the data to prove that what they were doing was working. The data wasn’t just about donors. These leaders wanted to know that what they were doing was actually having a positive impact on their beneficiaries. I’ll never forget Tess Reynolds from New Door Ventures telling me, “It is really hard to raise a million dollars. If I am going to work hard to get that money, I need to know beyond a shadow of a doubt that what we’re doing works.” 

But as indicated by the 6 percent number, it’s really hard for most nonprofits to make good use of their data, because most nonprofit leaders are not data scientists—they get into the work because they care about the cause. The good news is that you don’t have to be a data scientist to tell a good data story. All organizations have the capacity to create a data culture, no matter how big or small, or data savvy or not, they are. Organizations can improve their data cultures in at least four ways:

1. Be clear about outputs vs. outcomes. One of the most important lessons I learned in my research is that organizations must do a better job of distinguishing between outputs (how many people are participating in their programs) and outcomes (how their programs are actually changing lives for the better). When Rey Faustino started One Degree, a Yelp-like platform for social services in the San Francisco Bay Area, his organization was thrilled to be able to show that a year after launching, 40,000 people had visited the site. But quickly he realized that this was just a “vanity metric.” It wasn’t really a measure of how many people the organization was helping to get social services, let alone how their lives were improving because of access to those services. One Degree had to shift its measurement systems to move past tracking outputs (such as what types of services visitors searched for and whether they downloaded an application to receive government benefits) to tracking longer-term outcomes (such as whether they were actually accessing benefits, their experience with them, and how their lives improved as a result). Now it tracks the data even farther down the pipeline, and personalizes the user experience on the site to keep track of how their clients benefit from their services.

2. Get creative about metrics. If organizations want to uncover true indications of whether their programs are making a difference, they need to get creative about how they measure. Row New York is an organization that pairs rigorous athletic training with tutoring and other academic support to empower youth from under-resourced communities. To develop this model, it drew on extensive educational psychology research about how “grit” contributes substantially to success in school and life. Like so many others, when it started out, Row tracked things like number of participants, growth, and fitness levels. But success wasn’t just about kids showing up; Row needed to show that the program was influencing their rowers’ lives. Eventually, Founder Amanda Kraus came up with inventive measures of success. By tracking both attendance and daily weather conditions, the organization was able to show which students were still showing up to row even when it was 38 degrees and pouring rain. Those indicators of grit tracked with students who were demonstrating academic and life success, proving that their intervention was improving those students’ outcomes.

3. Measure in a mission-driven way. For some organizations, staying true to your mission may mean recognizing the limitations of our data-hungry nonprofit sector. Rob Gitin of At The Crossroads, an organization that serves homeless youth in San Francisco, knew he needed data to measure his organization’s progress, but because he was serving hard-to-reach youth (who sometimes require more 200 encounters with an outreach worker before they will engage with the organization), his data wasn’t going to look as impressive as organizations that were taking on the “easier” cases. He also didn’t want to fall prey to tracking someone else’s definition of success, such as helping a kid find housing, when what they really wanted was to get clean. But he still needed to figure out whether their interventions were working, and after many agonizing, late-night conversations with his team, At The Crossroads broke down its model into a clear set of achievement phases. This allowed the organization to work with each youth to set their own unique goals, while also tracking a consistent set of metrics. 

Photo courtesy of At The Crossroads

4. Be honest with data. If organizations are going to collect data, they also need to be ready to be honest about what that data is telling them. The organizations I interviewed who were building strong data cultures couldn’t afford multimillion-dollar randomized control trials, but that didn’t stop them from applying the strictest standards against their data to pressure test it. For example, Braven, an organization that helps low-income college students graduate and get jobs, paid an informal control group of students who were not in their program with Amazon gift cards to compare their performance with the ones in the program. In other words, Braven wasn’t just satisfied with improving students’ assessments; it wanted to test the counterfactual to ensure that its students were performing better than they would have had they not participated in the program. Pressure-testing your data to ensure that positive results are actually connected to your programming, versus other factors, is critical to being honest with your data.

Ultimately, for organizations to get the most out of their data, they need funders to support data-driven cultures through unrestricted grants that pay for impact measurement; capacity-building support to help develop better systems for tracking data; and the patient, long-term capital that organizations need to ensure that they can be honest with their data without worrying that their funding will get pulled if it doesn’t tell them what they want to see. 

Some of the most important practices of successful startups can’t be measured—things like believing in people and building trusting relationships. But figuring out what we can measure and measuring it effectively is essential to the success of organizations that want to achieve impact. 

Read more stories by Kathleen Kelly Janus.