Using Data for Action and for Impact

Everyone in the social sector, it seems, is discussing how to use data to deliver social good. The coming years will no doubt see the continued growth in data-driven nonprofits and social businesses, operating in an environment that increasingly seeks and even demands evidence of impact.

Unlike the for-profit sector, where the metrics are straightforward and focused on profit, the social sector reports to multiple masters—the community being served, staff, donors, and policymakers. Nonprofits and social businesses must be able not just to show what they do, but also to demonstrate the impact of what they do in terms of multiple bottom lines including financial, social, and environmental. Because of the complexity of doing this, most social sector staff and managers struggle with data: getting it, using it correctly, and maintaining privacy.

Despite the challenges for the social sector in managing data effectively, the data imperative is here to stay. The dropping cost of technology makes collecting data far more affordable and easier than in the past. Data will make the work of social change agents more effective and will build the case for support for the best programs and enterprises.

The goal of this article is to demystify the data explosion and to provide practical advice to nonprofit and social business leaders for harnessing these trends in two important ways. As a sector, we need to use data to respond to today’s needs, to manage our teams better, and to improve our efficiency. This is what I call “data for action.” At the same time, and I hope with much of the same data, the social sector needs to use data to establish that our interventions lead to lasting change. This is the Holy Grail for donors, policymakers, and social sector leaders. This is what I call “data for impact.”

The true power of data will come when we learn how to link these two approaches seamlessly. That’s because data for action and data for impact are two sides of the same coin. If we can collect the data we need for making our programs better, we will have taken a critical first step toward delivering evidence of our ultimate social goal, lasting impact.

Becoming a data-driven organization, however, is not simply a technical challenge. It requires some wrenching adjustments to organizational culture, both for nonprofits and social businesses and for donors, government agencies, and impact investors. And the objective of better outcomes through data requires confronting failure in its most positive aspect: learning what does and does not work.

Three Essential Questions

At the most basic level, organizations collect data to answer questions. The three core questions that most social change organizations need to answer are, How much did we spend? How much did we do? and How much did it matter? If we had the right data, we would be able to answer these questions. As we answer them, we would gain insights that would enable us to ask better, more detailed questions about our work. And ultimately, that would lead to our organizations having greater social impact.

Most people working at nonprofit organizations and foundations will recognize that these three questions align closely with standard program evaluation design. Donors, especially institutional ones, want to see a logic model that explains how a program’s activities drive lasting change. How does a dollar spent lead to lower infant mortality, less pollution, or fewer poor people?

The standard evaluation logic model typically has five elements: inputs, activities, outputs, outcomes, and impacts. These elements link closely to our three core questions. The first question, How much did we spend? is part of the input piece of the logic model. Money is one of the central inputs for a project; it funds the staff. The second question, How much did we do? concerns activities and outputs. And the third question, How much did it matter? explores outcomes and impacts.

Making a difference is the ultimate objective of everyone in the social sector. The third question is also the most difficult to answer, so we have to work up to it.

The Data Supply Chain

One way to understand and manage social sector data is to think of it as a data supply chain. At the front end is the individual whom our organization serves. This person is part of a community that is likely to be suffering from a range of disadvantages, including economic deprivation, conflict, health impairment or disability, environmental impacts, discrimination, and lack of educational opportunity. Whether the person is a beneficiary, client, user, or employee, the ultimate goal of social impact work is to change his or her life for the better.

In the data supply chain, data flows from basic program activities steadily toward greater and greater levels of aggregation. The frontline staffer of a nonprofit, for example, typically delivers services to many individuals in the community being served. Program or branch managers oversee multiple staff members. The head of an organization oversees multiple managers. Program officers and individual donors generally support multiple organizations. Foundations and donor collectives track multiple program areas. National policymakers want to see progress at the national level. And international policymakers want data that transcends national borders.

 

All of these stakeholders are seeking data, but there is a wide gulf between the day-to-day experience of a person living in poverty and the work of an international policymaker at an organization like the United Nations. The time frame and number scales are vastly different. The person in poverty is living in the now, and the availability of a crucial intervention may literally be a matter of life or death. International policymakers generally deal in trends on an annual or long-term basis, such as principal indicators for the Sustainable Development Goals, where the data of millions of people is aggregated into a single number.

The data being sought by audiences further along in the data supply chain is typically different from that collected through program activities. Many of our programmatic interventions are intended to have positive long-term social impacts that our programs don’t actually measure. For example, secondary and tertiary education are expected to make graduates more employable, but attendance records and report cards do not measure employability.

The NGO Riders for Health is a good example of the challenge of determining what is measurable at the program level compared to the social impact goal. The idea behind Riders for Health is simple: Government health ministries in many developing countries operate outreach programs to deliver health-care services such as vaccines, prenatal screenings, and HIV prevention education to rural populations. Health ministries need motorcycles and other vehicles to reach these people. Riders for Health operates a preventive maintenance program in sub-Saharan Africa designed to ensure that more of these vehicles are available for the outreach programs. Their theory of change is that if more of these vehicles are available, more health-care services will be delivered and better health outcomes will result for more people. Basically, more motorcycles today equals lives saved tomorrow.

To monitor their work, Riders for Health collects the type of action-oriented data you would expect: numbers of vehicles, distances driven, and fleet availability (the percentage of vehicles available to be used by health-care workers). Riders for Health has data showing that if fleet availability increases, medical personnel use the vehicles more frequently, reaching more people and delivering more services. It’s assumed that this increase will save lives over time, because the services being delivered, such as vaccinations, have been shown in the past to be effective. Ultimately, the health ministry is looking for decreased mortality and illness, data that is different from the data collected directly by the vehicle maintenance staff of Riders for Health. And yet that life-saving impact is the entire reason Riders for Health exists.

Program staffs will keep focusing on using data to make programs better today, and donors will keep looking for long-term social impact. But we do need to keep the needs of both groups in mind as we design data systems for social good programs. We will have to bridge these differing timescales, using data to meet the needs of disparate stakeholders in the data supply chain. Now let’s look at how the data supply chain relates to the three essential questions.

How Much Did We Spend?

First, a bit of good news. We already have an example of a data system that works reasonably well across the social sector. It’s the financial data supply chain. And it answers the first of the three questions: How much did we spend?

At each step in the data supply chain there are strong incentives for nonprofits and social businesses to collect accurate financial information. Whether it’s a desire to get paid, to know how actual figures compare to budget, to report to investors, or simply to comply with financial accounting standards, most organizations are reasonably good at tracking financial data.

Financial data has multiple positive attributes. The units are the same, or can be easily converted if different currencies are in play. How you calculate the data is also set. Financial accounting standards are well established. Finding skilled talent who can calculate the numbers is easy because accounting is similar across organizations. Tools to work with financial data are readily available and work quite well in both the for-profit and nonprofit fields. Interchange and aggregation of financial data are well understood. We even have privacy standards that apply to personal financial data, such as salaries. These attributes mean that we can answer the How much did we spend? question reliably and cost-effectively.

Coming back to our example of Riders for Health, not only does Riders have a good handle on how much it costs to fund its operation in a given country, but the organization also understands how much it costs to purchase, fuel, maintain, and insure a given vehicle. In Zambia, Riders for Health made a bold commitment to purchase vehicles and supply all aspects of vehicle operation for a single price. For example, the organization charged $0.30 per kilometer travelled by a health ministry motorcycle Riders supplied and managed. Making that kind of price commitment underscores the importance of having a detailed understanding of the cost of delivering the service, especially when making a large financial commitment to purchase and operate a fleet of vehicles.

This is not to say that all organizations are perfect at financial accounting. Grassroots NGOs have been known to skimp on financial data tracking. But these kinds of gaps are relatively easy to diagnose and, with modest resources, remedy. So knowing how much nonprofits and social businesses spend is straightforward.

How Much Did We Do?

Answering How much did we do? is more complex. That’s because the delivery of programs and services is where social change takes place, and where our beneficiaries, customers, and users become part of the data supply chain. By and large, creating positive social change requires active responses to acute needs. Whether it’s responding to a natural disaster with food and water or alleviating poverty with microloans and livelihood training, much of the day-to-day work of social change revolves around transactions. Some activities respond to immediate demands: A domestic violence survivor might need immediate access to a safe house for herself and her children, or an accident victim might need emergency medical care. Other activities build over time, such as education or infrastructure improvements for clean water. The common link between these activities is action. This is what social sector workers live for. They are focused on making a difference.

The first step in answering How much did we do? is collecting data about the problems that the organization is trying to solve. Information about needs and activities is the core of data for action. It directly drives activities to respond to those problems.

Data about these actions is quite important to program management. Measuring program activities is relatively straightforward. Knowing that a program trained 1,500 people last year using a three-hour course on HIV transmission prevention is a useful statistic for the program manager, especially if the prior year’s number was 500. This data is also crucial for continual program improvement and management. This is the other aspect of data for action: using the data to manage people and improve programs.

In the Riders for Health example, there is a simple way to measure activity: the distance each vehicle is driven. This information is important not only for maintenance, but potentially for health program management. If a given outreach office uses its vehicles far less than another office with a comparable community to serve, that data alerts health ministry leaders to other logistical, medical, or management problems that should be investigated.

The data supply chain starts to break down, however, when we start aggregating activities across different organizations. For example, larger foundations, impact investors, and government agencies frequently fund many different organizations all working on the same social issue or need. Just like the organizations they fund, these funders also need to provide information on their activities to their backers. Yet the organizations they fund rarely deliver identical services. For example, it’s hard to know how to compare data about 15-minute training sessions by one organization to three-hour sessions by another. Does it make sense for a donor to sum up the total number of students served by all of its education grantees when the level of engagement with those students might vary from a onetime visit to a website to six hours a week for an entire school year?

Furthermore, it’s difficult to know from activity data alone whether the actions were actually effective in reaching the ultimate goal. Are HIV infections reduced in the people we trained compared to people who didn’t receive the training? Are the 15-minute training sessions similar in effectiveness to the three-hour sessions? Or as those in program evaluation might say, outputs are not the same as outcomes.

This question of assessing effectiveness leads us to the last of the three questions.

How Much Did It Matter?

How to measure the ultimate impact of a social program is the most challenging question. Everybody in the social sector wants to know whether the work he conducts or funds is making a difference. The core rationale for funding social action, whether philanthropic or impact investment, is that it makes the world a better place.

The data supply chain—which works reasonably well in measuring spending, and passably when measuring activity—is actually broken when it comes to measuring overall impact. Many nonprofit program teams simply assume that their work has lasting impact, typically without conducting additional studies to find out if that is actually the case. Many nonprofits are founded to meet an acute need in society: They see a hole that needs filling and fill it. They heal the sick, feed the hungry, house the homeless, and educate the uneducated. They experience the rewards of seeing transformation on the ground. They know they are making a difference. Unfortunately, that might not be the case. Many randomized control trials have been conducted on social programs that found no effect. That is, something that the program staff was sure was making a difference, wasn’t.

There are multiple reasons why it is difficult to measure how much the work matters—the question of ultimate impact. Here are five of the most important:

• The first challenge is timescale. When we ask about the ultimate impact of an intervention, we have just leapt past the timeframe of most programs. It is hard for a program such as job training, which touches an individual for an hour, a day, a month, or even a year, to know whether that person was measurably better off five or ten years later. This is especially true when funding is committed in annual amounts.
• The second challenge is the difficulty of identifying and measuring direct, causal impact. The gold standard for measuring impact is a randomized controlled trial. These are required to approve a new drug, for example, but can cost tens or hundreds of millions of dollars for a single drug. Measuring the effectiveness of drugs is relatively straightforward if the goal is curing a disease. It gets more difficult when you attempt to measure the trade-off between increasing the number of years a patient will live against the quality of life the patient will have over the time period.
• The third challenge is privacy. Nonprofits often collect information about society’s most vulnerable people, usually about what makes them vulnerable. It is important to safeguard information about an individual’s status as a refugee, a survivor of rape or domestic violence, or a person who is gay or lesbian. Nevertheless, pursuing the social good of privacy can also make measuring the impact a program has on people over time more difficult.
• The fourth challenge is resistance. Staff members might feel that data collection or analysis activities are a diversion from their primary role of delivering products or services. There may also be uncertainty about how the impact data will be used, and what the consequences will be if an activity is shown or perceived to have minimal impact.
• The fifth challenge encompasses all of the others: lack of money. Measuring impact over longer timescales, developing randomized controlled trials, collecting data, and protecting privacy all cost money. But relatively few donors, investors, or government agencies want to fund this work. Although the rhetoric in the social sector demands greater evidence of impact, in practice the typical funder prefers to buy more activities.

This is not to say that the typical funder does not care about longterm impact. Donors are simply choosing to adopt activity measures as a cost-effective proxy for impact. In many cases, this strategy is entirely workable. If some other research has been conducted on the effectiveness of a vaccine in preventing a disease, it is reasonable to measure the cost and scale of a vaccine-delivery program and have confidence about the impact that it has on the long-term health of children. But as the cost of measuring impact goes down and the importance of demonstrating impact goes up, more donors will need to step forward to invest in serious impact research.

In spite of all of the challenges in measuring impact, the question How much did it matter? is not going away. In fact, it is more important than ever as social entrepreneurs, governments, investors, and donors alike aim higher and expect more from their collective action.

The Time for Data is Now

Two important societal trends are furthering the data revolution. The first is that smartphones already are, or shortly will be, in the pockets of just about every person working for social impact organizations or living in the community being served. Their ubiquity makes collecting data much easier, more reliable, and less expensive than ever before. By all but eliminating the traditional, less effective ways of collecting evaluation data at the end of a multi-year project, or tacking data collection on top of existing programs, smartphones and other digital devices make the lives of frontline staff easier and their work more effective.

The ability to collect data digitally, instead of on paper, is just one benefit made possible by digital devices like the smartphone. Another will be that smartphones provide data trails created by each individual user of a mobile phone. It also is increasingly easy to add data collection devices to essential infrastructure, like a wirelessly connected flow meter on a borehole well. Just as Google knows where all the traffic jams are, we will increasingly know more and more about programs and the people and communities they serve.

For example, when the Riders for Health intervention was tested with a major randomized control trial funded by the Bill & Melinda Gates Foundation, program evaluators attached GPS units to a subset of the motorcycles to validate the trip reports submitted by health service staff.¹ An unintended consequence of collecting these data trails was to identify roads and trails that staff members were using which weren’t yet on maps, and to create improved maps. It also allowed staff to spot clinics where relatively little outreach was occurring.²

By getting better answers to questions such as How much did we do? social sector organizations can start asking better questions. Imagine the power of finding out that one of your five core program activities is effectively useless. That would be tremendously exciting to most mission-driven leaders, but scary to the people who staff that activity.

The second important societal trend that is driving the data revolution is the emergence of “big data.” Big data is one of the hottest topics in the technology community today. Unfortunately, most social enterprises are a long way from collecting big data today.

Definitions of big data vary, but most agree that it involves using sets of data that are so large that one must use complex techniques to process the data and gain insights from it. A phone book, for example, is a traditional data set: a collection of the names, addresses, and phone numbers of people living in a town. An example of big data would be to collect all of the GPS locations of all of the mobile phones in that same town every second of every day for a month. That quantity of data would break a spreadsheet!

Big data works best when the data collected actually covers a large fraction of the phenomena being observed. For example, Amazon has a pretty good idea of what books Americans are reading, because a large proportion of the books purchased or read by Americans is sold by Amazon. Having such a large trove of data enables Amazon to make powerful inferences that help its business model. A great example is recommendation engines: Amazon can predict your interest in buying a specific book that is often purchased by customers with book purchase patterns similar to yours.

For-profit businesses are in the forefront of collecting and making effective use of big data. For the most part, however, the nonprofit sector does not have this kind of information about its customers or clients. If you are collecting data only on paper, you’re never going to have big data. If you assess impact on only a small sample of your community at the end of a multi-year project, you’re a long way from big data. As mobile devices play a bigger and bigger role in life generally and social action in particular, the opportunity to collect massive amounts of data cost-effectively will improve.

How could we collectively solve social problems if we had access to all the relevant information? I was once asked what big data could do for a particular foundation’s biggest challenge: the 5,000 students who dropped out of high school in the city where the foundation operates. My answer was that with a modest amount of funding and access to existing data, the foundation should be able to answer with confidence what 90 percent of those dropouts were doing. With comprehensive data from social media and mobile phones, we could probably tell which former students were dealing drugs, playing video games, working, being home schooled, and so on.

Unfortunately, it is difficult to get our hands on this kind of data. For the most part, the mobile phone and social media companies won’t share it with us. Some pioneering social innovators are now constructing programs from scratch that are designed to collect big data. For example, Crisis Text Line was founded when the operators of DoSomething.org, a teen volunteerism program, received a disturbing number of texts from teens in emotional crisis. This observation led them to create a separate organization to focus on this different and urgent social need. Millions of text messages later, Crisis Text Line now has a huge amount of data that can be analyzed, leading to rapid program improvements, better training of the crisis intervention volunteers, and insights into the mental health issues of American youth.

Massive data collection gives us the tools to better understand the social problems our organizations address. With better understanding, we will be able to zero in on better questions to measure the effectiveness and long-term impact of our interventions.

Data is Disruptive

Although digital data offers the promise of significantly improving the social sector’s effectiveness, it will not happen without some disruption within the sector itself. All social sector organizations will have to change the way they think and operate in order to become data-driven. They will collect data they’ve never collected before. They will come up with privacy policies to control the use of sensitive data. New skillsets will become important for recruiting and career development. Funders, nonprofits, and social businesses will also need to take a longer-term perspective of their work. They must be more willing to accept risk and, frankly, failure.

Donors, impact investors, government agencies, and other funders are entirely justified in wanting to know that their resources are being used well and are making an impact. Many funders, however, are demanding evidence of impact without being willing to invest in the costly and time-consuming work of gathering that evidence. If the social sector is going to benefit from the digital data revolution, funders will have to pay for it. For this shift to become reality, investments in improving measurement need to be seen as essential program costs. And that will require a mindset shift.

Funders and nonprofits will also have to think within longer time frames. But it’s difficult for nonprofits to do that when many are funded with one-time, one-year grants. Shifting one’s focus to longterm impact requires major changes in grantmaking and investing.

Part of demanding more evidence of impact is engaging in the journey of understanding. If we are truly to shift from measuring How much did we do? to How much did it matter? we need to recognize that we will frequently learn that our traditional approaches are not particularly effective. Donors need to support innovators who are willing to shed ineffective activities in favor of more promising ones.

Social entrepreneurs also need to embrace technology and data as indispensable parts of our enterprises. Organizations need to invest in obtaining better data for action. This investment is a core program expense, not just a compliance cost. And making the investment will better prepare social sector leaders to talk impact and metrics with donors from a position of strength, buttressed by deep and real knowledge of what is going on in the field.

As a deeply optimistic technologist, I have to point out that computers are dumb as bricks. We still need humans to ask what the data means. We need people to tease out the difference between correlation and causation. Inquiring minds need to dig into the reasons behind a startling statistic, rather than accept it as representing reality.

We need to use data to make smart people even smarter. Humans are quite good at wrestling with questions and coming up with hypotheses. The purpose of the data tools is to make issues easier to understand and to make it possible to test hypotheses. It may be true that it costs twice as much to administer a vaccine per child in one province as it does in another province, but that fact alone is useless unless we understand far more about why.

The Path Forward

Broader access to digital data can fundamentally shift the way that organizations pursue social change. To achieve it, we need to invest in the capacity of enterprises for carrying out their day-to-day work, as a core part of their program or their business enterprise. This is collecting data for action, for responding to the urgent needs of the communities we collectively serve. If we can make data an essential tool for frontline workers, rather than a post hoc distraction, we will get more and better data. With that data, we will be able to ask better questions, increasing our efficiency in delivering our products and services.

By focusing our efforts on the front end of our data supply chains, we will be ready to provide the data for higher-level consumers of it. If organizations have better data about How much did we spend? in tandem with How much did we do? they can be held more accountable to the people and communities they serve, and to the funders and customers who want to buy more social good. With that data in hand, nonprofits and social businesses will be better prepared to answer the most important question, How much did it matter?

Data for action and data for impact. Our ability to combine these two aspects of data collection and analysis will fundamentally transform the practice of social innovation, ultimately delivering social impact far more effectively.

Read more stories by Jim Fruchterman.