social_impact_GPS_satellites

(Illustration by Yarek Waszul) 

Measuring social impact is a quixotic pursuit. We both should know. Over the last several years, each of us has worked at leading social investment funds—Root Capital and Acumen Fund—measuring, managing, and communicating the social impact of our respective organizations’ investment portfolios. We have done a reasonable job counting outputs and aligning our performance with our own and other impact funds through the adoption of the Impact Reporting and Investment Standards (IRIS). But the social entrepreneur or investor who seeks to measure the real social impact of his investments needs information about how people’s lives changed as a result of a project or product, against the nonexistent counterfactual of what would have happened without it.

These changes are often unobservable, as in the case of improvements in a program recipient’s well-being. When the changes are observable, they may not be quantifiable. Even when they are quantifiable, they are denominated in different cultural and economic “currencies.” How can one compare a long-lasting bed net that protects a Kenyan child against malaria to a loan to a small-scale farmer to buy fertilizer that improves crop yield and increases income?

Yet we know social impact when we see it, and when program recipients or consumers experience it. A to Z Textiles, one of the investees of Acumen Fund, manufactures insecticide-treated bed nets that the Tanzanian Ministry of Health distributes to rural areas. We have seen the malaria infection rate decrease there, but we don’t know exactly how many cases of malaria were prevented, and we never will. Do we need to know precisely? Perhaps not. There is a danger in our desire for precision. It distracts us from attainable and accurate estimates of social impact, when in most situations, accuracy (“the truth”) is more important than precision (calculating estimates to two decimal points).

The distinction matters because the problem of social impact measurement is unsolvable precisely, but quite solvable imprecisely and accurately. Many impact assessment methodologies seek to create a single estimate that is both precise and accurate. We have found that most of these methodologies are too expensive or complex to scale across a large number of projects, while lighter touch methodologies are often imprecise, inaccurate, or both. This leads to a sense among practitioners of being stuck. No matter how we try to measure social impact, either the data are unavailable or we cannot accept the results as fact, knowing how much fudge went into the calculation.

Estimating Impact

Even as experts continue to refine existing impact methodologies, we as practitioners must work to increase the rigor with which we combine information from multiple methodologies to tell a compelling and accurate story. By combining the information contained in several imperfect but practical estimators, we can triangulate a more accurate estimate of the impact of our work. We liken this process of triangulation to a global positioning system (GPS), which combines the signals from multiple satellites to triangulate a fairly precise estimate of one’s position on the face of the Earth.

The analogy is simple. In theory, scientists could design a GPS in which three satellites could determine the location of a GPS receiver anywhere in the world, estimating latitude, longitude, and elevation simultaneously. In reality, commercial GPS units use signals from up to 10 satellites to correct for measurement error in calculations based on the first three signals. By combining information from multiple sources, global positioning systems can overcome their own limitations to triangulate highly accurate estimates of a position.

Acumen Fund and Root Capital are two leading impact investing funds that aspire to generate exceptional social impact and a financial return by investing in enterprises that benefit the poor in developing countries. Acumen Fund provides long-term debt and equity capital to early-stage ventures to commercialize socially valuable innovations, and Root Capital provides short-term and long-term credit and financial management training to rural enterprises, such as farmer cooperatives that connect small-scale farmers to markets.

Our portfolio of hundreds of enterprises, operating in various sectors across Africa, Latin America, and Asia, represents a heterogeneity of social impacts that presents a particular challenge for measurement. In our work, we have found no single methodology that is both precise and accurate and that allows us to assess all of these diverse types of impact simultaneously. Like most practitioners, we use various impact assessment tools and methodologies to understand various dimensions of our impact.

Each of these tools and methodologies can be thought of as an “impact satellite.” The results from each methodology may be individually imperfect, but we can combine, cross-check, and synthesize them to improve their collective accuracy. To direct capital toward its greatest impact, we not only must increase the quality and quantity of information we gather about impact; we must also employ a GPS-like approach to increase the rigor of the triangulation process that we employ, to piece together dissimilar impact information from disparate sources into an integrated whole.

A GPS measures three dimensions of location (latitude, longitude, and elevation). We think about impact information as having three primary dimensions as well: type of impact—the nature of the impact(s) on each person or organization; scale of impact—the number of people or organizations affected; and depth of impact—the amount or intensity of change experienced, per type of impact, per person affected. Type of impact may be articulated as outputs or ideally as outcomes, and depth of impact is the change in subjectively experienced well-being (or in economists’ terms, utility) associated with those outcomes. In theory, then, the social impact of a project or investment is the sum of changes in well-being (for example, depth), for all types of impact, for all people affected (for example, scale).

In addition to basic output metrics, we also include enterprise- or project-reported information, site visits by our staff, case studies and other reports by third parties, qualitative and quantitative surveys (including randomized controlled trials), data gathered using new approaches to mobile technology, and literature reviews. We fold in cost data to evaluate cost-effectiveness, and when possible, we gather data to provide a counterfactual to establish causality and attribution. Our impact teams aggregate and analyze the data for internal review and strategy-setting meetings, and for external reporting. Acumen Fund uses the Pulse system it developed, whereas Root Capital has a system integrated with its credit application process. We have adopted IRIS, managed by the Global Impact Investing Network, to allow for comparability of outputs across funds.

Finally, we combine the information from our impact satellites to create a complete and accurate picture of impact. Many existing approaches focus on reducing all types and depths of impact to a common numerical index. In contrast, we seek to integrate and triangulate across different types of information, in order to rank or categorize interventions by level of impact. Our emphasis at this stage is on maximizing accuracy, not precision. Once we are confident in the accuracy of our assessments, we can shift our focus to increasing precision, and perhaps comparability.

Specifically, we take as inputs our estimates of the scale, types, depths, cost-effectiveness, and causality of each project’s impact. By comparing projects, we begin to establish standards for what constitutes unacceptable, acceptable, and outstanding levels of impact, first for each dimension, and then across dimensions. Consistent application to a portfolio of projects over time facilitates the accumulation of learning and allows patterns to emerge, as the impact of each project is illuminated with context generated by the rest.

For example, Root Capital’s loan officers use our Social and Environmental Scorecards to evaluate prospective clients along the three dimensions of impact. Scale is measured by the number of small-scale farmers reached. For each of four types of impact (incomes, treatment of workers, community, and environment), loan officers indicate the strength of clients’ practices—that is, the likely depth of their impact. We refine the client selection criteria moving forward. We supplement these ratings with deeper studies at selected representative clients. Our impact team then synthesizes these ratings to categorize our portfolio of loans by level of impact and to refine the client selection criteria moving forward.

The central tension in implementation is to maintain an intellectually coherent approach—to keep our eye on the big picture even as we manage the details. A strong theory of change provides a center of gravity for the impact satellites. We find that each of the satellites exerts its own centrifugal force, constantly threatening to spin out of orbit—and becomes an exercise in data collection for its own sake rather than one that supports an organization’s mission and strategy.

The ultimate outputs of the GPS approach are, first, to organize our thinking and structure our internal conversations about impact; second, to sort projects ex post into categories of impact—such as failure, status quo, success, and game changer—with greater accuracy than previously; and third, to refine the hurdle rate for application in future deal selection. The GPS approach helps us to converge on a single, internally consistent, and well-articulated point of view on which projects have greatest impact, for use both in setting internal strategy and in external communication.

We are just beginning to learn the art of combining information from disparate methodologies into a coherent, internally consistent, and accurate categorization of investments by level of social impact. Although the GPS approach sets some broad parameters, it relies on the creativity and motivation of practitioners to get it right. We are each responsible for identifying which methodologies are feasible and appropriate, and then for navigating the hurdles that plague the endeavor of impact measurement. The GPS approach can help practitioners who seek to measure impact to the right order of magnitude, even if the numbers after the decimal place may not be precise. We aspire to triangulate our impact reliably and consistently; if we can, it will help us to direct our efforts to the projects that offer the greatest promise for change.

Tracker Pixel for Entry