Predicting a ‘Kilo of Impact’

(Illustration by iStock/sorbetto)

Impact investments surpassed $502 billion in 2018 while philanthropic giving is conservatively estimated at $485 billion, adding up to about a trillion dollars in impact. But the dollars are easier to budget than the impact. Post-facto studies and longitudinal data collection methods are too costly and belated to provide investors with data they need to compare investments and allocate funds to maximize social returns. Upfront analysis is necessary not just to evaluate whether something is a good investment, but—often more critically—whether it’s a better investment than something else. In the absence of timely assessments, investors often rely on the “bad science” of simply counting outputs (the number of lives impacted or solar panels installed) and assuming a correlation between outputs and impact. 

Enter the burgeoning field of predictive impact analysis. At Thoughts in Gear, we began exploring the possibilities when a private investor challenged us to measure what impact installing a solar-powered refrigerator for vaccines in a health center would have on the surrounding community. We discovered that it could be done: The social value of the vaccines that passed through each unit could be determined by using current public health research to calculate the potential illness and mortality—and the associated costs—and focusing on a specific geography where we had direct health center data. Through a series of iterations, we created a model that associated each action with an indicator and assigned a value (cost in $ PPP) to that indicator based on global health research. With the summary of those costs, we quantified most of the social costs averted by the expected vaccine doses stored in each cold chain unit.  

This discovery has fundamentally changed how we think about impact measurement. Instead of relying on measurement to prove the value of a past investment, determining the value of an intervention before it starts allows investors to consider social value as part of the total cost of ownership (TCO), making it possible for decision-makers to factor in social impact alongside calculations of financial value. What we hadn’t expected was that this decision-making benefit would also help Ministries of Health to advocate for their public health systems. Capturing the social value helped make a clear case for investment in immunization programs and infrastructure: When pitching both internally or to international funders, governments could show the future cost savings that dwarfed the cost of immunization interventions. Recognizing how powerful this was, we have gone on to predict impact in sectors beyond health, including how English-language education can increase lifetime earnings for underserved students and how replacing kerosene with solar energy sources can reduce public costs.

A New Approach

"If you can look into the seeds of time, and say which grain will grow and which will not, speak then unto me. " - William Shakespeare’s Macbeth

Predictive analytics are not new but it’s only in the past two years that it’s become possible to predict impact. Newly developed approaches like the impact multiplier of money (IMM) and the Anticipated Impact Measurement and Monitoring (AIMM) system draw on previous methodologies, such as social return on investment (SROI). But while SROI provided a basic structure for quantifying social returns, these new predictive methodologies go a step farther by linking targets to expected social returns through validated research. Put simply, instead of presuming the correlation between outcomes and long-term impact, predictive impact analysis uses validated research to quantify the value of an intervention in concise, monetary terms.

The objective of the entire exercise is a monetary figure that can compare apples and oranges, what we jokingly call a “kilo of impact.” In order to achieve that, a calculation model uses data from research to allocate value to each expected outcome. Drawing data points from the research, each indicator is entered into a cost-benefit model, which calculates the total social cost or benefit. The quantified result, i.e. a monetary sum, is what enables us to compare one investment to another, across sectors.

Why haven’t we been doing this all along? Perhaps because the approach requires some expertise in each sector and data analytics to form the model or because it requires upfront investment in measurement from the investor. We are also creatures of habit, and the post-facto evaluation approach has been the field’s standard for decades. But there has never been a better time to change that. Our access to global research and data is exponentially expanding. Analytic tools, including AI, are continuing to accelerate our data processing capacity. In parallel, impact investing’s global growth is the perfect opportunity for impact measurement innovation.  

How Do You Do It?

It’s easy to get bogged down in the jargon, but predictive impact analytics can be boiled down into a standardized process:

1. Define impact trajectory: At the most basic level, defining the intervention’s intended impact and theory of change is a process every investor should go through with their prospective investments (though advanced practitioners may adopt scenario planning methods), producing a documented pathway to the expected impact.

2. Make a model: This step can be less or more complex. After mapping a set of measurable indicators to the intended impact and theory of change, build formulas into the model to link one indicator to the next, factoring in assumptions and adjusting for time, like using the SROI approach to forecast rather than evaluate after the fact.

Ideally, the model should be a replicable tool that can be used for multiple calculations. For example, Y Analytics uses a formula to calculate each investment’s IMM, while the IFC’s AIMM has a project rating metric; at Thoughts In Gear, our quantification of the potential social value of a single product—a cold chain unit for storing vaccines—allowed for replications in multiple countries by adjusting the country-specific indicators.

3. Link to research: Since most interventions are not entirely novel in their approach or context, there’s no need to re-invent the wheel. By entering proxy values into the model, based on previous data sets, an organisation can hypothesize their intended impact, using external research to validate whether or not there is a correlation between their intervention and that impact and to what degree. It can get a little messy, since the proxies must fit reasonable bounds of similar geographies and populations, not to mention being drawn from a substantial data set. (When outcomes can’t be accurately linked to a proxy impact value, the investor and/or organization need to encourage research institutions to fill in the research gaps.)

4. Quantify the results. By pulling values from validated research, one calculates how much can be ascribed to an intervention’s expected outcomes for a particular population and aggregates those distinct benefits or costs. For public health and environmental interventions, for example, this would be the summary of costs/losses avoided if the target number was reached; for education, this would be the summary of benefits linked vis-à-vis research after the average change in knowledge and behavior change was measured. Whenever possible, multiple research sources should be used to create ranges and derive averages from diverse findings, and in each case, using more conservative estimates to avoid over-valuing any single intervention.  

Not every element of impact can be captured numerically, and unquantified impact should be highlighted alongside the summary figure. However, since quantifying impact allows the resulting figure to be weighed alongside expected revenue when a company makes investment decisions, as much should be quantified as possible.

5. Monitor the path: After the investment has been made, investors should track progress along the intended trajectory to confirm the predictions were accurate. Nimble monitoring presents the opportunity to revise targets and expectations based on the inevitable shifts. In social impact measurement, we need to incorporate as much flexibility as possible, so the predictions need to be verified along the way. Technology can play a critical role here, whether it’s a mobile app to easily enter grassroots data points or the aggregation of multi-party data in a cloud-based portal. But the organisation itself only needs to monitor the scale at which they are doing what they set out to do. If teaching English is the vehicle for improving livelihoods, for example, an organization only needs to measure whether they are effectively teaching English, to how many students, and over what period of time.

6. Adjust accordingly: While impact measurement is often taken as a linear process, with a definitive end point, it should always be a cycle of feedback, learning, and revision. Before reporting the resulting impact, investors need to adjust the indicators in the predictive model with the monitoring data they’ve received from implementing organizations, ensuring that the actual path and the intended path are in alignment. This includes adjusting the original models based on new data and research as it becomes available. Immunization was a great case study to show how the continual advancement of research provided more specific values and a wider array of data to input into the existing model. When a new vaccine was trialed, we could factor that into the calculations. The model itself allowed us to easily input the new figures. As this becomes more sophisticated, we could use technology to automate what is now a manual process of identifying the research, pulling and inserting values.  

Predicting the Future of Predictive Impact Analytics

When we become comfortable with predicting impact, we will start to think about impact measurement differently. As it becomes a part of how impact investors and philanthropists think, the focus can be to:

1. Shift the burden. Rather than making implementers responsible for costly, academic evaluations, investors should provide the necessary funding to measure impact. By giving organizations the tools to monitor their impact trajectory, they can be supported in collecting the right kind of data, but leave more robust research to external experts.

2. Embed it. Investors can build predictive models into their entire investment process, making it the basis for both social returns due diligence and impact reporting. Developing technological tools to manage custom predictive models and collect monitoring data will streamline the process.

3. Cultivate better data. Research institutions need to work with the social sector to identify gaps of knowledge where a deeper evidence base is required. This research can be used to link the outcomes of interventions with their impact, and thereby, quantify the impact. Philanthropists and global institutions can play a critical role in supporting this research and create databases of evidence that can be linked to predictive impact models.  

Silver Linings, Not Silver Bullet

Predictive impact analysis is not the answer to everything. It will not ensure that every investment achieve its objectives. It does, however, provide a tool to more effectively allocate resources, and, if fully utilized, could save a huge amount of funding (and thus increase impact). And though this methodology depends on a level of analytical rigor and continual monitoring that may not be accessible to all investors and social impact organizations at this time, the shift in mindset is one that most organizations can adopt, towards both quantified impact and forward-looking measurement.

When this shift in mindset happens, the benefits will be felt at scale. Predictive impact analysis creates an opportunity to share the burden of impact measurement. Rather than making measurement the responsibility of resource-strapped organizations, investors would take on the analysis as part of their due diligence process, leaving social impact organizations to focus on monitoring, which simultaneously enhances their operations, and allowing research organizations to fill gaps in intervention-specific evaluations. Each actor is internally incentivized to incorporate measurement in this way, thereby increasing the likelihood of data utilization by all parties.

The initial applications of predictive impact analysis appear promising, but it will take more time and case studies to prove the true value of the approach. Social impact is inherently complex, and while this is a tool we can leverage, it is a piece of a large system of design, implementation, and measurement that mirrors that complexity.

We’ve been searching for a silver bullet to measure our kilo of impact. What if what we’ve needed was a crystal ball?

Read more stories by Meeghan N. Zahorsky.