Layering Evidence for Impact Investing Success

Impact investors must bridge the gap between evaluation theory and practice. (Photo by iStock/Radachynskyi

The field of impact investing turns 12 this year. As any parent knows, 12-year-olds can be precocious, and while they’re still young, they’re also maturing and ready to take on more responsibility and accountability. The same is true of the impact investing field. In particular, now is the time for more maturity and accountability in measuring social impact, and to achieve it, we must bridge the gap between evaluation theory and practice.

The COVID-19 pandemic and the resulting economic recession, and the need to support the growing racial justice movement make it all the more urgent that impact investing be an effective vehicle for social change. To that end, the field must keep high-quality evidence and measurement front of mind as the world endures and eventually recovers. In this article, we explore some of the factors contributing to the mismatch between traditional evaluation methodologies and impact investing, as well as the limitations of current practices. We also present two strategies for strategically using evidence throughout the investment lifecycle.

Between Theory and Practice

What counts as high-quality evidence? How does an investor predict impact before making an investment, and what should they use to measure impact post facto? These questions highlight the gaps between theory and practice when it comes to assessing evidence for impact measurement.

The most common measurement practices in impact investing rely on accounting principles, common sense, and easily quantifiable outputs. Many investors use information straight from enterprise balance sheets to roll up basic descriptive data about outputs. For example, two of the most common outputs reported include the estimated number of jobs created and/or the potential increase in revenue; both are easy to understand and relatively easy to collect. Investors also rely on gut-based assessments to drive decision making. If an investor is thinking about investing in a recycling company, they may assume that the company has an environmental impact without attempting to quantify it—because, of course, recycling is good.

Unfortunately, current practices focused on easily quantifiable outputs can sometimes mask the true story of impact (or lack thereof). If an impact investor is reporting only on jobs created, for example, we don’t know who is employed, what their socioeconomic background is, what the job means to them, or whether the job is better than their last. This practice feels a bit like the early comments about COVID-19 being a “great equalizer” that didn’t discriminate among different strata of people when in reality people have experienced the pandemic in dangerously different ways along lines of socioeconomic status, race, and type of employment.

In academia, randomized controlled trials (RCTs) have long been used by leading researchers and journals to assess evidence. This is because the randomization in RCTs is designed to eliminate the bias ingrained in other types of study designs. Some journals have even gone so far as to reject the term “predictive” to describe a variable unless the evidence is reinforced by an RCT.

Very few impact investors use RCTs to project their impact––and even fewer use them to measure impact after the fact. There are many reasons for this, including the fact that RCTs require significant resources, as well as careful control and manipulation of the study design. RCTs are designed to prove causality of the thing being studied in the trial’s specific context. In social impact and impact investing, so many factors bleed into an intervention. For example, in a curriculum-focused intervention in education, teacher training, leadership buy-in, policy shifts, awareness, and child health may all affect results. How do you isolate them? The more complex the investment, the harder it is to manipulate or control.

Common sense metrics and RCTs can be useful tools in the due diligence and measurement phases of an investment, but both have drawbacks that limit their use among investors.

Emerging Best Practices

Given the current measurement capacity of impact investors and the lack of alignment with traditional evaluation methodologies, what would it take to bridge the gaps between theory and practice? Investors with the most-advanced measurement schema have adopted frameworks that help them weigh different standards of evidence alongside opportunities to improve strategy and advance equity.

Investors can use patient capital to build impact evidence that better enables scaling of innovative products or social interventions. (Photo by Adobe Stock, image by Sorenson Impact Center)

Frameworks that consider equity are emerging as common practice, and the best of these use both qualitative and quantitative data. Measurement that strives to make equity central will always prioritize context, including the relationships, identity, and power dynamics surrounding the measurement. By their very design, RCTs remove or limit contextual factors from the evidence. Many underrepresented groups have decried quantitative data without context, insisting that to understand equity, we must incorporate qualitative methods.

As part of our work with investors large and small, we have been developing two promising frameworks to help advance measurement sophistication: horizontal layering and vertical layering. Each of these approaches to using and collecting evidence assumes the investor is making direct investments, as opposed to investing in impact funds. Both enable a better focus on context, and emphasize the relationship between investment strategy and measurement.

Vertical Layering: Building Evidence for a Singular Solution

Comparable evidence is not available for every type of investment, especially for new and innovative products or enterprises. It is rare, for instance, to have studies of two different programs that use comparable units of measurement, let alone similar study designs. When high-quality evidence about the social impact of an investment is lacking, investors can use their capital to support the creation of needed evidence by making repeated or long-term investments. Commitment to evidence-building through the investment of both capital and time requires that investors embrace an intentional approach and a patient strategy. We call this practice vertical layering, because it builds the evidence like rungs in a ladder, effectively elevating one solution above all others.

Within this framework, the investor limits their portfolio to a small number of products, services, or interventions, and commits to supporting them over a long period of time. This might look like a five-year commitment to investing in educational technology supporting English language learners. The investment helps facilitate measurement by providing the stability and patient capital to collect data, build capacity, scale a solution, and/or support an evaluation. After five years, the investor’s commitment could help demonstrate the most superior product, drive additional investment, and implement the technology in a greater variety of school populations.

This approach is common among philanthropic mission- and program-related investments, as opposed to exclusively return-seeking strategies. The Evidence-Based Policy team at the philanthropy Arnold Ventures, for example, funds RCTs related to criminal justice, education, health, and other policy areas to identify strong interventions, replicate the evidence, and then help scale the most successful interventions. Recently, Arnold Ventures invested $6 million to support implementation and evaluation of a transitional health care model at nine hospitals in order to replicate the evidence. The model, a promising nurse-led discharge program that helps complex care patients successfully transition out of the hospital, has prior evidence of cost savings, but the additional investment could help scale the solution by proving effectiveness across multiple populations and sharing the results.

Vertical layering also holds a lot of potential for impact investing measurement. Turner Impact Capital, for example, uses a focused, long-term investment and measurement strategy to help promote solutions to social problems related to housing affordability. Its approach involves seeking out opportunities to invest in markets typically held by government. Since 2015, the firm has purchased more than 9,000 multi-family, low-income housing units with a focus on preserving workforce housing. Its model includes services for families like homework help centers, on-site health screenings, and community watch programs. These services have led to lower rental unit turnover and maintenance fees that can negatively impact financial returns.

To successfully craft and scale this investment model for workforce housing development, Turner gathered evidence about social and financial outcomes of tenants over time to learn how social support strategies could impact financial return. Using this evidence to replicate the most-promising solutions, it scaled the strategy across multiple locations. Ongoing dedication to measuring both financial and social impact ties Turner’s investment to patient and long-term impact measurement that helps it refine the strategy that best serves families. The more high-quality evidence it builds, the more effectively it can scale their housing investments.

Horizontal layering builds evidence across multiple solutions in a system. (Image by Sorenson Impact Center)

Horizontal Layering: Building Evidence Across a System

Some impact investors prefer to keep a more-expansive portfolio to diversify investments, develop a deeper network, or make themselves nimbler. For these investors, a good alternative strategy is horizontal layering, which marries wider-ranging portfolios with different types of evidence collection to better support and understand system-wide impact.

A horizontal layering approach starts with a theory of change for a social system, such as education. This theory, or hypothesis about how to intervene and create more-favorable system outcomes, should identify potential intervention opportunities in the system that the investor can use to build their investment strategy—for example, improved teacher professional development or STEM curriculum. Pairing a measurement plan with each intervention opportunity is essential for this framework. Because the types of investments in the system are likely to vary, the measurement schema might also include a combination of balance sheet data, qualitative data, and even quantitative data from evaluation. Investments in STEM curriculum, for example, would be measured based student learning outcomes, whereas evaluating professional development for teachers might rely on more-qualitative data. Horizontal layering builds evidence across important levers in the system in an attempt to understand a causal chain that articulates how the system works.

As with vertical layering, philanthropy is already employing this approach. For example, the Howard G. Buffett Foundation’s investments in global food security are organized across a spectrum that considers short-term and long-term impact. On one end of the spectrum, it invests in direct food and water assistance, and on the other, agriculture and crop production research. It also invests in agriculture cooperatives and employment. By reviewing the evidence this range of investments produces, the foundation is better positioned to understand and change relationships in the complex food security system. This review led to a multi-partner initiative with a Ghanaian farmer and conservationist, Kofi Boa, to develop tools that can address multiple issues in the system. These include conservation-based planters, customized seeds, recommendations for cover crops, and infrastructure improvements.

By investing at different entry points in the system, impact investors can similarly analyze different types of evidence about solutions for social or market change. The accelerator Camelback Ventures uses horizontal layering to refine strategies that support women and entrepreneurs of color. Its fellowship program addresses disparities in capital distribution and investment in companies with diverse founders by focusing on multiple solutions and measuring their effectiveness along the way. For example, the program collects qualitative feedback about the impact of coaching on fellows’ ability to develop sustainable enterprises; tracks revenue growth over time to monitor the specific impact of its capital investment for the target population; and monitors and counts fellows’ connections to industry experts and potential funding partners. After six years, Camelback has used its insight about building connections and networks to develop three distinct tracks, with curriculum tailored for entrepreneurs in education, conscious tech, and local economies.

Balanced Investing and Transparent Reporting

Over the next several years, we are going to see a range of activities directed at COVID-19 economic relief and recovery efforts for small businesses. Measuring the impact of all these activities—including crisis relief, innovative loan structuring, capacity building, and research—could help investors across philanthropy, government, and impact investing identify relationships between recovery solutions, isolate effective strategies, and learn more about the resiliency of systems at play. For this type of cross-sector, far-reaching learning to occur, investors must facilitate sound measurement strategies by sharing information and helping identify causal trends. A comprehensive strategy would include both vertical and horizontal layering to better understand individual interventions and the system-wide impacts.

Horizontal and vertical layering can help prioritize social impact, starting with due diligence and throughout the investment lifecycle. Moving toward thoughtful deployment of existing measurement frameworks and the development of new and improved tools will also improve transparency in the impact investing space. Currently, approximately 75 percent of impact investors only report to internal stakeholders, and only half of impact investors make their reports public. Transparency about impact measurement through regular reporting will help push the best solutions forward and reduce internal bias in the field.

Improving impact measurement requires engaged leadership and dedicated capacity. For the field to authentically align wealth building and social impact, investors should approach measurement with an eye toward building and sharing evidence with the field. The market response and the impact evidence must align for true social impact to occur. This alignment is more crucial than ever, as financial markets appear detached from social and environmental progress, and the world continues to grapple with COVID-19. Now is the time for investors to insist on layering evidence to create meaningful impact.

This work was supported by a grant from the Bill & Melinda Gates Foundation. The findings and conclusions contained within are those of the authors and do not necessarily reflect positions or policies of the foundation.

Read more stories by Kendall Rathunde, Daniel Hadley & Gwendolyn Reynolds.