This Is Not an Impact Performance Report

(Photo by iStock/Pavel1964)

In a world of increasing transparency, we expect that what’s on the label will reflect what’s inside the package. This is as true for an “organic, cage-free” label on a carton of eggs as it is for a B Corporation Certification or a fund categorized as “ESG.” These terms communicate something specific to the buyer. Their credibility rests on whether what’s on the label is consistent with the product itself.

In the last five years, impact investing has grown tremendously: Today, more than $40 trillion of assets are classified as ESG. Accompanying that growth, there’s been a marked increase in activity around how to measure and manage impact. This work is essential: It will enable investors to direct capital to make the most change, and it will empower investors and companies to manage impact performance with the same rigor as financial and operational performance. In recent years the impact investing sector has made progress developing better, more accessible, and more transparent ways of talking about impact, from the IMP’s five dimensions of impact to the IRIS+ indicators and the IFC Operating Principles.

Yet despite this progress, we have yet to set a clear minimum expectation around what constitutes “good enough” impact data to judge performance. Is it an articulation of intent? Assurance that certain practices are being followed? Or do we need hard data about material impacts for people and planet? Lacking this shared minimum expectation, current impact performance reports typically rely on basic operational data that are represented as impact created. As a result, most impact “reporting” comes up short: It serves as a self-affirming indicator of good efforts rather than an objective view of performance.

What’s Special About Performance?

As impact reporting has become more mainstream, there is increased recognition within impact investing of the need to strive for top-notch “impact performance.” Historically, impact was often treated as binary: Either it was occurring or it was not. If an intervention was known to be “impactful”—often thanks to an academic study—it was typical to act as if the impact question had been settled, and, therefore, no additional data from the company was needed to confirm that it was, in fact, creating impact.

This approach, though appealing and seemingly pragmatic, was fundamentally flawed. To understand why, think about the shift in our collective understanding of human learning and development that has happened thanks to the introduction of the concepts of “fixed” and “growth” mindsets. This work, popularized by psychologist Carol Dwyck, showed that our most basic beliefs about who we are and our abilities (for example our intelligence, personality, morality) shape our ability to learn, grow, and improve. Those with a “fixed” mindset believe that intelligence is static. They prioritize looking smart and capable and they therefore tend to avoid challenges, are more likely to give up easily when faced with obstacles and ignore useful negative feedback. Conversely, those with a “growth” mindset believe that intelligence can be developed, so they are more likely to embrace challenges, persist in the face of obstacles, and learn from criticism.

Impact investing has a similar potential to leap forward if we recognize that we need more than “evidence” of impact—something fixed, proven once, that either exists or it doesn’t. We also need continuous “impact performance” data—data that is dynamic, fluid, and iterated upon, so that we can grow our impact. We tend to slip up when we confuse the two.

We see the limitations of the “fixed” impact mindset every time we think we’ve discovered the “truth” about the impact of any given intervention (a microfinance loan, an improved school curriculum). It is, of course, helpful to uncover studies that show that an intervention works. But relying on studies alone and applying their findings to (somewhat) similar interventions ignores something essential: differences in impact performance are the result of the specific actions of a company, the specific characteristics of a product or service, and the specific ways the company interacts with customers in a local context. These differences, by definition, are invisible if one relies on a static definition of “this product or service (always) creates this much impact for a customer.”

And yet, accepted best practice is to tick the mental “yes” box on impact and then apply the “proven” impact variables to standard operational data. This can conjure up lofty impact numbers, but the underlying, flawed premise of this all-too-common approach is the assumption that entire categories of interventions (e.g., every microfinance loan given to everyone everywhere) are essentially the same from an impact perspective and should be treated as such.

By contrast, an impact performance mindset is grounded in the knowledge that each intervention has a different impact in each place and each moment in time. Moreover, these differences are nearly always large enough to merit our attention. Understanding variances in performance, whether in impact or any other area, is the prerequisite for improvement.

For example, imagine two consumer loans from two different providers in the same country.  One is a simple digital transaction, and the other has significant community support and wraparound services. If we discover that the second loan has consistently greater and longer-lasting results on clients’ financial well-being and empowerment, then we’ve learned something essential about how we can allocate our resources towards the greatest impact. Our job, whether as operators or investors, is not to say “we are supporting lending to marginalized populations, and here’s the typical impact of these sorts of loans.” Rather, our job is to identify the top performers by differentiating between those who create outsized impact within a given category from the bottom or middling performers.

Social interventions of all types exist in much too dynamic a context to blindly extrapolate from a single anecdote or study as broadly as we do. We would never assume that each supermarket, airline, or online marketplace has the same operational and financial ratios per unit sold. Yet we act as if the well-studied impact of one intervention in one place and one moment in time can represent all interventions of that type globally.

From Theory to Practice

All this might sound sensible, but it’s a far cry from current accepted practice. Current practice is to starve impact measurement of any real resources, thereby forcing practitioners to prioritize pragmatism above all else. Constrained in this way, we scramble to identify the data we can most readily produce and try to deduce potential impact from these data.

For example, consider the recent GIIN report titled, “Understanding Impact Performance: Financial Inclusion Investments.” One would expect that such a report would be anchored around (1) the types of impacts that matter to a client of a microfinance or fintech organization; and that, (2) as a “performance report,” it would allow one to understand the relative impact performance of financial inclusion impact investors and financial inclusion companies. Unfortunately, it accomplishes neither of these two objectives.

How well does it map to the first objective: reporting on the sorts of impacts that matter to microfinance clients? Not very well. To see why, imagine the simple exercise of asking low-income microfinance clients why they borrow or save with a given MFI. These clients would likely tell you that they want to have more savings (so they can invest more), better withstand financial shocks, or be able to send their kids to school in the future; or they want increased income from growing a small business; or perhaps they are eager to have more say in family financial decisions.

Yet the metrics in the GIIN Financial Inclusion impact “performance” report (found on pages 20-21) do not include these sorts of simple, client-level impact metrics. In fact, of the 28 “impact” indicators in the report, only five of them bear even a slight relation to these essential client outcomes. This is because, as is typical, the report anchors in the data that are readily available—for example, number of loans disbursed or percentage of loans provided in local currency—and tries to deduce (potential) client-level outcomes from these data. But while the number of loans a microfinance organization has disbursed does give a sense of reach, it tells us nothing about the things we really care about: what’s the impact of a microfinance loan on these clients of this MFI in this time period?

What we have in the GIIN report, then, is a very useful “state of the sector” snapshot that helps us understand the size of the sector, average loan sizes, portfolio at risk, and a bit of data about who is being served. This is all very informative. Almost none of it is “impact performance.”

To be impact performance report, it would have to, at a minimum:

1. Be anchored in the impact priorities of the affected stakeholders (customers, employees, planet),
2. Have at least some impact performance data gathered directly from these stakeholders, and
3. Have data that allows for comparison of the relative impact performance of different enterprises or impact funds engaged in the same or similar activities

Our Open Secret: Spotty Impact Data

The spottiness of impact data is our industry’s most open secret. The question we must therefore answer is how best to solve it.

The GIIN view, according to this same report, is to continue down our current path: collect operational metrics that companies can produce and use improved primary research to transform these data into impact data. In their language, this is the path of “improved triangulation.” (From the report: “Resource constraints associated with directly collecting outcomes data ... highlights an opportunity to expand the industry’s ability to triangulate to outcomes based on sets of outputs using evidence-backed analytics and research methods.") Similarly, the TPG Rise Fund’s IMM methodology, as explained in a 2019 Harvard Business Review article, is anchored on triangulating operational metrics with “existing research [that] verifies that they are achievable and measurable.”

Our concern with triangulation is that it is fundamentally at odds with uncovering the differentiation that is at the heart of impact performance.

Imagine once again two lending companies both operating in Uganda. Both companies have similar written policies, similar-sounding products, and roughly equal-sized client bases. Now imagine that one organization has loan officers who regularly harass clients, who gave no payment holidays during COVID-19, and who do not follow their own written policies and procedures. No amount of better triangulation will ever pick up the differences in clients’ experiences of these two organizations and the resulting difference in clients’ savings, earnings, repayment rates, household gender dynamics, or kids’ educational outcomes.

If triangulation cannot pick up even blatant gaps between written policies and actual behaviors, why would we bet on it as the lynchpin for understanding impact performance?

We shouldn’t.

Impact triangulation is by its very nature self-limiting, a cul-de-sac and not an on-ramp to impact performance. Impact triangulation assumes that a customer served is a customer served and that the impact uncovered in one (or five) studies of similar-enough organizations is universal. If only transforming people’s lives for the better were that simple!

Our objective should be to go directly to the stakeholders for whom these enterprises are meant to create better outcomes and, quite simply, ask them whether or not those outcomes are occurring. Even if this data is not perfect, even if self-reported data is often subjective, surely it is better to ask and get the data than it is not to ask at all.

Impact Measurement Innovations

The good news is that we are living in a period of massive acceleration and innovation in the collection of beneficiary-level outcomes data by some of the world’s leading impact investors and social enterprises. Flourish Ventures, Omidyar Network India, Global Partnerships, REGMIFA, BRAC, the Rockefeller Foundation, Ceniarth, and Solar Sister have all recently shared their experience with collecting client-level data. In a similar vein, last year we at 60 Decibels published a Why Off-Grid Energy Matters report that includes a true ranking of impact performance, aligned with the Impact Management Project (IMP), for 59 off-grid energy companies, all based on what their customers said about the impact they experience.

Or, for a particularly time-sensitive example, think about the importance of understanding COVID-19 impacts globally. It is obvious that one couldn’t assess this by any sort of triangulation. And yet, by directly asking more than 52,000 customers in 32 countries, 60 Decibels has, from April to December of 2020, gotten a detailed and differentiated understanding of how COVID-19 has affected people’s lives. Surely if we can get this data, monthly, during a global pandemic, then there must be a way for our sector to gather these data in calmer times.

A Higher Standard

In 2001, 20 years ago, Jed Emerson famously quipped: "We fund endless studies to guide us toward a vague truth. Still the answer remains: We simply do not know. Instead, we have created what is all too often a collective dance of deceit whereby funders are told what they like to hear, and grantees are freed of true accountability for their efforts."

Too much impact capital, and too much effort, is being deployed to keep on settling for vague estimations of impact performance. We have a duty to the people being served by impact investments to push past “likely outcomes” and triangulated output data and work towards impact performance. Indeed, if we agree that our work is about these customers, we must collectively call foul every time their voices are absent in a report about their lives and their well-being. Without their direct feedback, these reports quietly reinforce the faulty and pervasive logic that it is possible to assert that one is having impact in client-facing investments without ever hearing directly from clients.

Why are we still doing this? Instead of participating in this same dance of deceit, let’s agree that it’s high time to get off this sullied dance floor.

The Way Forward

To collectively improve, we make the following three recommendations:

1. Be wary of “impact performance” reports that don’t allow you to assess performance. “Performance” means that some organizations do better, and some do worse. There are leaders and laggards. If your performance rating methodology doesn’t allow you to distinguish in this way, it is, quite simply, not a performance rating.

2. Be transparent about where your data came from. All impact reports should clearly signal where they do and do not have outcomes data, as well as explain where outcomes data are based on direct data-gathering as opposed to “impact math.” The field takes a step back each time we blur the line between operational metrics and outcomes data collected from stakeholders. If we, as impact investors, truly care about customers and believe that the impact on their lives matters as much as returns to shareholders, then our aspiration must be to hear directly from these customers when assessing impact.

3. The impact measurement and management conversation must prioritize outcomes data. Until then, packaging and repackaging outputs with glossy reports featuring big numbers will continue to be both burdensome and futile. At best, these output-based reports serve to justify investment decisions and attract more capital. At worst, they perpetuate the dance of deceit at the expense of learning about what works to create better outcomes for customers.

It’s high time we reject the premise that impact performance data doesn’t exist when it does, in abundance. We can do better than the common, harmful deception that has gone on for too long in our industry: impact reports that do not report on impact, and performance reports that tell us nothing about performance.

Read more stories by Tom Adams, Lindsay Smalling, Devin Olmack & Sasha Dichter.