Measuring the Indirect Impacts of Market Innovators

Conventional wisdom insists, “If you can’t measure impact, it doesn’t exist.” The pressure has never been higher on nonprofits, donors, aid organizations, and impact investors to demonstrate that we’re achieving what we’ve promised. Anecdotes, imagery, and faith no longer cut it.

Organizations and funders have recently poured plenty of money, time, and intellectual energy into identifying and proving the development impact of our projects and investments. Last year alone, funders spent millions on randomized controlled trials that use experimental models to tease out causal linkages between actions and outcomes. Foundations now routinely employ teams for “monitoring and evaluation.” And an entire industry has sprung up to encourage better tracking, reporting, and standardizing of social performance metrics at the company and organizational levels.

While these efforts are crucial, we worry that such focus on measuring only direct outputs and proving causal certainty might miss something important—specifically, we might miss a constellation of indirect impacts that are very real, even though we don’t yet have the tools to understand or articulate them.

Take the case of “market innovators,” which, as the Omidyar Network articulated in a 2012 report, are those pioneers—usually start-ups—that “contribute meaningfully to a sector by de-risking the generic model of an innovation or product.” These innovators generate positive sector-level externalities—informing, inspiring, and instructing others’ efforts to build robust sectors addressing the needs of the poor—even though not all may individually succeed in scaling up. Start-up pioneers often spend years laboring in the trenches to reach the first handfuls of customers. But the value of market innovators lies not just in the first customers it reaches on its own, but also in the ways that it enables others to serve millions more in the years that follow.

Sounds great in theory, but how can we measure this?

At Accion Venture Lab, we are on the hunt for precisely these externality-generating innovators. We support the world’s most innovative “financial inclusion” start-ups—those companies seeking to help underserved customers access better financial services and tools to improve their lives.

We’ve learned that supporting these innovators is hard and uncertain work, and, unfortunately, there are only a few of us looking at these seed-stage start-ups. Clearly, one big barrier to further investment in market innovators is that folks who do this work can’t readily explain or prove the indirect impacts they’re having. This is a special challenge for a number of reasons. It’s hard to define impact trajectories early in a start-up’s life. It’s also hard to attribute quantifiable impact to the activities of a single company within a sector. Even when these indirect impacts happen, the timeframe over which the impacts become visible often takes years—long enough to outlast the patience and “results frameworks” of most funds and donors. And it’s uniquely difficult to make apples-to-apples comparisons at the level of indirect impacts, even more so than the commensurability challenges of traditional cross-sector impact metrics.

So, bearing all this in mind, how might we start to measure and talk about indirect impact?

For us, this question isn’t one of simple intellectual curiosity—the answer is integral to securing the support of funders and other partners for these start-up pioneers. We don’t claim to have all the answers, but here are a few things we’re trying:

• Define alternate pathways to scale: Start-up innovators won’t always (or even often) be the ones to achieve scale on their own. Many times, scale comes only when other players get involved, whether as partners, acquirers, or imitators. Let’s tolerate and even celebrate these alternative “pathways to scale”—when we can track a clear line from the initial experiment to a scaled effort, let’s highlight the first mover’s role in enabling success (see this example from Kenya).
• Focus on the sector, not the firm: To borrow from Omidyar Network, we’re interested in building sectors, not just firms—so let’s start measuring that way. For example, let’s keep track of not only how many customers one of our investments reaches, but also how many customers the sector reaches as a whole. Yes, attribution of general-sector results to specific firm actions will be a significant challenge, but let’s start collecting the data now and hope our tools and understanding of causal linkages can improve with time.
• Make efforts to raise awareness and monitor “buzz”: A start-up can’t be a demonstration case if no one knows about it, so we are proactive about sharing our companies’ stories, successes, as well as failures and lessons. We track these instances, and the time we spend speaking and writing for broader audiences. We also actively monitor our start-ups’ media hits and general industry buzz as a (very imperfect) proxy for influence. Could we someday have a Klout score for market innovators?
• Tolerate different output data at different innovation stages: Output measures look very different during the various stages of an innovation’s life cycle. There is a danger in thinking that firms pursuing more-proven, later-stage innovations are having more impact simply because they have more concrete results. Start-ups may generate lower-volume outputs, but these may lay the foundation for steeper growth and impact trajectories in the future—that is, going from 0 to 1,000 customers may suggest a bigger impact than, say, 1,000,000 to 1,001,000 customers.
• Be patient:  As time goes on, the job gets relatively easier. As companies evolve, we can point to indirect impacts more clearly: the emergence of copycats or competitors; measures of “crowded-in” incremental investment in particular sectors; the relative speed with which subsequent entrants achieve break-even; specific anecdotes or theories about how a given management team might have inspired, mentored, funded, or founded similar enterprises (potentially conveyed through visualization); and actual macro-level indicators suggesting that the company has demonstrably “moved the needle” in a sector or region. We hope for a day when we can construct and measure an “indirect impact multiple” for investing early. If one entrant serves a million customers but lays the groundwork to serve millions of customers, is there a way we can index this?

Is any of this as rigorous, objective, or quantifiable as a randomized controlled trial or firm-level social performance metrics? No—at least not yet. We’re in the early days of making all of this real, though the stakes are already high. Without credible proof of indirect impact, innovators may continue to face a “pioneer gap” of funding and resources. So if you believe these impacts exist and can be game-changing, let’s try to figure it out.

After all, if impact exists, we must be able to measure it.

Read more stories by Rishabh Khosla & Paul Breloff.