Getting Beyond Hype: Four Questions to Predict Real Impact

Through a kind of magic I don’t fully understand, some emerging poverty solutions—products, services, technologies—become shiny new objects that trigger something akin to a feeding frenzy. Celebration at CGI and SOCAP and TED, breathless articles in Wired and Fast Company, endorsements from celebrities, awards of all kinds—all that stuff is great if the thing has real potential for impact. Too often, though, the appetizer becomes the entrée and hyperbolic celebration displaces systematic evaluation.

That’s a shame, because we really do need to sort good ideas from bad, and it ought to happen before we stoke the fires of publicity. At Mulago, we’ve spent a lot of time trying to figure out if start-ups with a new “thing”—a product, service, or technology—are likely to create real impact in the lives of the people we’re trying to serve. Over time, we’ve evolved a set of four questions that help us make better predictions. Because we’ve been thinking a lot about cookstoves and indoor pollution lately, I’ll use improved cookstoves to illustrate how those questions work.

1. Is it needed?

Given the nature of the thing, is there a profound social impact to be had? Every thing ought to have a mission (preferably in one that’s eight words or less). If you can’t figure out a clear mission that matters, all you may have is a solution in search of problem—and some things ought to be killed while they’re young.

We might reasonably ascribe various missions to better stoves, including: improve health, prevent deforestation, save time and money, and decrease carbon emissions. Since indoor air pollution is a major killer and bad stoves are the major contributor, we think that the mission to “Prevent Respiratory Disease in Poor Families” is a compelling one and rises above all the others (especially since the evidence connecting the other potential missions is considerably more sketchy). So the answer to question one is yes, absolutely, we need better stoves.

2. Does it work?

If the thing is deployed as designed, under the conditions for which it was designed, will it have real impact? In the case of stoves, this is mostly about emissions: Does a given stove reduce pollutants enough to drive a significant improvement in health? Recent research indicates that most improved cookstoves don’t. It turns out that you have to reduce more than 85-90 percent of pollutants before you see any real reduction in respiratory illness (Burnett et al, 2014). A stove that emits 50 percent less pollution doesn’t accomplish much. To date, the only stoves that make the cut are the expensive “forced-draft” models that use a built-in fan to stoke a much more efficient burn. The answer for most stoves, however admirable the effort may be, is no. Really.

3. Will people use it as designed?

Behavior, behavior, behavior. Despite all that user-centered-design, iterative-rapid-prototyping stuff, people use and abuse things in ways that leave even the best designers scratching their heads. Too many—way too many—products have made their way into wide distribution without evidence that people use them as intended. Successful things, though, usually have these qualities in common: They fit well with local customs and culture, are easy to use right and hard to use wrong, and need little maintenance (and when they do, they’re fixable and you can get parts).

Everyone has heard about stoves that people use improperly—or not at all—but field researchers have handed the stove industry another fun surprise: Even when people do use that wonderful new stove, they often use the old one too (Ruiz-Mercado et al, 2011). It’s become known as “stove-stacking,” and you see it when you visit homes—there’s the new stove, fired up and roaring, and there’s the old three-stone fire still belching smoke in the corner. Oh no! If you want to fulfill the mission—prevent respiratory disease in poor families—not only do you have to provide an expensive stove, you might have to provide two! It kind of sucks, but that may be what it takes to get to yes.

4. Will it get to those who need it most (a lot of them)?

Nothing matters if the thing doesn’t reach the people it is intended to serve. If you don’t have a reality-based idea about distribution, you should hold off on the design process until you do. Three things matter above all else:

1. Price. If it’s too expensive, they won’t buy it. Knowing the customer’s price point is your first order of business, and you need to design the thing to hit that number.
2. Distribution channel. Poor people live in a world of market failure, and there are a finite number of ways to get things to them—the hardest is to set up your own network.
3. Sales. Somebody has to make the transaction happen—who are they and how are they going to do it?

And, of course, if the thing costs more than about 20 bucks, you’d better think about financing, too…

This is where stoves have always struggled: The affordable ones are inadequate, and the good ones are unaffordable. Even with the (unreliable) assistance of carbon credit subsidies, stoves that can fulfill our health mission are still too expensive. Emerging technologies will almost certainly lower the price of forced-draft stoves, but for the time being, they’re mostly out of reach for those who need them most. Financing can help, but poor people often have more urgent things to do with available credit.

One organization working on the stove problem that gets to “yes” to all four questions is Inyenyeri, a Rwandan company that leases ultra-low emission, forced-draft cookstoves to households at a nominal cost (about $7 per year), and sells customers the fuel pellets to burn in them. The pellets cost less than charcoal, and people who can’t afford to buy them can trade biomass for pellets. Inyenyeri leases families two or even three stoves to account for stacking. It’s still early days and there are a lot of moving parts, but it’s headed in the right direction.

The four questions are unforgiving; getting a no on any one of them means you won’t pass go on your way to impact. That doesn’t mean that a no answer should lead you to abandon the whole effort, but it should refocus your efforts on getting to yes. Stick with R&D, and avoid the big splash until you’re pretty sure you’re there—it’s the least we owe the people we serve.

Read more stories by Kevin Starr.