When Less Impact Per Client = Greater Social Good

Over the past decade or so, social sector evaluators have increasingly coalesced around a pretty simple formula for measuring the effectiveness of single, direct-service interventions: Select one meaningful indicator that best encapsulates the benefits an intervention creates; measure this as rigorously as possible for both clients and a control group (to determine impact); and compare impact to the donor cost to determine cost effectiveness, or net impact. For some indicators (such as child diarrhea in developing countries), we have evaluated enough interventions in this way to make comparisons, allowing a deeper understanding of which kinds of solutions create the greatest “bang for the buck” (in this case, point-of-use chlorination).

The assumption, of course, is that greater bang for the buck is better and that as organizations scale, their impact (and net impact) per client should grow over time. One important driver is organizational learning—measurement should identify opportunities to tweak the intervention to increase its impact. A second is innovation—measurement should identify new products or services that drive up impact per client. A third is economies of scale—as organizations grow, their cost to deliver the intervention should decrease as a result of purchasing economies, the spread of fixed costs over more clients, and so forth.

At One Acre Fund, our impact per client (measured as the extra income generated by the average farmer in our program relative to similar farmers in their communities) followed this traditional path in our “proof of concept” phase—it steadily increased as, for instance, we delivered agricultural trainings more effectively, launched new impactful products like solar lights, and achieved bulk discounts on farm input purchases. But in more recent times, our impact per client has fluctuated, and we believe that our experience holds important lessons for mission-driven organizations, their funders, and evaluators.

Here we share four strategies our organization has undertaken during our “scale-up” phase that have put downward pressure on our impact per client, but have simultaneously increased our potential social good:

1. Reaching a more underserved population: After five years of continuously improving our program model in Kenya and Rwanda, in 2011 One Acre Fund decided to launch in Burundi, the absolute hungriest country on the planet. By almost any measure—baseline farmer knowledge, average arable land size per farmer, availability of quality seed—Burundian farmers are worse off and agricultural impact is simply harder to generate than anywhere else in East Africa. Our impact in Burundi measured in the traditional way (income generated per client) is about half of the average of our other countries; hence, our weighted-average program impact declined as a result. But our percent increase in income per client is highest in Burundi, reflecting the dramatic life improvement our model is creating. The lesson: Differences in target population can translate to differences in conventionally measured impact, requiring an expanded set of success measures to understand the true social good an organization is having.
    
2. Enhancing measurement rigor: Calculating impact can appear deceptively simple: Just compute the number of diarrheal incidents or school days attended over a period of time—or in our line of work, just calculate farm income (selling price per output less cost per output; all multiplied by total output). But the truth is that every organization makes a host of assumptions that can have great bearing on the final magnitudes they calculate. For instance, in 2014, One Acre Fund undertook a comprehensive assessment of how agricultural organizations and their evaluators measure farm income. We discovered little consistency but felt that it was important to implement the most rigorous approaches we found (for example, estimating the value of unpaid labor expended by family members), even if our impact estimates are lower relative to prior years. We believe continuously enhancing measurement rigor is the right thing to do, as it provides an organization with even more accurate data for decision-making and increases the social good it creates in the long run.
    
3. Widening the lens: While the growing view that we can capture impact in a single indicator is convenient for comparative analysis, it doesn’t always represent the reality of what mission-driven organizations are trying to achieve. At One Acre Fund, single-season farm income is indeed the best single way to measure success, but our unique, long-term relationship with farm families means we also care deeply about the long-term productivity of their lands. Enabled by greater scale, in recent years we have trialed several (and launched some) new products—such as forage and tree legumes—that are highly beneficial for long-term soil health (and hence, the sustainability of farm income), but have zero or negative impact on single-season farm income. Net-net, we believe these products are highly valuable for farm families, even if our impact, conventionally measured, looks lower. The lesson: Efforts to generate new forms of impact that enhance overall social good may actually result in lower conventionally measured impact per client.
    
4. Creating your own competition: As we shared in a previous post, mission-driven organizations are keenly aware of the magnitude of the social problems they address and often seek to grow well beyond the
   microscopic “market shares” they typically command. While there are many pathways to achieving transformative scale, one that our organization is investing in is a government partnerships unit; it works alongside African governments on joint implementation of nationwide farm programs. In the territories we serve in Rwanda, for instance, One Acre Fund has partnered with government to recruit, train, finance, and supply 150 “agro-dealers” (mom-and-pop retailers) who actively compete with our own program. With nearly half of non-One Acre Fund farmers (our traditional control group for measurement purposes) improving their incomes through this partnership, our core program impact per client will decline, all else equal. But by addressing a vast new segment of the market with impactful, cost-effective programs (albeit not as high-impact as our core model), the overall social good we are creating has gone way up. As before, a decline in impact per client may correspond to an increase in total social good when organizations pursue transformative scale strategies that seek to improve entire markets or systems.

These four actions may stall a steady march upward in conventionally measured impact per client. Of course they do not in any way excuse a lack of laser-like focus organizations should have on increasing their impact per client in the three ways described earlier. However, they do suggest a need for a far more nuanced understanding of why and how impact may fluctuate over time, and how organizations may need to expand their definitions and measures of success in response.

Read more stories by Matthew Forti.