Measurement & Evaluation

Measuring Social Return on Investment Before You Invest

Using a social return on investment framework, organizations can estimate the future impact, cost, and scale of programs before they begin, and allocate resources for greater impact.

Nonprofits serious about maximizing their social good typically use measurement to: improve programs during implementation and prove those programs post-implementation. And at One Acre Fund, which helps smallholder farmers prosper, we’ve found that the real-time learning and discipline that emerge from this kind of measurement drive greater impact per client over time.

But in recent years, as we’ve grown and pursued multiple delivery models in multiple geographies, we began to wonder whether we could use measurement data before implementing programs to determine which ones should get off the ground, and with what level of resources, in the first place.

Internally, we refer to this measurement forecasting as the “Therese challenge.” Therese Nyiransabimana was the hundred-thousandth farmer enrolled in One Acre Fund’s Rwanda program, which launched in 2007. We currently reach Therese via our direct service program, which provides a complete bundle of farm inputs, finance, training, and market facilitation. We could devote funds the following year to innovate new farm and non-farm products for Therese; to grow this direct service program for Therese’s neighbors; to invest in our agrodealer program, supplying farm inputs like seed and fertilizer to retail shops in Therese’s community; or to expand other programs or countries.

The realization that we needed a framework for resource allocation led us to explore and adapt valuation tools commonly used by private enterprises (and pioneered by REDF for social purpose enterprises). We came to define the metric social return on investment (SROI) as the ratio of impact generated per client, to net cost per client; in our work, impact is the incremental profits from our programs, and net cost is the program expenditures required less earned income from our clients. Given our increasingly diverse portfolio, we aim to use SROI to compare direct service programs, partnership programs, R&D programs, and even speculative new programs—all across multiple geographies.

For instance, in 2014 it cost $10.40 in field expenditures to serve each farmer in our Burundi direct service program, and that program generated $67.40 of impact per farmer; the SROI of our Burundi program was thus 6.5. For 2016 resource allocation, we forecast the cost and impact per farmer in Burundi (taking into account initiatives to improve transaction size per farmer, staffing ratios, and so on).

A more challenging program for SROI is new product R&D, which trials dozens of innovations over multiple years. Only a few of these innovations ultimately roll-out into our direct service program, but with several assumptions (notably the likelihood of roll-out, and the expected adoption percentage and impact per client each year once launched), we could calculate and then discount future-year impact (much like a private sector company would discount future profits). We could then compare the resulting “present-value” of impact with the cost of running the R&D department to generate an SROI.

Finally, our program and country leaders can forecast their impact and cost paths at varying investment levels to estimate the “marginal SROI” of investing an extra dollar—for instance, if we have an extra $5 to spend per farmer, how much additional impact per farmer could our Rwanda direct service program generate? From there, we can make intelligent investment decisions that maximize social good given budget constraints.

We are in the extremely early stages of using SROI, but wanted to share four lessons that have already emerged and how we’ve adapted:

  1. SROI should not be the only program investment criteria. Many other factors drive the attractiveness of investing in one program over another. Most notably, scale (and scale potential) is critical—generating a large marginal SROI over 5,000 farmers is not as exciting as a slightly lower SROI over 250,000 farmers. Other factors are the strength of or confidence in the impact estimates, purchasing power that an extra dollar of income creates, and spillovers that one program may have on another. We are therefore developing an “investment scorecard” so that program leaders can report a wider range of criteria.
  2. Integrating multiple forms of impact is a challenge. We use income impact as our common metric; yet certain programs within One Acre Fund may disproportionately generate other forms of impact, such as nutrition or soil health. While for some impact types, conversion factors (to income impact) may exist, this nonetheless complicates SROI comparisons. We are thus allowing program leaders to describe other forms of impact in their scorecards.
  3. “People” allocation may be a more relevant metric than budget allocation. As a labor-intensive operation that values similar skillsets across programs, we realized a more pragmatic question to ask program leaders is: “How much more impact could you create with an extra person?” Internally, we are therefore looking at both impact per cost and impact per full-time employee, but we plan to use the latter to allocate discretionary hires in our 2016 planning.
  4. To go deep requires infrastructure: We realized that to use SROI for resource allocation, it needed to live outside just the measurement-and-evaluation and finance functions. Hence, we assembled a broad working group to set ground rules, motivate honest and accurate forecasting, hold programs accountable for previous forecasts, and make resourcing decisions transparent.

As we look to greater use of SROI in the social sector, the pioneering work of organizations like Population Services International (PSI) in global health provide a powerful template. PSI, which markets affordable health products and services, estimates its effect on disease burden much like a company measures profit. Since 2007, PSI has tracked the health impact of its work as the number of disability-adjusted life years (DALYs) averted; this captures both deaths and illness PSI’s health products and services prevent. According to Amy Ratcliffe, PSI’s director of program analytics, this metric allows PSI to aggregate impact across its portfolio, as well as disaggregate it for comparisons, cost-effectiveness analysis, and forecasting within countries or health areas. Since the DALYs averted metric is specific to a given year, country, disease outcome, and intervention, PSI can consider complex investment trade-offs such as, “What can we gain from scaling up malaria control in Mali?” Or, “What are the long-term implications of delaying HIV testing in Zimbabwe?”

PSI has gone one step further in recent years, standardizing its health impact metrics with leading external models endorsed by academics, partners, governments, and donors. In doing so, PSI has greatly improved both efficiency and confidence in its own measurement, and driven greater alignment and collaboration in its field.

Although we are much earlier in our journey than PSI, we are confident in the potential of SROI to further improve our own impact through more-disciplined resource allocation decisions. We also believe it will eventually play a central role in how donors, nonprofits, and other stakeholders interact in the social sector. Although an SROI framework can quickly become complicated, we believe any nonprofit with a growing portfolio of (largely) service delivery programs can define its programs and geographies, select its single best impact and cost metrics, and develop a process around SROI to get started.

Tracker Pixel for Entry
 
 

COMMENTS

  • Dylan Edwards's avatar

    BY Dylan Edwards

    ON November 18, 2015 11:34 PM

    Thanks for this contribution. I think it is really important for impact investors to set impact targets and expectations for impact before deploying capital. However, I have found SROI quite a limited instrument for making these forecasts.

    This is partly because of lesson #2 in your list - integrating multiple forms of impact *is* a challenge. It is one thing for funds such as yours, which are generally focussed on creating one main impact metric (in your case increasing incomes to smallholder farmers), to use SROI. It is quite another for funds with a broader investment mandate. For example, in a health fund, how do we compare the social return of investing in a manufacturing facility to that of investing in building new clinics? Or even to that of investing in the latest telemedicine app for the developing world? And some funds have an even broader focus - the so-called “sector agnostic” impact investors. How do they weigh up impact in education vs health vs renewable energy vs affordable housing?

    My suspicion is that in these cases the meaning of SROI will be lost. More important will be benchmarking important impact metrics against peer organisations. This will allow investors to seek out deals that are “best in class”.

  • BY Adam Thomson

    ON November 19, 2015 12:50 PM

    Thanks for leading the charge Matt and Jake. Your commitment to transparency and accountability is inspiring. It’s evident to me that you hold yourself to accountable to these standards not because you have to, but because you are passionate about being the best iteration of yourself. That passion is contagious. Excited for more posts and lots of learning to come.

  • Matthew Forti's avatar

    BY Matthew Forti, One Acre Fund

    ON December 2, 2015 09:38 PM

    Dylan and Adam - thank you both so much for taking the time to respond.  Adam - thank you for recognizing how passionate we are about this important topic and hope we can spur others to join in the fun!  Dylan - thank you for raising the important point that SROI can be a much more complicated tool when comparing within or across sectors that have multiple impact metrics.  There are many possible adaptations (attempting to split costs related to each outcome; attempting to weight outcomes of interest, creating a scorecard where other outcomes can be described - which has been our preferred approach).  At the end of the day, probably no one right answer on this difficult question, but forecasting impact, cost, and scale as inputs seems to me still quite important.

  • BY Nathan Huttner

    ON December 3, 2015 09:10 AM

    Thanks, Matt and Jake! This is really exciting work. Three thoughts:
    1. You mentioned risk in relation to your innovation pipeline, which makes a ton of sense. Have you given any thought to risk or variability in returns on the direct service portion of your portfolio? It seems like some kinds of programs or interventions might have a higher variance in impacts, and that might be worth taking into account as you decide where to invest.
    2. If you are taking risk into account, do you ever consider how your investments work as an overall portfolio, and what the portfolio risk might be? If one country offers really high returns for its projects, you might still choose to diversify investments to other countries to reduce your portfolio risk by limiting exposure to that country’s politics, economics, and weather. Obviously, portfolio optimization can be modeled for SROI, too, but only you can know whether the juice is worth the squeeze.
    3. Your comments about staff time and not dollars invested serving as the scarce resource made me think that there may be other analytical tools that could be useful, both before or after investment. For example, the International Planned Parenthood Federation has used Data Envelopment Analysis to estimate the efficiency of its clinics; that’s a great tool for dealing with problems where you have multiple inputs (dollars, people, infrastructure) that aren’t easily traded off in the near-term, and multiple outputs (though that might be less relevant in this case). This kind of approach might be a useful analytical way to uncover some of the more troublesome tradeoffs you all are uncovering.
    Keep up the great work!

  • Matthew Forti's avatar

    BY Matthew Forti, One Acre Fund

    ON December 7, 2015 04:58 PM

    Nathan - thank you for such a thoughtful set of comments!  On #1 and #2 - impact variance and portfolio diversification are absolutely important factors that we do implicitly consider, but should definitely build more formally into the SROI framework we proposed.  On #3 - thank you so much for sharing the Data Envelopment Analysis tool - we’ve not seen something like this before and it does sound quite relevant -  we forward to digging into this as we prepare to refresh our analysis and increase our sophistication in 2016.  Thank you again for these great thoughts!  -Matt and Jake

Leave a Comment

 
 
 
 
 

Please enter the word you see in the image below:

 

SSIR reserves the right to remove comments it deems offensive or inappropriate.
 
 
 
 
Measurement & Evaluation

Practice Safe Stats! A PSA

Featuring Jake Porway 1

Listen to Jake Porway, founder and executive director of DataKind, talk about how to “practice safe stats”—that is, how to create data visualizations that are both accurate and inspiring.