Lessons from a Development Impact Bond

On July 5th, the year-one results of a three-year Educate Girls development impact bond (DIB) in India came out. Among the first of its kind, the DIB targets roughly 15,000 children and 166 government schools in Rajasthan, with the aim of enrolling out-of-school girls in school and improving both girls’ and boys’ literacy in English, Hindi, and math. Although a lot remains to be tested and proved, early results suggest the model has great promise for improving the way programs are financed and delivered on the ground. The program has so far enrolled 44 percent of all out-of-school girls identified in year one and has achieved 23 percent of its total target for learning progress. In our calculations, the DIB is on track to meet its 2018 targets for both enrollment and learning gains. It also has produced a step change in how the service provider, Educate Girls, achieves impact.

A group of girls listening to their teacher in a Rajasthan, India, village targeted by the Educate Girls development impact bond.

DIBs work much like social impact bonds, offering investors returns based on positive social outcomes. With DIBs, however, the outcome payer may be a donor such as a foundation or multilateral development agency, rather than a government, and the main goal is improve results in international development. The traditional donor-driven model of development funding rarely grants enough flexibility to service providers, requiring them to follow a set of pre-defined mandates. Among other benefits, DIBs and other results-based financing programs (also known as pay-for-success) introduce incentives that shift the focus away from activities and inputs and toward results and impact, giving service providers discretion and encouraging them to innovate.

The Educate Girls DIB was conceived in 2013 as a proof-of-concept of this idea. UBS Optimus Foundation agreed to invest upfront capital in Educate Girls, the Indian NGO implementing the program. Our organization, Instiglio, a nonprofit specializing in results-based financing, provided the technical expertise and advice to design the DIB, and is helping develop a system to track outcomes and implement course-corrections in real time.

Meanwhile, an independent evaluator, IDinsight, is conducting a randomized evaluation to assess learning gains and validate enrollment using school data. If the intervention has achieved the desired results when it ends in 2018, the Children’s Investment Fund Foundation—a UK-based philanthropic organization acting as the outcome payer—will pay UBS Optimus Foundation its initial investment, plus a return based on the intervention’s success rate. Success in enrolling out-of-school girls will account for 20 percent of the outcome payment, and improved achievement in English, Hindi, and math will correspond to 80 percent.

When we structured the DIB, we envisioned that it would open the door for a new development practice, potentially unlocking new funding streams to support better programs and positive social outcomes. But we were eager to see how it would play out in practice. Over the past year, we have seen it bring about incredible changes made possible by a toolset and culture that promotes data-driven decision-making, and a strong adaptive leadership style at Educate Girls that empowers staff to test new approaches.

Data-driven Decision-making

We discovered that the right incentives can stimulate innovation in programmatic design and delivery practices, encourage staff to focus on outcomes, and give them a sense of ownership in the intervention’s success.

With the support of UBS Optimus Foundation, we built real-time, mobile, data-collection systems that fed into visualization platforms; these helped train the Educate Girls teams in data analysis and extracted performance insights that could inform program management decisions. This regular tracking and communication of results and targets boosted the program staff’s motivation to make an impact, and created a culture more focused on performance outcomes. To encourage these trends, the DIB also introduced a performance bonus tied to the project’s outcomes for staff and volunteers.

Our mid-year assessment process offered a clear example of how Educate Girls’ shift toward surfacing challenges early and collectively solving problems with the aid of data could help performance. Data on learning outcomes revealed that children were not improving their literacy in math as quickly as expected. Students who had initially received higher math scores were on average learning only about half as much under the DIB as students with lower initial scores. Further analysis showed that volunteer teachers struggled to teach math at higher levels and as a result were over-focusing on Hindi, which they felt more comfortable teaching. Once the challenge became clear, so did the solution. Educate Girls helped teachers develop skills to teach math at higher levels, and provided clear guidelines on how to structure class lessons and allocate time equally among the three subjects. By the end of the year, students had achieved the biggest learning gains in math compared to other subjects.

Adaptive Leadership at Work

In the course of our partnership with Educate Girls, we also came to appreciate that certain pre-conditions at the organization—especially in leadership style—were essential to realizing the DIB’s potential.

The transition from being accountable for inputs to being accountable for results requires a steep learning curve for any service provider. The existing culture and leadership style at Educate Girls played a critical role in enabling this change. The organization’s leaders gave the field staff flexibility to innovate and spent hours each week problem-solving with them. They went beyond their supervisory role to become advisors, supporters, and enablers.

For example, while reaching an important decision about whether the organization should invest significant time in creating individual class plans to improve learning outcomes, the team turned to CEO Safeena Husain for a final approval. After providing her thoughts, Husain delegated the decision to the project manager, because the project manager would better understand whether the field staff would find it useful. Leaders’ openness to listening to perspectives from the field and giving agency to field staff have proved to be critical to achieving results in places like Rajasthan, where one village can have completely different needs from the next.

Safeena Hussein addressing members of Team Balika, a group of community mobilizers working in the field.

We also can’t underestimate the role of Vikram Solanki, Educate Girls’ most experienced program manager, who leads the DIB field work. Solanki role-models high performance standards and fosters a culture of learning, encouraging his team to regularly talk about challenges and failures, and asking hard questions that help them find important answers. He is also willing to let go of long-held beliefs when the data contradicts them, and manages the extraordinary pressure of delivering on the DIB’s ambitious targets while not losing sight of Educate Girls’ core mission. In one of our hallway conversations, he shared his fear that the DIB could reduce the NGO’s work to a simple exercise of accounting for results. The next day, he started his team meeting with four words: “Children first, results second.” These words embody his approach of keeping the beneficiaries at the center of every decision.

What’s Ahead for DIBs?

Even with all of Educate Girls’ underlying capacity, the DIB’s first year was very much an exercise in “flying the plane as we were building it.” We were pursuing ambitious DIB targets while mobilizing a team to think about, prioritize, test, and correct program design and delivery practices.

Obtaining accurate data posed another challenge. We had to rely on government data to estimate the size of the target population (children in or eligible for public schools). As it turned out, these statistics were off by approximately 10 percent, which impacted our ability to price outcomes fairly. We did not build in a larger baseline survey or verify government data, and as a result, we set overly ambitious learning targets. We have since revised our targets, but future DIB organizers can avoid this issue by conducting baseline surveys and making sure to verify government data.

There is legitimate concern with future DIBs that many service providers may not have the minimum capabilities to deliver on the high standards the DIB model demands. We have seen the positive power of incentives with Educate Girls, but strong incentives combined with lower capacity can easily lead to attempts to game the system, cream-skimming, and even harm to the beneficiaries.

Currently, development organizations have a tendency to want to implement an impact bond without really understanding whether the approach is appropriate, given the problem it aims to solve and the data it needs to collect. But the problem and context should always drive the effort. It may be necessary to invest in capacity-building as a preparatory phase to implementing a DIB—or, better yet, decide to implement simpler and less “high-stakes,” results-based financing models as a starting point. For example, in Morocco, a government agency was considering a SIB in a jobs training program, but few providers had the capacity to deliver results, and performance data was sparse and weak. Among other risks, we worried there would not be enough information to accurately price outcomes. We advised the government to implement a first round of funding with simple performance bonuses tied to verified results, on top of traditional funding. This would allow the government to develop an accurate understanding of the performance of the jobs training program and nudge providers to start building performance management systems, so that they could later consider more-sophisticated models of results-based financing.

We believe development impact bonds can effectively address massive, protracted, and messy global development issues, allowing investors and service providers to scale and continually improve proven programs. However, the fact that the Educate Girls DIB, which targets just 15,000 children, is currently the largest DIB in its reach illustrates just how far we are from having a functioning market for social outcomes.

For the results movement to succeed, those developing and implementing impact bonds must take a thoughtful, paced, and rigorous approach. Development organizations and governments need to be in a position to pay for success at scale, service providers need to improve their capacity to deliver impact at scale, and it must be possible to measure outcomes in a scalable manner. Reaching this point will require the mobilization of large amounts of capital and ample time to master the right methods. 

Read more stories by Avnish Gungadurdoss.