Measuring Impact: Keep It Clear and Simple

It’s a truism that what gets measured gets done. But in the case of international NGOs seeking to measure the effectiveness of their programs, the doing is often in the hands of local communities, and the funding that fuels the doing comes from donors on another continent. Keeping measures simple and highly meaningful to both donors and those in the local community can make all the difference in motivating stakeholders toward a shared vision of healthy lives and livelihoods.

Take for example World Vision’s experience in scaling up access to clean water in Africa. Three years ago, we changed our approach to ensuring that communities had access to clean water, sanitation, and hygiene (WASH), and set an aggressive goal to reach five million in need over five years. The effort would require a rapid expansion of the program, including building enough wells to help roughly 200,000 people annually in three countries of West Africa and helping five times that population per annum across ten countries. The new approach also required that we partner with five times the number of local communities—communities that would own, plan for, invest in and maintain the project.

Nothing could be more important for the health and welfare of children in the region, as contaminated water and poor hygiene and sanitation account for more than half of childhood deaths in sub-Saharan Africa. But to attract funders and to seriously engage local communities, World Vision needed to measure and communicate both the challenge and potential impact in ways that felt relevant to villagers, teams drilling the boreholes for new wells, and potential donors.

A starting point was to adopt a uniform measure of the need: access to clean water. This is something that each country in the program heretofore defined differently. We settled on the following definition in dialogue with communities: Access would mean having a protected clean water source available 12 months of the year within a 30 minute round-trip walk from a person's household. By this measure, only 45 percent of the people living in African communities where World Vision donors had longstanding investments actually had access to clean water. That left 15 million in need. These numbers alone sounded a wake-up call. Even more people lacked latrines and safe hygiene practices. By creating a simple metric that described the challenge, we were able to communicate the need to donors and set an aspiration for local communities: to reduce their daily trek for clean water to 15 minutes each way.

The next set of measures related to ensuring that clean water access could be sustained (a critical element of any improvement). Important measures to that end include forming community water committees that are responsible for the water point, establishing a process for collecting $1-$2 annually from individuals accessing water to save up for future repairs, and training 7-8 people nominated by the committee to perform routine maintenance. The committees must also ensure that villagers understand the importance of keeping water pure by using water containers with lids, building a protective apron around the pump, and keeping animals away. Beyond measuring these inputs, we also measure outputs—both the pace of drilling and their upkeep ten years out. One such study in Ghana indicated that 90 percent of boreholes drilled more than ten years ago are still operational; but it also found that the pacing of maintenance was sometimes erratic.

These simple measures of need and community commitment allowed us to grow funding, which in turn allowed us to hire the expertise we needed to grow activity. We used our measures to make a strong business case to existing donors—among them, David and Dana Dornsife, who have invested in our West Africa water program for decades. Entrepreneurs themselves, they pledged $35 million over five years to fund additional technical expertise, retain contractors, purchase equipment, and implement an ongoing system of measurement. The increased managerial capacity allowed us to flex our approach to operations so that we could hire contractors, identify local volunteers, and collaborate with government and private drillers to quintuple the pace of accessing clean water across ten countries. The Dornsifes observed that we had “the right people … in place to accomplish this scale-up, particularly at the senior management ranks in Africa and the US, “and their commitment attracted additional donors. Thanks to simple measures and multi-year commitments from donors, World Vision WASH teams could plan activities over several years for the first time and not be hamstrung by fluctuations in year-to-year budgets.

Finally, any project needs simple metrics of effectiveness and results. In scaling up WASH, we measure our effectiveness in reduced cost per family member accessing clean water. In the first two years of the program, we’ve seen this drop from $80 to $50, thanks to creating more productive water points and adapting drilling technology to the needs of the terrain—in some cases inexpensive, manual augers are doing the trick. At the same time, we measure our impact in reduced incidence of diarrhea and dysentery, as well as by the number of girls who are able to attend school because they aren’t spending hours fetching water. The incidence of diarrhea has dropped 70 percent, and we are hearing stories of more and more girls attending school, which we’ll quantify in our next independent assessment.

Measures can motivate. By last year, we had grown the rate of access to clean water, with training in hygiene and sanitation, five times, reaching 1 million people per year. As important, we are sharing results and methods across our network, so that measurement converts to broad-based learning.

Read more stories by Larry Probus.