Don’t Let Conventional Measurement Wisdom Fragment Your Impact

Social impact bonds. Pay for success. Evidence-based funding. Investing in what works. To what do we owe this explosion of “pay-or-performance” funding models?

Most of the rhetoric is some variant of: “There are too many ineffective and inefficient nonprofits getting funding and not really having any impact.” While this may be too strong a statement, research clearly shows that many nonprofits lack the resources, expertise, and conviction to continuously measure and improve their programs. So, it is not a stretch to imagine that society would be better off if resources were directed to more effective nonprofits.

But a far greater problem lurks beneath the surface, and it has to do with whether all of the individual impacts nonprofits generate actually add up to something meaningful. A recent observation relayed to me by the leader of a community-based organization in Boston could not have summed it up better: “You know what, I’ve been doing this work for over ten years, and while we’re making some headway, the truth is we haven’t done enough to change the trajectories of the young people in our city.” 

The sad truth: Conventional wisdom on measurement perpetuates this status quo. It fragments impact into stages that can be evaluated, rather than encouraging organizations to aim for trajectory change. 

For example, suppose you run a summer sports program for middle school youth who are stable (physically, mentally, emotionally, etc.) but nonetheless disadvantaged and at risk of dropping out of high school. You measure students’ level of physical activity, involvement in crime, and substance abuse before and at the end of their time in the program, and the data you collect is promising. So you contact an evaluator. “We think our program is working, but we want to know for sure—what should we do?”

Conventional wisdom on performance measurement will take you down the path of one or more rigorous external evaluations to attribute outcomes to your specific program, to show that your program creates more benefits for society than it costs, and to show that it is replicable to other settings. And with this bullet-proof data in hand, voilà, you are now “evidence-based” and ready to attract the resources you need to dramatically scale.

While this pathway through evaluation seems reasonable at first glance, something is missing, and it relates to the youth leader’s observation cited earlier: We don’t have a clue whether this program is changing, or contributes to changing, the actual trajectories of the youth it serves. For some (those who would have committed serious crimes in the middle school years if they had not been in the program), it may have; but with a relatively stable target population, such cases are few. What about the rest? Once they enter high school, do they revert to pre-program levels of crime and substance abuse? Do they graduate, or do they end up dropping out of school at the same rate as they might have without the program?

“Not to worry,” the measurement gurus tell you, “You’re working on an important predictor of high-school graduation, so you can just use the literature to show your program is helping youth down this path.” But such advice makes the fairly big leap that other high-quality programs are addressing the more than dozen other critical predictors of high-school graduation for the specific youth in this program.

The fact is, far too often, we rely on a patchwork of programs, at best evidence-based on one or two dimensions, working with different youth in an uncoordinated fashion. Consider the 76 promising or proven interventions in the Promising Practices Network’s registry. Less than one-third are impacting more than one indicator of child well-being, making it highly unlikely that they are changing the ultimate destiny of any given youth.

At the end of the day, it comes down to accountability. More nonprofits need to explore what it means to hold their organizations accountable for meaningful change in the trajectory of a client, however they define it. Doing so may lead the organization to consider a more multi-service, client-centered approach (like Latin American Youth Center in Washington D.C.) or to collaborate with other organizations that provide the rest of the services needed (like the Strive Partnership in Cincinnati). Or it might lead to a commitment to measure long-term change, which could in turn lead to the discovery of the next great intervention (like Nurse-Family Partnership).

And therein lies the problem with “pay-for-performance” models. To the extent these funding mechanisms reward programs that generate single, short-term outcomes (e.g., keeping a job for six months, improving test scores in one grade), our society won’t really be any better off. Developing and proving an intervention that can produce short-term outcomes is rare and laudable, but impact could be dramatically amplified through a simultaneous (or subsequent) pursuit of longer-term success by those who have perfected these models. Instead of fragmenting impact into measurable bites, we need to purpose measurement to create solid pathways to better, ultimate outcomes for those we serve.

How are you using measurement to ensure your beneficiaries access the supports they need for long-term improvements?

Matthew Forti is the Performance Measurement Capability Area manager at the Bridgespan Group, an advisory firm to mission-driven leaders and organizations. Matt has supported several leading organizations in the design and implementation of performance measurement systems that promote continuous learning and improvement, the primary focus of his blog, Measuring to Improve. Matt is also the founding board chair of One Acre Fund, a nonprofit that assists over 50,000 smallholder farming families in East Africa to triple their crop yields.

Read more stories by Matthew Forti.