Late last year, the White House and a number of former Obama administration officials began rolling out an apparently coordinated campaign to promote an “outcomes mindset” for government and the social sector. Unfortunately, while this agenda includes many positive recommendations, its central message is worrisome.
The outcomes-focused paradigm proposes leveraging data and new technology to achieve “21st-century solutions” to the nation's most pressing social needs. But in reality, outcomes-based policies and funding have been around for decades, and their history is far from universally positive.
A New Paradigm for Government?
At first glance, this new outcomes paradigm and its associated proposals seem encouraging. They suggest that government agencies focus on program outcomes—such as foster care placements, asthma reduction, and college graduation rates—and urge a number of promising policies, including increased investments in evidence, data, and place-based initiatives. (Former administration officials have fleshed out some of the ideas—particularly outcomes-based funding and building an outcomes marketplace—in various articles and reports, including a recent SSIR series.)
In some respects, outcomes-based financing is akin to pay-for-success, a strategy the administration has supported in various forms for several years. And taken as a whole, it seems like just the sort of legacy-cementing package that one might expect from an administration entering its final year in office.
The Disappointing Track Record of Outcomes-based Policies
Despite this compelling framing and the strength of many of the individual proposals, outcomes-oriented policies generally and outcomes-based financing in particular have been around for years, and their history has been surprisingly poor.
One of the earliest examples could be found at the Pentagon during the Kennedy and Johnson administrations, when Robert McNamara and his band of “whiz kids” from RAND developed performance metrics that produced a false sense of progress during the Vietnam War. McNamara's methods were briefly exported as performance-based budgeting to the rest of the federal government, before the administration later discarded it. The hubris of the era was aptly described in books like David Halberstam's The Best and the Brightest.
Paying for outcomes arguably got its start with the Job Training Partnership Act (JTPA) of 1982. This Reagan-era law established performance standards for job placement and earnings, and these metrics were subsequently incorporated into performance-based contracts with nonprofit workforce providers.
Unfortunately, later evaluations found that serving people who were easier to employ primarily drove improved results, where they existed. At its nadir, the Government Accountability Office accused the program of disparate treatment of women and minorities. According to experts involved in the program's national evaluation, “The JTPA standards did little to enhance the impact of the program on employment and earnings.”
Mathematica Policy Research found a similar pattern when it evaluated the Job Corps program. Its study rigorously analyzed the true impact of providers who were rated highly based on their employment outcomes. It found that while Job Corps increased participant earnings overall, the higher-rated providers seemed better only because they were working with more advantaged populations. Because payments and contract renewals were tied to these measures, they probably punished providers that worked with people who had greater needs and were harder to serve.
In health care, performance-based accountable care organizations established under the Affordable Care Act may seem promising, but it is too early to know how successful they will be. The administration is aggressively pushing value-based payments, but studies of their use in nursing homes and hospitals have shown little effect so far.
Meanwhile, the success of outcomes-focused police strategies like CompStat in New York City and CitiStat in Baltimore has been hotly debated. Some question whether they may have contributed to tensions between police and African American communities in those cities. (To its great credit, the Obama administration has shown tremendous sensitivity to these issues and the need to build community trust with its Police Data Initiative, but it is unclear how much impact it will have.)
The record has been similarly troubling in education, where the outcomes-oriented policies of No Child Left Behind were so badly implemented that it accomplished the rare feat of uniting the left and right in opposition. It was largely repealed in December, just as the administration’s new outcomes-focused paradigm was rolling out.
These examples might seem cherry-picked, but they aren’t. More often than not, programs that seem successful based on outcomes alone turn out to have little impact once subjected to rigorous analysis. Unsurprisingly, the history of tying payments to such questionable outcomes is no better, with negligible effects or worse reported across a wide range of initiatives and correspondingly few real successes that have been credibly validated by rigorous evaluation.
The reasons why outcomes-based policies fail become more obvious after reviewing the literature. The most commonly cited pitfalls include:
- External Influences: External factors that have little to do with the program itself, like varying local poverty and unemployment rates, often drive program outcomes. Such factors make it difficult to know what really caused an outcome, and can lead to inappropriately rewarding or punishing providers for factors beyond their control.
- Cream Skimming: One of the simplest ways to boost program outcomes is to work with populations that are easier to serve. Providers can do this on purpose, but they can also do it unintentionally, as can happen when they are working with motivated individuals who volunteer for a program or are serving a geographic location with higher income, greater education, and less poverty.
- Gaming and Fraud: Some program operators, when faced with difficult-to-achieve performance standards, may choose to game the numbers or engage in outright fraud. History suggests that such gaming is widespread in outcomes-based funding systems and most prevalent when the stakes are high.
- Tunnel Vision: Emphasizing or paying for certain outcomes can lead program operators to focus on those outcomes to the exclusion of others, no matter how important the others may be.
- Superficial and Short-lived Outcomes: Even when they are real, many outcomes can be short-lived and suffer fade-out effects over time.
Mere improvements in data technology won’t easily address issues like these. These pitfalls reflect the inherent limitations of outcomes themselves, which can’t reliably predict a program’s effectiveness on their own. We can usually only determine program impacts—the actual value-add of a program apart from any external forces or participant characteristics—through rigorous evaluations, ideally a randomized controlled trial or similarly credible design.
Unfortunately, outcomes and impact are often unrelated, which is why a program that seems to produce better outcomes may actually create no impact at all. Worse, sometimes they point in opposite directions, as can happen when a program works with harder-to-serve populations, resulting in seemingly worse outcomes, but higher value-add and greater impact.
The divergence of outcomes and impact is the primary reason why outcomes-based policies fail. We can trace nearly all of their shortcomings back to this simple idea.
Improving on the Past
This history would probably shock a casual observer; an increased government focus on outcomes might seem like refreshing common sense. Given the actual track record of outcomes-based policies and funding systems, however, it might be tempting to conclude that we should avoid them altogether.
This too would be a mistake. The administration is right: The status quo is unacceptable, and we must find ways to do better. Fortunately, although “21st-century solutions” to these challenges don’t lie just around the corner, there are still reasons to be cautiously hopeful.
One is the continued development of evidence-based policy. Although the origins of the evidence movement can also be traced back to the last century, it is only now beginning to get real traction. As the field advances and we learn more about which programs work and why, policymakers may become more adept at designing outcomes metrics that come closer to true impact.
Rigorously evaluated pay-for-success projects are another reason for optimism. Most of the recent pay-for-success initiatives in the United States have featured such evaluations. One that didn’t, an early learning program in Salt Lake County, drew fire after its outcomes-based results were called into question. A better approach might be one used in Santa Clara, California, which coupled its outcomes measures with a randomized controlled trial to validate the results.
The Santa Clara project is especially noteworthy, because it demonstrates the potential synergy between outcomes and impact. Outcomes-based approaches are simple, flexible, and scalable, but their performance is uncertain. Rigorous evaluations can measure impact, but they are usually costly, difficult, and slow. Together, they are more powerful than they are alone.
A final reason is more prosaic: Policymakers may become better at designing outcomes metrics, not because of any particular advances in technique or technology, but simply because they have already made the worst mistakes and are now poised to do better.
A dose of such realism, combined with a greater reliance on evidence and a willingness to learn from the past, could transform the administration’s focus on outcomes into an important step forward. By openly acknowledging the challenges and dangers, recognizing the difference between mere outcomes and true impact, and demonstrating how this time we will do better, the administration could show that what it’s really calling for is not just an outcomes mindset, but an Outcomes Mindset 2.0.