Government

The Promise and Peril of an “Outcomes Mindset”

If government is going to champion outcomes-based policies, let’s learn from our mistakes.

Late last year, the White House and a number of former Obama administration officials began rolling out an apparently coordinated campaign to promote an "outcomes mindset" for government and the social sector. Unfortunately, while this agenda includes many positive recommendations, its central message is worrisome.

The outcomes-focused paradigm proposes leveraging data and new technology to achieve "21st-century solutions" to the nation's most pressing social needs. But in reality, outcomes-based policies and funding have been around for decades, and their history is far from universally positive.

A New Paradigm for Government?

At first glance, this new outcomes paradigm and its associated proposals seem encouraging. They suggest that government agencies focus on program outcomes—such as foster care placements, asthma reduction, and college graduation rates—and urge a number of promising policies, including increased investments in evidence, data, and place-based initiatives. (Former administration officials have fleshed out some of the ideas—particularly outcomes-based funding and building an outcomes marketplace—in various articles and reports, including a recent SSIR series.)

In some respects, outcomes-based financing is akin to pay-for-success, a strategy the administration has supported in various forms for several years. And taken as a whole, it seems like just the sort of legacy-cementing package that one might expect from an administration entering its final year in office.

The Disappointing Track Record of Outcomes-based Policies

Despite this compelling framing and the strength of many of the individual proposals, outcomes-oriented policies generally and outcomes-based financing in particular have been around for years, and their history has been surprisingly poor.

One of the earliest examples could be found at the Pentagon during the Kennedy and Johnson administrations, when Robert McNamara and his band of "whiz kids" from RAND developed performance metrics that produced a false sense of progress during the Vietnam War. McNamara's methods were briefly exported as performance-based budgeting to the rest of the federal government, before the administration later discarded it. The hubris of the era was aptly described in books like David Halberstam's The Best and the Brightest.

Paying for outcomes arguably got its start with the Job Training Partnership Act (JTPA) of 1982. This Reagan-era law established performance standards for job placement and earnings, and these metrics were subsequently incorporated into performance-based contracts with nonprofit workforce providers.

Unfortunately, later evaluations found that serving people who were easier to employ primarily drove improved results, where they existed. At its nadir, the Government Accountability Office accused the program of disparate treatment of women and minorities. According to experts involved in the program's national evaluation, "The JTPA standards did little to enhance the impact of the program on employment and earnings."

Mathematica Policy Research found a similar pattern when it evaluated the Job Corps program. Its study rigorously analyzed the true impact of providers who were rated highly based on their employment outcomes. It found that while Job Corps increased participant earnings overall, the higher-rated providers seemed better only because they were working with more advantaged populations. Because payments and contract renewals were tied to these measures, they probably punished providers that worked with people who had greater needs and were harder to serve.

In health care, performance-based accountable care organizations established under the Affordable Care Act may seem promising, but it is too early to know how successful they will be. The administration is aggressively pushing value-based payments, but studies of their use in nursing homes and hospitals have shown little effect so far.

Meanwhile, the success of outcomes-focused police strategies like CompStat in New York City and CitiStat in Baltimore has been hotly debated. Some question whether they may have contributed to tensions between police and African American communities in those cities. (To its great credit, the Obama administration has shown tremendous sensitivity to these issues and the need to build community trust with its Police Data Initiative, but it is unclear how much impact it will have.)

The record has been similarly troubling in education, where the outcomes-oriented policies of No Child Left Behind were so badly implemented that it accomplished the rare feat of uniting the left and right in opposition. It was largely repealed in December, just as the administration’s new outcomes-focused paradigm was rolling out.

These examples might seem cherry-picked, but they aren’t. More often than not, programs that seem successful based on outcomes alone turn out to have little impact once subjected to rigorous analysis. Unsurprisingly, the history of tying payments to such questionable outcomes is no better, with negligible effects or worse reported across a wide range of initiatives and correspondingly few real successes that have been credibly validated by rigorous evaluation.

Common Pitfalls

The reasons why outcomes-based policies fail become more obvious after reviewing the literature. The most commonly cited pitfalls include:

  • External Influences: External factors that have little to do with the program itself, like varying local poverty and unemployment rates, often drive program outcomes. Such factors make it difficult to know what really caused an outcome, and can lead to inappropriately rewarding or punishing providers for factors beyond their control.
  • Cream Skimming: One of the simplest ways to boost program outcomes is to work with populations that are easier to serve. Providers can do this on purpose, but they can also do it unintentionally, as can happen when they are working with motivated individuals who volunteer for a program or are serving a geographic location with higher income, greater education, and less poverty.
  • Gaming and Fraud: Some program operators, when faced with difficult-to-achieve performance standards, may choose to game the numbers or engage in outright fraud. History suggests that such gaming is widespread in outcomes-based funding systems and most prevalent when the stakes are high.
  • Tunnel Vision: Emphasizing or paying for certain outcomes can lead program operators to focus on those outcomes to the exclusion of others, no matter how important the others may be.
  • Superficial and Short-lived Outcomes: Even when they are real, many outcomes can be short-lived and suffer fade-out effects over time.

Mere improvements in data technology won’t easily address issues like these. These pitfalls reflect the inherent limitations of outcomes themselves, which can’t reliably predict a program’s effectiveness on their own. We can usually only determine program impacts—the actual value-add of a program apart from any external forces or participant characteristics—through rigorous evaluations, ideally a randomized controlled trial or similarly credible design.

Unfortunately, outcomes and impact are often unrelated, which is why a program that seems to produce better outcomes may actually create no impact at all. Worse, sometimes they point in opposite directions, as can happen when a program works with harder-to-serve populations, resulting in seemingly worse outcomes, but higher value-add and greater impact.

The divergence of outcomes and impact is the primary reason why outcomes-based policies fail. We can trace nearly all of their shortcomings back to this simple idea.

Improving on the Past

This history would probably shock a casual observer; an increased government focus on outcomes might seem like refreshing common sense. Given the actual track record of outcomes-based policies and funding systems, however, it might be tempting to conclude that we should avoid them altogether.

This too would be a mistake. The administration is right: The status quo is unacceptable, and we must find ways to do better. Fortunately, although "21st-century solutions" to these challenges don’t lie just around the corner, there are still reasons to be cautiously hopeful.

One is the continued development of evidence-based policy. Although the origins of the evidence movement can also be traced back to the last century, it is only now beginning to get real traction. As the field advances and we learn more about which programs work and why, policymakers may become more adept at designing outcomes metrics that come closer to true impact.

Rigorously evaluated pay-for-success projects are another reason for optimism. Most of the recent pay-for-success initiatives in the United States have featured such evaluations. One that didn’t, an early learning program in Salt Lake County, drew fire after its outcomes-based results were called into question. A better approach might be one used in Santa Clara, California, which coupled its outcomes measures with a randomized controlled trial to validate the results.

The Santa Clara project is especially noteworthy, because it demonstrates the potential synergy between outcomes and impact. Outcomes-based approaches are simple, flexible, and scalable, but their performance is uncertain. Rigorous evaluations can measure impact, but they are usually costly, difficult, and slow. Together, they are more powerful than they are alone.

A final reason is more prosaic: Policymakers may become better at designing outcomes metrics, not because of any particular advances in technique or technology, but simply because they have already made the worst mistakes and are now poised to do better.

A dose of such realism, combined with a greater reliance on evidence and a willingness to learn from the past, could transform the administration’s focus on outcomes into an important step forward. By openly acknowledging the challenges and dangers, recognizing the difference between mere outcomes and true impact, and demonstrating how this time we will do better, the administration could show that what it’s really calling for is not just an outcomes mindset, but an Outcomes Mindset 2.0.

Tracker Pixel for Entry
 
 

COMMENTS

  •  Raul  De Arriz's avatar

    BY Raul De Arriz

    ON January 13, 2016 11:53 AM

    Incisive analysis, and excellent recommendations. Indeed Outcomes should be measured by their impact, and not only by the initial or self reported metrics, to ascertain effectiveness of programs.

  • Heidi De Wolf's avatar

    BY Heidi De Wolf

    ON January 14, 2016 08:32 AM

    Is the issue that we focus on outcomes from the wrong perspective? Improving quality of life from a professionals point of view differs dramatically from that of the person the outcome relates to. Also, I often say to professionals: ‘Are you doing for or doing with?’ in other words are they trying to deliver the outcome for the person, or are they helping to remove the barriers that stop the person from achieving their own outcomes. On a larger government scale however, government has to be clear about its purpose. Is government not there to help remove barriers to the dynamic culture and infrastructure healthy societies are organised around rather than getting involved with delivery of preventative and acute services?

  • BY Dr. Kim Dunphy

    ON January 28, 2016 06:29 PM

    Thankyou for this thoughtful and thought-provoking article. I concur with the authors that what is required is more reflective consideration of outcomes: what we are looking to achieve for whom and why, rather than abandonment of the idea of outcome focus. We must continue to improve how we conceptualise and measure change in a holistic way, considering all stakeholders, as well as improving understanding of cause and effect. In my field of the arts, two recent related publications are: Dunphy, K. (2015). A holistic framework of evaluation for arts engagement, in L. MacDowall, M. Badham, E. Blomkamp & K. Dunphy (Eds), Making Culture Count: the politics of cultural measurement, (pp. 243-263). London: Palgrave, that discusses outcomes of arts participation as part of holistic schema of change, and with reference to the range of stakeholders impacted; and TFCC (2016). Cultural and creative spillovers in Europe: Report on a preliminary evidence review,
    that investigates ‘spillover effects’, which seem to be indirect impacts of arts activities that are additional to direct impacts. Those authors report significant challenge in measurement of indirect changes, largely because of lack of clarity around causal mechanisms.

  • BY Laura Boyd

    ON February 8, 2016 09:44 AM

    Well done!

Leave a Comment

 
 
 
 
 

Please enter the word you see in the image below:

 

SSIR reserves the right to remove comments it deems offensive or inappropriate.