The Randomista Revolt

In 1959, a doctor called David Sackett realized that much of what he’d been taught in medical school was wrong. Most decisions about treatment were based on someone’s opinion (“do this because I think it’s right”) or conventional wisdom (“do this because that’s how it’s always been done”). This meant that two doctors with differing opinions might hand out different advice. Sackett, working in Chicago, was troubled. What was he supposed to believe?

Sackett and other doctors went on to argue that medical practices should instead be based on evidence from research, such as randomized trials showing whether a drug actually works. This term “evidence-based medicine” was published in 1991 and quickly caught on, becoming the main way in which Western medicine is practiced today.  

Evidence-based medicine is part of a bigger movement, an “evidence revolution.” My book Beyond Belief: How Evidence Shows What Really Works, tells the story of how evidence has been embraced beyond medicine—in government policy, policing, conservation, management, international development, education, and even parenting—as well as the huge barriers it faces. It does that through the stories of the rebels and mavericks, like Sackett, who challenged how decisions were being made in their fields.—Helen Pearson

* * *

In 2001, Jon Baron started the Coalition for Evidence-Based Policy, a non-profit group in Washington DC dedicated to increasing the effectiveness of government with evidence. Inspired by medicine, Baron and his team wanted to see more social policies put through rigorous trials to show they worked, just like a drug. (This is part of the so-called ‘what works’ movement.) Baron and his team faced resistance — but they would bang away at the idea to anybody in Congress who would listen. They hoped that this spark of an idea would eventually land on tinder and someone in government would say, “they’re absolutely right — let’s test what works.”

Fortunately for Baron, government attitudes were slowly changing. In the 1990s, political leaders such as Bill Clinton and Tony Blair talked of solving social problems using the most effective methods rather than having fixed ideas about what those methods should be. President George W. Bush’s administration appointed some key champions of evidence-based policy, including Whitehurst. Baron was campaigning as economist Esther Duflo and other “randomistas” were pioneering the use of randomized trials of policies to combat poverty.

Momentum built further when Barack Obama took office in January 2009, and made clear he wanted to support programmes backed by evidence. “The question we ask today,” he said in his inauguration speech, “is not whether our government is too big or too small, but whether it works—whether it helps families find jobs at a decent wage, care they can afford, a retirement that is dignified. Where the answer is yes, we intend to move forward. Where the answer is no, programs will end.” The Obama administration was later called “the most evidence-based administration in history.”

Eventually, then, Baron’s spark landed and took. He found a small group of influential policymakers who supported evidence-based policy. The most important allies were at the Office of Management and Budget (OMB), the influential agency that oversees the president’s budget. Evidence is a very useful tool for those managing government spending because it helps make rational decisions about where to invest and where to cut. Robert Shea, associate director of the OMB during the Bush administration, and Peter Orszag, the first head of OMB under Obama, were important advocates for evidence-based policy, as were senior officials Robert Gordon and Kathy Stack.¹ The office started encouraging agencies to evaluate whether their programmes were working, and later launched an ‘evidence team’ dedicated to this work.

Baron’s vision of using legislation to drive the use of evidence was realised when the Obama administration implemented a series of programmes called the Obama evidence initiatives. A shining example was the Investing in Innovation (i3) Fund, run by the Department of Education.

Starting in 2010, researchers or educators who had developed an innovative way to help educate children—a tutoring or reading program, say—could apply for a grant in one of three tiers. If the program had promising but preliminary results, its developers could apply for the lowest-tier grant of up to $5 million to test it in early trials. If the program had already passed through some trials, it could be granted up to $30 million for testing in a randomised trial. If the program already had positive results from a randomised trial, then it could win a top-tier grant of up to $50 million to scale up the program and replicate the findings. (The size of these grants was reduced in later years.) Only by accumulating evidence of effectiveness could a program climb from one tier to the next and win a greater share of funding.

One success story from the Investing in Innovation Fund is the Knowledge is Power Program (KIPP), a network of charter schools that serves students in predominantly low-income and under-served communities. They follow a model of high expectations, skilled teachers, extended school days, social and emotional development and preparation for college. Already backed by positive results, including from a randomised trial, in 2010 the Department of Education awarded the KIPP model a mammoth top-tier grant of $50 million to scale up and further test the approach.

KIPP doubled the number of schools it was reaching, alongside two randomised trials in elementary and middle schools. These showed that KIPP schools boosted the average level of children’s achievement in reading and mathematics by 5–10 percentile points compared with a control group. And, because the evaluation involved two trials, several US states and two age groups, it looked to be a strong and replicated effect—although it’s not yet clear whether these gains will translate into life outcomes such as better earnings or health.² As the KIPP model accumulated evidence it won funding to expand further afield. The i3 evidence initiative, now called the Education Innovation and Research (EIR) program, was still going in 2024.

The Obama administration eventually ran six “tiered” evidence initiatives including ones that aimed to prevent teen pregnancy, promote child development, support under-served communities and boost career training. Another well-known program to benefit from them was the Nurse–Family Partnership, in which nurses support first-time mothers while they are pregnant and during their child’s first two years. The program was found to have lasting, positive effects—including on educational outcomes for children—in four out of five major randomised trials. This helped it secure hundreds of millions of dollars in one of the evidence initiatives to expand.

But to Baron’s disappointment, these success stories were rare. Following the 80% rule, the majority of social programmes had little or no effect in rigorous trials—even if they’d previously looked promising in smaller studies. Such disappointing results give fuel to critics, who question the need for expensive randomised trials when they’re unlikely to show a social program improves on the status quo. Others chafe at the insistence on randomised trials over other forms of research.

But Baron is unapologetic. The fact that most programmes fail to show a meaningful effect in rigorous trials is precisely why, in his view, they must be done: to identify the exceptional few that do produce important effects and warrant expansion. Any less stringent test fails to sort the wheat from the chaff. If you’re going to build a body of programmes that you can be confident will improve people’s lives, “there’s no alternative to going down this path and getting to strong replicated evidence from randomised trials,” he says.

The Obama evidence initiatives faced other challenges. Some social programmes were found to be effective in trials but were never taken up and expanded. Some of the evidence initiatives were later watered down, cut back or eliminated.

Nevertheless, the initiatives are considered a milestone in the history of evidence-based policy. They “are opening a new chapter in the generation and use of evidence by the federal government,” Baron and Haskins wrote in 2011. The events of the last few years, they said optimistically, “lead us to believe that the role of rigorous evidence in federal policymaking and program implementation is here to stay.”