How Boosting Education Research Could Revolutionize US Schooling

Despite 15 years of real progress in raising US high school graduation rates, 17 percent of students are still not graduating with their class, and another 17 percent are graduating but are not prepared for postsecondary school. This means that for this third of students—most of whom are low-income or minority children—the education system is not working. This deficiency has deep consequences for their communities and for the nation, as most jobs that can support a family require a high school degree and some postsecondary schooling. This reality puts us at a crossroads: We can either strengthen the public-education system or begin to walk away from it toward private solutions.

Given that a vast majority of American children rely on public education to obtain a foundation for future success, strengthening the system is the smartest way forward. A little-noticed provision of the Every Student Succeeds Act (ESSA)—the new federal law that gives states more freedom to set their own agendas for improving the education outcomes of low-income and minority children—presents a strong path for strengthening public schooling.

Under ESSA, states must take an evidence-based approach to turning around the bottom five percent of schools, and work with local districts to craft and execute evidence-backed interventions. Despite moves by Congress to scrap the federal accountability rules devised to help states implement ESSA, the language of the law carefully defines “strong evidence”—the standard about which states and districts should feel most confident, according to ESSA—as at least one randomized control trial (RCT) with a positive, statistically significant impact on student outcomes.

Tying school improvement efforts to a tightly defined standard of evidence is a major change for education reform, which has traditionally relied on educator judgment and experience. ESSA is an attempt to move education from a craft to an evidence-based enterprise. A similar effort began in medicine a century ago and is largely responsible for the subsequent sharp increase in medicine quality. Improvements of a similar magnitude are necessary to make public education work for all students.

By linking the use of evidence-based practice to the mandatory reform of low-performing schools, ESSA also aims to address one of the biggest challenges to the emerging evidence-based movement in education: Evidence only matters if we use it to change practice. Until now, schools and school districts have had little incentive to discover the evidence base and how it can address their needs.

Yet one big challenge remains: Such evidence is scarce and difficult to secure.

The Education Evidence Hurdle

There are several reasons for this. For starters, only about 12 percent of RCTs evaluating education programs produce positive results; the vast majority result in weak or neutral results. In a way, this is encouraging, since it demonstrates that many conventional practices—and even innovative ideas—do not actually improve student outcomes and that there is room to make educational practices more effective. But the process of gathering enough meaningful evidence to provide educators clear guidance on improving low-performing schools requires that we make enough attempts to learn something and have tolerance for a lot of misses.

That brings us to the current lack of investment in education research. For the benefit of their balance sheets, corporate giants like Google and Walmart regularly conduct thousands of randomized studies on the viability of products. Imagine a world in which we invested with equal intensity in determining what works for young people’s education. Today, only tiny fractions of local, state, federal, and philanthropic education dollars are used to build the evidence base. But if even one penny of every federal dollar invested in K-12 education went toward evidence gathering, it would represent an order of magnitude increase—the scale required for revolutionary improvement.

Studying real, functioning schools inevitably brings additional hurdles—principals leave, funding gets cut, new mandates appear. Such volatility underscores the need for multiple evaluations of major approaches and interventions. I know this because I am experiencing it now: The 2016-17 school year is the last of a seven-year, multi-district randomized control trial on Diplomas Now—a school improvement and student-support model I helped devise for the highest-needs secondary schools.

On the Front Lines of Evidence Gathering

Diplomas Now is designed as a big intervention for a big problem. In the United States, the neediest students have become concentrated in a subset of schools not designed or prepared for intense educational challenges. As a result, these high schools become “dropout factories.” Many of their students fall off track to graduation during the middle grades—long before they even arrive in high school. The ultimate goal of Diplomas Now is to achieve a substantial increase in graduation rates in these challenged high schools, as well as foster a successful middle-grades experience that leads to success in the ninth grade—the “make-or-break” transition year for students attending high-poverty schools.

Created in 2008, Diplomas Now is a partnership of three national organizations: Talent Development Secondary (TDS), a Johns Hopkins program that develops reform models for underperforming schools; City Year, which puts teams of AmeriCorps members—typically recent college grads—in schools to provide one-on-one mentorship and tutoring to students who require extra support; and Communities In Schools, which provides aid to the most vulnerable students confronting difficult circumstances outside of school.

A City Year corps member tutors a student after school at Clinton Middle School in Los Angeles. City Year is one of three Diplomas Now partner organizations. (Photo by Elliot Haney)

The teams work closely with teachers and administrators to identify students falling off track to graduation based on their attendance, behavior, and course performance, and then work to change their path. Prior research has shown that these three “early-warning signs” are highly predictive indicators that a student is on the path to dropping out. When we help students attend school every day, focus in class, and complete their assignments, these warning signs disappear, boosting a student’s odds of graduating in high-poverty environments from 25 to 75 percent.

In 2010, the US Department of Education awarded Diplomas Now a $30 million Investing in Innovation Fund (i3) validation grant to conduct one of the largest RCTs to date of a comprehensive school-reform and student-support model. To gather solid evidence, we needed a sample of at least 60 schools across 11 school districts—30 (randomly selected) to implement the Diplomas Now model and 30 to engage in other ambitious reform attempts.

In the schools selected for both groups in the study, educators faced intense challenges. A third of the students were chronically absent, a third had been suspended, a third were held back a grade, and barely more than a third had grade-level math and reading skills. All of the schools required comprehensive and multi-tiered interventions focused on both building educator capacity and addressing student needs.

Since the study was an effort to validate a promising but not yet scientifically proven model, it was necessary to use the grant funds to subsidize the cost to garner school districts’ participation. The Diplomas Now model costs about $500 per student—about half the amount a high-poverty school receives in Title I funding (federal dollars intended to help schools provide support to students living in poverty). But that modest per-pupil cost adds up quickly in a study involving 40,000 students.

We also had to follow students over time, as they progressed through middle school into high school and through high school until graduation. Thus the intervention and the need to subsidize it were multi-year commitments. We initially expected to subsidize about a third of the cost, with schools and districts covering the other two-thirds. The Great Recession of 2008, however, flipped that expectation; the critical years of the study included the only two years in the past four decades when total federal, state, and local education funding declined.

To cover this heightened price tag, we needed to separately raise an amount equal to the federal i3 grant, so we turned to the philanthropic sector to secure additional funds, including $11 million from the PepsiCo Foundation. Keeping the study alive was an exhausting and hair-raising experience with several down-to-the-wire moments. In fact, getting across the finish line required a $3 million loan from the Johns Hopkins School of Education, which Diplomas Now is still paying off.

The good news is that—despite trying times—the project’s independent evaluator, MDRC, found that schools were able to rapidly implement a core component of the Diplomas Now model: early-warning systems. Diplomas Now staffers successfully worked with school leaders to monitor early-warning indicator data and implement intervention plans for struggling students.

MDRC also issued a report in June 2016 showing that across 30 high-needs middle and high schools, Diplomas Now had a statistically significant impact on increasing the number of students who ended sixth and ninth grade with no early-warning indicators. More important than the magnitude of the interim impact (which was a 3.6 percentage-point increase after just one year of a multi-year treatment) is the fact that this was the first empirical verification that it is possible to change students’ trajectories as late as sixth and ninth grade.

Students at Chicago Talent Development Charter High School prepare to walk across the graduation stage. (Photo courtesy of Diplomas Now)

These students reversed course on a path toward dropping out and started on a path aimed at graduating. Further, modest initial gains may become quite large if they continue year after year, as students progress from sixth grade to graduation. This is why it was so essential to study impacts over multiple years.

The Cost of Evidence and Working Together to Find a Solution

For me, MDRC’s early impact results represent a decade-long quest to learn how to best support students and teachers in the most challenged schools. The time, effort, and sacrifices necessary to achieve results reflect the great challenge of gathering strong evidence in education. People—and organizations—can easily become overstretched, and few people fully understand the cost.

TDS—the partner organization I co-direct—devoted most of its resources to keeping the Diplomas Now study alive, at the expense of other priorities. But when we look at the impact, the burden was worth it. Because of the study, we not only know which schools will produce the next generation of dropouts and which students likely will not make it without meaningful interventions, but also we have an approach based on “strong evidence” that we can continually improve to change students’ lives.

Support for an evidence-based approach to education is growing. National advocacy efforts, such as those pioneered by Results for America—an organization that helps leaders adopt an evidence-based approach to policymaking—are having an impact. Moreover, some core funders of an evidence-based approach to education reform, such as the Laura and John Arnold Foundation, the Schusterman Family Foundation, the Ford Foundation, and the Edna McConnell Clark Foundation, have stepped forward to catalyze both evidence gathering and scaling evidence-based efforts.

ESSA’s focus on evidence-based strategies for the lowest-performing schools in the United States has the potential to boost the demand for evidence-based efforts—yet this approach is still in the fragile early stages of development. Our study will likely produce a number of additional leads that deserve large RCTs of their own, but due to lack of funding, they will likely go un- or understudied—and that’s a shame.

Still, the nearly 1,000 high schools where graduation is not the norm demand solutions. Much good can be done with the evidence scientific studies produce—particularly for the most vulnerable students. More broadly, the public-education system requires that we produce additional evidence to work for the third of students it currently fails.

Lawmakers, state and local officials, the corporate and philanthropic communities, and researchers must unite to determine how to effectively channel more resources into generating sound evidence of what works, for whom, and under what conditions. They then need to support the development of interventions and programs built on that evidence and implement them in the neediest places. When the nation’s most vulnerable kids are counting on it, what could be more important?

Read more stories by Robert Balfanz.