(Illustration by Eric Nyquist)
How do researchers evaluate a program’s effectiveness in providing accurate information to policy makers at scale? Although many experts see randomized controlled trials (RCTs) as the gold standard to assess whether an intervention works, field testing can provide added feedback critical for scaling, a new paper argues.
Lead author Francesco Agostinelli, an assistant professor of economics at the University of Pennsylvania, first learned about a nationwide educational program in Mexico—a partnership between the World Bank and CONAFE, Mexico’s government agency tasked with providing schooling services in rural and marginalized communities—from Matteo Bobba, a professor of economics at the Toulouse School of Economics in France. The program sends recent university graduates to disadvantaged rural communities for two years to serve as mentors in schools for instructors and students. The Mexican government assessed the program with an RCT, which found that the program had minimal impact.
The pair of scholars joined forces with Ciro Avitabile, a senior economist at the World Bank, who was in the field observing the CONAFE program. Working together, the three researchers ran two independent field experiments in Chiapas, Mexico’s poorest state, to determine whether a critical modification—connecting the mentors with students’ parents to get them involved in supporting their children’s education—might improve its performance.
“In the academic literature, there is great interest in understanding how much we can learn—and what we can learn—about the impacts or effectiveness of a particular program,” Agostinelli says. “In a field experiment, researchers can fix problems that arise. But when you move to the real world, the perfect conditions disappear. The challenge is figuring out how to get beyond the bubble of the RCT.”
In one experiment, the researchers applied the original version of the program. In the second, the researchers employed the newer “plus” version that had mentors also interact with the parents of students, including engaging them in one-to-one meetings in the home. The tests determined that parental engagement made the program successful.
In rural areas of Mexico, where one school typically serves children from several villages in a multigrade classroom, school closures are not uncommon. Teachers can be difficult to find, and when the number of enrolled students dips below six, educational services are automatically terminated. By studying the original mentoring program together with the “plus” version, the researchers identified that success depended on mitigating school closures, which parents could do by participating in local parental associations where they could vote to keep a school’s doors open. To avoid selection problems and maintain the purity of the experiment as much as possible, the researchers worked with a team to prevent schools from closing.
“There was a lot of subtle knowledge that mattered for the particular context we were studying,” Agostinelli says. “We had to understand the role parents played in the community and in the educational system. The changing situation we faced—again, which usually does not come up in the RCT—is that schools can close according to different decision makers.”
As long as schools remained open, the mentoring program was scalable, the researchers found, leading to higher test scores and levels of educational attainment, boosting the probability that children moved from elementary school to secondary school. In both versions of the CONAFE program, mentors helped local teachers by serving as substitutes and providing remedial education for students lagging behind. The program’s enhanced version, however, trained mentors to interact with parents, which made all the difference by encouraging them to join local committees where they could vote to reverse automatic school closures. The researchers’ findings contribute to a growing body of literature that underscores the critical role parents play in successful schooling.
“This paper, the result of a coherent research agenda, delivers important results both on the mechanisms of child development and on the design of scalable interventions that could foster development among children in disadvantaged circumstances,” says Orazio Attanasio, the Cowles Professor of Economics at Yale University. “The scalability of such interventions is one of the current big challenges in this area.”
Find the full study: “Enhancing Human Capital in Children: A Case Study on Scaling” by Francesco Agostinelli, Ciro Avitabile, and Matteo Bobba in Journal of Political Economy, vol. 133, no. 2, February 2025
Read more stories by Daniela Blei.