Beyond Randomized Controlled Trials

J-PAL Director Abhijit Banerjee, left, attending "From Evidence to Policy: How Rigorous Evidence Can Help Inform Better Policy in MENA" in Dubai. (Photo courtesy of Community Jameel)

Last year’s Nobel Prize in Economics, awarded to Abhijit Banerjee, Esther Duflo, and Michael Kremer for “their experimental approach to alleviating global poverty,” is a testament to the innovations they spurred in development economics over the past two decades. But the laureates have repeatedly emphasized that the use of randomized controlled trials (RCTs) in development economics was part of a broader movement of integrating innovation and evidence into social policy and practice. Equally important to the research was seeding an evidence-to-policy (E2P) community, in parallel, that helped and will help the laureates’ research, as the awarding committee put it, “dramatically improve our ability to fight poverty in practice.”

The Abdul Latif Jameel Poverty Action Lab (J-PAL), which Abhijit and Esther co-founded in 2003, and of which Michael is a long-time affiliate, now includes almost 500 researchers and more than 400 staff in many countries and is one of the key nodes in a network of E2P organizations and individuals working to advance the use of data and evidence to better understand underlying policy challenges, and to design, pilot, evaluate, and scale innovative solutions for these problems. J-PAL, along with partner organizations such as Innovations for Poverty Action (IPA), have built the infrastructure to fund and implement RCTs, to conduct policy outreach based on insights from this research, and to build the capacity of stakeholders to apply this evidence to policymaking. By leveraging a research infrastructure that includes more than 30 J-PAL and IPA offices around the world, a large and growing network of affiliated researchers have partnered with social innovators in NGOs and governments to rigorously evaluate the impact of promising anti-poverty programs through almost 1,000 RCTs in more than 50 countries and in almost all sectors of development, including agriculture, climate, education, firms, gender, governance, and labor.

More than 400 million people have been reached by programs that were found to be effective by researchers in J-PAL’s network and were then scaled up by our partner organizations. A stand-alone policy outreach team of J-PAL staff, spread across almost a dozen countries, not only summarizes and synthesizes research evidence into actionable policy lessons but builds long-term partnerships with local governments and NGOs to scale up effective programs. Thousands of researchers and policymakers worldwide now commission RCTs or use the evidence from them to inform their decisions. This has required dedicated training teams who have created a suite of in-person and, increasingly, online courses to build the capacity of thousands of decision-makers in governments, NGOs, foundations, and other development organizations to conduct RCTs and interpret their results.

Each of these three foundational pillars of J-PAL—field research, policy outreach, and capacity building—required flexibility, a deep understanding of local contexts, and close collaborations. Above all, a long series of innovations were required not just in the methodology and the econometrics behind RCTs, but also in: data collection in the field (for example, innovative survey design to elicit accurate information and maximize responses), experimental design (how to minimize spillovers and attrition), transparency (a trial registry, replications, and data publications), scalability (evidence synthesis, replications, a generalizability framework, monitoring systems, and cost collection), and capacity building (the use of custom trainings and online courses).

Sadly, evidence-informed decision-making is still the exception rather than the rule. Daunting challenges persist in poverty, equity, and increasingly in climate change. In this article, we dive deeper into the key innovations that were behind the success of J-PAL, the learnings from some of these, and new innovations that we believe will be central for us to take on the challenges facing us—while recognizing, of course, that there are many more dimensions on which dozens of E2P organizations are working and innovating every day.

Research Innovations

A unique aspect of J-PAL’s role within the E2P community is our focus on methodology. As noted by the Nobel Prize committee, RCTs are a particularly rigorous method of measuring the impact of social programs and determining the causal mechanisms to explain why and how policies do or do not work. But relevant research in the field includes more than just RCTs. Qualitative work is essential to design good randomized evaluations, and descriptive research, administrative data, and continuous feedback from participants are all essential in interpreting and applying insights from RCTs. Over the past two decades, J-PAL made a series of efforts to help develop a robust research infrastructure by establishing regional offices around the world, supporting hundreds of large-scale field experiments, and improving research transparency and quality.

For example, in response to the challenge of publication bias in the social sciences—the fact that studies showing positive results are more likely to be published—J-PAL helped to establish the norm that new RCTs must be recorded in the American Economic Association’s registry for randomized controlled trials, which J-PAL helped to launch in 2012 and continues to support. During the research design phase, J-PAL encourages researchers to design studies to ensure that the data they collect will be reliable and accurate: reducing spillovers by choosing the most optimal level of randomization (for example, at the entire school level rather than at the classroom level) and minimizing attrition (through stratification). To improve the quality of data during research implementation, J-PAL and IPA helped develop standards for frequent revisiting of data by “back-check” surveyors who reconduct surveys in a sample of households. Additional innovations to improve the quality of data included moving surveys themselves from paper forms to higher-quality digital data collection with tablets, as well as moving beyond surveys to measure complex outcomes through games, experimental vignettes, and implicit association tests in the field. Post-data collection, J-PAL and IPA have developed robust processes to ensure the confidentiality of participants and the transparent and timely publication of data. J-PAL also provides a replication service to affiliates in which RAs use the original data and rewrite the code used for analysis in order to confirm that the results can be confirmed.

In addition to helping set standards for how randomized evaluations are conducted, J-PAL has also created special funding initiatives to identify pressing policy areas for research and encourage a coherent research agenda. In areas including agricultural technology adoption, crime and violence, education technology and post-primary education, employment, financial inclusion, health care delivery, governance, jobs, and social inclusion, J-PAL has worked with policymakers and researchers to identify gaps in knowledge and run competitive funds to test promising innovations to help fill these gaps.

J-PAL 2.0

In its post-Nobel phase, one of J-PAL’s priorities is to unleash the treasure troves of big digital data in the hands of governments, nonprofits, and private firms. Primary data collection is by far the most time-, money-, and labor-intensive component of the vast majority of experiments that evaluate social policies. Randomized evaluations have been constrained by simple numbers: Some questions are just too big or expensive to answer. Leveraging administrative data has the potential to dramatically expand the types of questions we can ask and the experiments we can run, as well as implement quicker, less expensive, larger, and more reliable RCTs, an invaluable opportunity to scale up evidence-informed policymaking massively without dramatically increasing evaluation budgets.

Although administrative data hasn’t always been of the highest quality, recent advances have significantly increased the reliability and accuracy of GPS coordinates, biometrics, and digital methods of collection. But despite good intentions, many implementers—governments, businesses, and big NGOs—aren’t currently using the data they already collect on program participants and outcomes to improve anti-poverty programs and policies. This may be because they aren’t aware of its potential, don’t have the in-house technical capacity necessary to create use and privacy guidelines or analyze the data, or don’t have established partnerships with researchers who can collaborate to design innovative programs and run rigorous experiments to determine which are the most impactful. 

At J-PAL, we are leveraging this opportunity through a new global research initiative we are calling the “Innovations in Data and Experiments for Action” Initiative (IDEA). IDEA supports implementers to make their administrative data accessible, analyze it to improve decision-making, and partner with researchers in using this data to design innovative programs, evaluate impact through RCTs, and scale up successful ideas. IDEA will also build the capacity of governments and NGOs to conduct these types of activities with their own data in the future.

J-PAL is not alone in acknowledging the significance of administrative data. In the US, groups such as the Actionable Intelligence for Social Policy Initiative and The Lab at D.C. are utilizing administrative data at the state and local government level to better understand the impact of policies on the communities they serve. Internationally, organizations such as Development Gateway and Global Partnership for Sustainable Development Data are working with government agencies to improve the access and use of their administrative data for evidence-based policymaking. Even organizations that historically do not conduct randomized evaluations, such as the International Monetary Fund, recognize the important role of utilizing administrative data to improve statistical quality and support government agencies in their policymaking.

The second big scope for innovations in research for J-PAL and the wider impact evaluation and evidence-informed policymaking communities is building more expansive and ambitious research agendas in particular sectoral areas. One of the most pressing areas for new research is climate change, which threatens to undo decades of progress and whose consequences are already being felt through heat-related mortality, deteriorating food security, and extreme weather events. People living in poverty are the most vulnerable to dramatic changes in temperatures that affect agricultural livelihoods and make it more difficult, expensive, or dangerous to live in locations affected by sea level change, natural disasters, and extreme temperatures. Without tackling climate change, making progress in solving many other problems will be significantly less impactful—for example, if significant parts of Rio de Janeiro are underwater due to rising ocean levels, if the air quality is too dangerous to go to school in Delhi, if fires disrupt travel and infrastructure in Australia, or if it’s simply too hot to go to work in Saudi Arabia.

With one foot in research and another in policy, the E2P community is well-placed to tackle this issue. First, researchers must test technologies from the lab in the field. Measuring the real-world impact of new technologies is crucial to understanding their externalities and the political economy factors that will affect their potential to scale. Second, researchers need to test strategies to improve the effectiveness of climate policy and regulation, as some of the countries that will be hardest hit by the consequences of climate change also grapple with limited state capacity to design and enforce regulation. Finally, researchers must test interventions that affect behaviors in order to reduce individual burdens and encourage certain behaviors.

An example of this work in practice comes from Gujarat, India, where making environmental auditors more independent improved the accuracy of pollution audit reports, leading, in turn, industrial plants to pollute less. Based on the results of a randomized evaluation by J-PAL affiliates, the Gujarat Pollution Control Board (GPCB) reformed its environmental auditing system in 2015, issuing new guidelines that require random assignment of environmental auditors. Members of the research team continue to work closely with officials in Gujarat and other Indian states on environmental policy design and evaluation.

Refugee populations and migration is another vital area for innovation in experimental research to inform policymaking. The United Nations estimates that there are now more than 70 million people worldwide who have been forcibly displaced from their homes, the highest figure recorded in history of refugees, asylum seekers, and internally displaced people. Many experience compounding vulnerabilities: over half of refugees globally are children, millions are stateless and have therefore been denied legal protections, and many come from or live in countries with serious safety and security concerns.

It is vital that the E2P community double down on efforts to learn how to address challenges to education, safety, and employment for populations who can often be displaced for decades. At J-PAL, for example, our European Social Inclusion Initiative evaluates programs and policies to foster the social inclusion of migrants and refugees in Europe. The first round of funded projects includes evaluations of programs that, for example, foster social ties to promote the integration of migrants in Sweden, build social cohesion between Turkish and Syrian schoolchildren, and counteract the social exclusion of immigrants in Finland. IPA also recently launched a Humanitarian and Forced Displacement Initiative to improve the lives of those who have been forcibly displaced and the communities that host them through policy-relevant research.

Other areas where innovative research questions are now being asked include gender, the private sector, taxation, and urban infrastructure. Development economics no longer only studies questions of gender through disaggregated data, but goes much further to ask questions such as: how we can most effectively address gender disparities and inequality at scale, whether existing development programs are closing the gender gap in human development, how gender dynamics in families and society affect the impact of these programs, and how we can best measure changes in areas like agency and empowerment. Researchers such as Lori Beaman, Pascaline Dupas, Erica Field, Seema Jayachandran, Rohini Pande, Simone Schaner, and many others are now designing innovative studies to answer these questions.

Although the private sector may seem to fall outside the bounds of poverty research, there is scope for experimental research. Income differences between countries can be explained largely by differences in firms’ productivity, for example, such that identifying policies that stimulate productivity growth or enable high-productivity firms to grow can have important consequences for poverty alleviation and social mobility. Bringing rigorous evidence to conversations about how to generate firm growth—and how this growth affects workers, their families, and the broader economy—can help inform and improve these policies. Researchers addressing these questions include David Atkin, Nicholas Bloom, Dean Karlan, and Antoinette Schoar, among many others. Recognizing the importance of emerging research in these areas, J-PAL formally launched new Gender and Firms sectors last year.

New data allows us to ask new questions, but the challenges faced by the world’s poor also continue to evolve. Researchers and E2P organizations must be flexible enough address these new problems and collaboratively find new solutions.

Policy Innovations

Randomized evaluations are valuable in precisely estimating the impact of a particular social program in a particular place and time, but causal impact alone is not always sufficient to improve policies and change lives. Programs and policies shouldn’t be scaled up simply because they meet a particular bar of effectiveness. Evidence-informed policymaking, particularly when adapting evidence-backed social programs from one context to another, requires a deep understanding of the global evidence, knowledge of the local context and local systems, a window of opportunity, political will for change, funding, and sufficient implementation capacity.

Policymakers face multiple constraints to using evidence to inform their decisions. On the “supply of evidence” side, these include challenges to accessing relevant research, the fact that impact evaluations rarely include detailed program cost data (budget constraints are as important for a policymaker as impact), and the difficulty of determining actionable take-aways when evaluations of similar programs show different results. Furthermore, will a program that was successful in one location achieve similar impacts in an entirely new context? On the side of “demand for evidence” from potential users, we also need to appreciate that policymakers need to make decisions on short timeframes and must factor in a range of political, administrative, and budgetary considerations. Evidence is, at best, just one part of the decision-making process. In working to translate experimental evidence into action, a key lesson we have learned is to work on both the supply and demand sides.

We have also learned to apply judgment about the types of tested policies and programs which should be scaled in different contexts and to build coalitions and long-term partnerships for change. J-PAL’s evolution toward basing policy staff in developing countries and working closely with local policymakers to apply insights from research happened gradually. We started by trying to make it easier for policymakers to access relevant experimental evidence on program effectiveness, the “supply problem.” We expanded the pool of policy-relevant research by launching new initiatives to spur projects in areas such as governance and urban services, requiring researchers to collect cost data in addition to impact data. To make the findings of randomized evaluations more accessible to a non-academic audience, the policy group began to produce two-page synopses of the relevant policy questions, the contextual factors, the implementation details, and the results of the programs being tested. These are written in a non-technical style and are available in a searchable database on the J-PAL website. To highlight evaluations that addressed particularly relevant questions for policymakers, we created glossy, expanded summaries called briefcases. To provide a bigger picture on the implications emerging across multiple studies in one thematic area, we developed syntheses called bulletins, which begin to reconcile conflicting results and draw coherent and actionable policy lessons. Many bulletins include cost-effectiveness analyses calculating the ratio of the impact each program achieves to the cost incurred to achieve that impact. Over time, to make these syntheses even more digestible and timely for policymakers, J-PAL developed shorter, two- or three-page policy insight products to highlight the emerging consensus on policy lessons from J-PAL academic co-chairs. To help policymakers appropriately apply these lessons to their contexts, we developed a framework on how evidence and interventions can generalize from one place to another.

However, we realized that even all of these collective efforts were insufficient to fully leverage the potential treasure troves of research—we needed to equally address the “demand” side of evidence-informed policymaking. To complement the policy teams developing these products at J-PAL’s global office at MIT, we hired policy staff based at J-PAL’s regional offices located within local universities around the world, whose job is to work directly with national, regional, and local policymakers to help them understand and apply experimental research.

Perhaps J-PAL’s most important learning centers on how to effectively create partnerships to scale tested innovations with governments. To many people, governments—particularly developing governments—seem ineffective, corrupt, or slow to innovate. However, governments are the pre-eminent players in global poverty and development, with unrivaled responsibility for social programs and incomparable reach and scale, and big gains in welfare can be catalyzed through smart partnerships. Over the last 20 years, 400 million people around the world have been reached by programs that were scaled up after they were shown to be effective through evaluations by J-PAL researchers. The vast majority of those scale-ups were led by governments.

We have learned three key lessons in how to build partnerships with governments to scale evidence-backed innovations:

1. “Globally informed, locally grounded” policy staff can identify and leverage policy windows. J-PAL policy staff based in regional and country offices or embedded in government departments have been critical in identifying windows of opportunity for evidence to inform decisions, as well as finding “evidence champions” within governments who can help make that happen. In Zambia, for example, frequent visits by the J-PAL Africa policy staff to understand the priorities of the Ministry of Education led to an opportunity to contextualize and scale up Teaching at the Right Level (TaRL), an approach pioneered by the Indian NGO Pratham and developed iteratively through multiple J-PAL evaluations led by Banerjee and Duflo. Through these interactions, J-PAL staff learned that Zambia was wrestling with low learning levels in primary schools and was looking for ideas to help students catch up after falling behind.

2. E2P organizations can support governments in understanding and adapting tested ideas to their context. We need to make it as easy as possible for governments to understand the mechanisms of what made policies work in other contexts and whether and how those programs could apply to their setting. In the Zambia case, we helped the Ministry of Education understand the different evaluated models of TaRL and the essential local conditions and implementation details to making the program work. We traveled with them on a “learning journey” to India to see how Pratham and Indian governments implemented the program. When they were convinced, we invited Pratham to Zambia to help them develop their own materials and models.

3. Coalitions are critical to supporting scale and sustainability. Even after contextualizing and piloting, governments have requested and benefited from support in integrating the model into their systems. This includes developing monitoring systems (including improving the government’s own administrative data), ensuring fidelity to core components of the original model, fundraising, and scale-up planning. E2P organizations such as J-PAL have helped develop coalitions of NGOs, researchers, and funders to support governments during this stage—achieving together what no organization could accomplish independently. In Zambia, the ministry leads implementation with J-PAL, Pratham, and the Flemish NGO VVOB providing implementation and monitoring and assistance from a group of funders led by USAID. The model is now improving literacy and numeracy for children in 1,800 schools across the country. With support from Co-Impact, this work is being extended to other African countries with the aim of reaching three million students over the next five years.

How will J-PAL apply this learning about catalyzing evidence for policy action from the last decade? First, we are doubling down on scaling through building multi-stakeholder partnerships with governments through a competitive fund. J-PAL’s Innovation in Government Initiative (IGI) works with governments to adapt, pilot, and scale evidence-informed innovations that have the potential to improve the lives of millions of people living in poverty in low- and middle-income countries. We are running IGI as a competitive fund to which J-PAL regional offices and partners, in collaboration with governments and researchers, can apply for support to help seed evidence-backed innovations for scale. In particular, the initiative is promoting the use of technology and data-enabled systems to reduce the costs of program delivery and monitoring, implementation science to adapt and pressure-test different models before scaling, and the systematic collection of cost data.

A second big policy theme of “J-PAL 2.0” will be to better leverage policy windows, the openings when partners are more open to informing decisions with evidence. J-PAL plans to prioritize more nimble, flexible, and fast support to partners and strategic organizations by sharing evidence, tested policies and programs that may be most effective in different contexts, and the critical mechanisms and details to making them work, as well as curating discussions between the researchers and implementers. We envision these opportunities may arise at specific times when organizations are revisiting their strategies or entering new multi-year planning cycles. For example, Ben Olken, J-PAL Co-Director, and J-PAL policy staff shared global evidence and frameworks for how to leverage philanthropy to improve governance with the William and Flora Hewlett Foundation as an input to its strategic planning. On April 16^th, Abhijit will present to 50 Finance Ministers during the World Bank’s spring meetings to share tested ideas for improving human capital through education, health, and social protection.

Finally, a third theme for the next decade of policy outreach at J-PAL will be to translate these most promising collaborations with policymakers into durable, long-term partnerships. Long-term partnerships—with other E2P organizations such as IPA, governments such as Peru’s Ministry of Education and the Indian state government of Tamil Nadu, NGOs such as Pratham, and donors such as the United Kingdom Department for International Development or the Bill & Melinda Gates Foundation—have been essential in contributing to a movement of evidence-informed policymaking. Many of the scale-up examples cited above were made possible by the trust and mutual understanding that arises through long-term partnerships.

Capacity Building and Diversity Innovation

To empower policymakers and funders with the tools to incorporate evidence into their work, J-PAL and many E2P organizations such as the World Bank and the Center for Effective Global Action (CEGA) have prioritized capacity-building efforts. J-PAL helps to develop the capacity of researchers who produce evidence and the policymakers and donors who use it. We have done this through a range of activities, from in-person training programs to university-level open online courses.

Over the past two decades, in addition to internal training for research associates, J-PAL created a suite of training courses, including our flagship training event: Evaluating Social Programs, a five-day executive education program that provides hands-on training to policymakers on randomized evaluations and becoming better commissioners and users of evidence. These open-enrollment courses are complemented by a range of custom in-person courses for partners such as the European Commission, the Indian Administrative and Economic Services, and Unicef. Since 2003, we have reached more than 8,500 participants.

What’s Next?

As we look to the future, three interrelated areas will more effectively increase the capacity of our partners and spur more evidence-informed decision-making. First, we will leverage the potential of online platforms to reach more people; second, we will integrate these courses into government civil service trainings; and third, we will help change the field by working with a greater diversity of researchers from different backgrounds.

In its early days, J-PAL relied heavily on in-person courses to train staff and partners. However, online platforms have the potential to dramatically increase access to this knowledge, such as the online MicroMasters program in Data, Economics, and Development Policy that we offer in partnership with MIT Economics. Though not the first standalone online courses on development, this program was the first of its kind to offer a fully online program grouping together a series of courses on policy and poverty to enable participants to earn an official MicroMasters credential: through five online courses and in-person exams, the program equips learners with the practical skills and theoretical knowledge needed to tackle pressing challenges in social policy. To increase access, the course is affordable: the cost varies depending on the annual household income of the learner, from $100 to $1,000 per course, and can be audited for free if the learner does not wish to sit for the certificate exam. The courses are offered several times a year, and learners can take anywhere from one to five courses at a time. The pricing structure, timing and pacing flexibility, quality, and online nature of the course allow for high participation worldwide: To date, more than 10,000 learners from 180 countries have signed up for a course, and more than 1,500 have earned an official certificate of course completion.

The second future priority for J-PAL is the lifelong and tailored learning of civil servants and government officials. Unfortunately, civil servants in many countries receive either very little training (concentrated around when they are hired) or receive training at very discrete steps in their career (for example, at five-year increments). Rather than conducting a series of one-off custom trainings with government departments, we will leverage online learning modules we are developing to prioritize long-term, institutional collaborations with government civil service programs. For example, under its partnership with the J-PAL South Asia regional office, the Government of India’s Department of Personnel and Training is now offering reimbursements for eligible staff to complete any of the MicroMasters courses. 67 civil servants signed up for classes in the pilot phase, and 38 of these have completed a course and passed the final in-person exam so far. Our expectation is that thousands more will join over the next few years. We are now working to expand the number of course offerings to dozens of options across numerous government departments with support from the Bill and Melinda Gates Foundation.

A third important priority related to capacity building is working with and strengthening local researchers in developing countries as part of a broader effort to diversify the worldwide researcher pipeline. Economics researchers often come from developed countries and elite institutions. This is true in the field generally as well as within the J-PAL affiliate network specifically. In addition to issues of equity and opportunity, this does a disservice to our wider goals of evidence-informed policy and poverty alleviation, as local researchers offer unique insights and perspectives on the challenges and potential solutions to poverty-related issues grounded in their knowledge of local context.

At J-PAL, the highly accessible MicroMasters courses are an important component of efforts to diversity research networks and invest in developing local capacity. Another example comes from the J-PAL Digital Identification and Finance Initiative, hosted by J-PAL Africa. The initiative’s director, Tanvneet Suri, and initiative staff have established pilot funding for African researchers, are integrating local researchers into the initiative as lead investigators and co-authors, and are providing targeted mentorship from senior affiliated researchers in our network. Other initiatives to better integrate researchers from developing countries include PhD fellowship programs, academic conference scholarships, and “matchmaking” efforts through other research initiatives to match local researchers with evaluation opportunities.

Gender diversity within the field of economics is also a priority, as women remain underrepresented in many aspects of the field. J-PAL recently established a Gender Working Group to examine internal issues concerning gender, recently held implicit bias training for staff, and is actively discussing ways to include more women in our network of research affiliates.

Conclusion

As the scale of challenges facing those living in poverty evolves to encompass new threats from rising sea levels and temperatures, new magnitudes of migration and displacement, and new realities in dense urban centers, the way that policymakers and researchers ask and answer questions must shift as well. The potential role of evidence to improve social policy has never been more important. To help face these challenges, J-PAL and the broader E2P community will continue to push innovations in research, policy influence, and capacity building. The Nobel Prize’s recognition of the work of evidence-informed decision-making is a major boon to the broader movement and will help us to more ambitiously push for change and to increase the role of research in reducing poverty.

Read more stories by Iqbal Dhaliwal, John Floretta & Sam Friedlander.