(Illustration by Øivind Hovland)
Many people who work in international development are familiar with the story of the PlayPump. Developed in South Africa, the merry-go-round-like machine, connected to a water pump, promised to provide both safe drinking water to rural communities and recreation to schoolchildren. The idea, which included a business model in which billboard ads on the water tanks would help to cover maintenance costs, attracted major international funding and visions of massive scaling in the early 2000s.
But several problems surfaced: The technology was effective only when an abundance of high-quality groundwater was present close to the surface, but often the underlying problem was water scarcity. The more remote the installation, the less reliable the maintenance, so many PlayPumps sat idle for want of needed repairs. In communities with fewer children, it often fell to the older women of the village to push the merry-go-round, an effort far more demanding than using traditional hand pumps—which had been replaced without consultation. Even after significant problems surfaced, plans to install thousands of PlayPumps across Africa continued. To their credit, many of those who promoted and funded the hoped-for scaling up of PlayPumps eventually recognized the problems and sought to share lessons from the experience. But these lessons were acted on only after significant (and avoidable) investment of time, energy, and funding.
Why was this “solution” promoted—and funded—so heavily before it had been thoroughly evaluated in multiple contexts? Why did widespread implementation continue even after evidence of serious problems surfaced? More broadly, if the international development community has been committed to evidence-based practice for a long time, why does the flow of evidence into practice not happen more frequently?
Our team of researchers at the Yale School of Management, with support from the William and Flora Hewlett Foundation, set out to understand why scenarios like the PlayPump are so common even in the most well-intentioned international development efforts. We started with a basic acknowledgment: Integrating evidence into practice is hard. Our effort, the Evidence in Practice (EinP) project, sought to understand the barriers to incorporating evidence into policy and practice in international development work, and to identify practices that illustrate more effective ways of doing so.
A variety of actors—including development practitioners (those responsible for designing and executing development programs, such as government agencies, NGOs, and implementation partners), academics, philanthropists, and impact investors—have grappled with this question of how to identify, support, and implement effective solutions. Their efforts to understand what constitutes evidence, how to apply it to a particular context, and the implications for each party’s roles have led to vigorous and often contentious debate.
Our research, conducted from January 2016 through January 2018, began with current relationships among development actors and their implications for the integration of evidence into development policy and practice. We conducted eight in-depth case studies in four countries—Ghana, India, Mexico, and South Africa—as well as extensive interviews with development experts around the world to answer several questions: What constitutes evidence? How, when, and why are decisions made about how to integrate evidence into practice? What do best practices have in common? What might a more effective model of evidence integration look like? Our work documented how long-term collaborations, based on inclusive negotiations that ensure genuine integration of multiple perspectives, are crucial for integrating evidence into practice. We draw on four of our case studies in this article.
Our research revealed a number of structural barriers to integrating evidence into practice. Governments and NGOs face scrutiny about the timely and reliable execution of their interventions. They are also evaluated on how efficiently and judiciously they deploy resources for an approved implementation plan. As a result, they often lack the time, resources, incentives, or skill sets necessary to generate their own evidence or to seek, interpret, and adapt evidence that others generate. They are also bound by strict timelines—such as donor budgetary cycles and electoral cycles—and program rules, which limit their flexibility to adapt to learning. Others, such as academics, have the ability and incentives to generate rigorous evidence but often lack the contextual understanding or the incentives to transform it into implementable formats.
Yet all these barriers also represent opportunities: Different types of stakeholders have their own assets (such as information or capital), strengths (such as academic legitimacy or decision-making power), and needs (such as publishable results or mandated responsibilities). They can help each other through mutually beneficial interactions in which assets and strengths are marshaled and needs met. We call these assets, strengths, and needs “currencies,” because they serve as media of exchange that allow different stakeholders to trade value and mitigate or circumvent structural barriers to integrating evidence.
The most robust examples of this type of exchange emerged from a process of negotiated collaboration: Communication among stakeholders clarified what each party could give and what each needed in return for the collaboration to succeed. This understanding set the groundwork for the parties to determine how to design, deliver, and share the emerging results of an intervention to ensure that researchers, policymakers, implementers, and others remained committed to the work. Such a negotiation can require significant compromise from all parties but is not zero-sum. The differences in what each stakeholder offers to and requires from collaboration allow for creative exchanges that benefit all participants—and the end-clients that all are seeking to serve.
Negotiated collaboration is unfortunately rare. Far more common is a transactional approach, in which stakeholders focus narrowly on traditional definitions of their own goals, roles, and expectations; rely on a fragmented and siloed understanding of the challenges; and misunderstand the perspectives and needs of other stakeholders. These transactional approaches neglect the opportunities that an exchange of currencies presents. When exchanged in service of solving shared problems, complementary stakeholder currencies create conduits not only for integrating existing evidence into practice but also for ongoing generation of evidence. In addition, the negotiated collaborations required to establish an exchange between stakeholders enhance trust and foster a culture of learning, which can render programs and relationships more robust and resilient. In other words, they “pay it forward” by investing in long-term partnerships that set the stage for the next cycle of learning.
Programa Primer Empleo
Let us turn to our case studies to examine these ideas in more detail. Our first, Programa Primer Empleo (PPE), was a Mexican federal job program launched in 2007. Given its uncontroversial objective—to partner with industry to create jobs for young workers—it had the potential to mobilize a broad coalition of stakeholders to discover solutions backed by evidence. It also had a strong political mandate: President Felipe Calderón won the 2006 election on a job creation platform. To deliver on this promise, members of Calderón’s transition team designed a nationwide program that would start immediately after his inauguration. Its centerpiece was a subsidy on social security taxes for Mexican firms that hired additional workers. In the 2007 federal budget, the program was allocated $3 billion Mexican pesos ($275 million).
While the program’s designers were convinced that they had incorporated all essential perspectives, Mexican companies did not believe that they had been consulted. The program included complex and intrusive rules that, while crafted with the legitimate goal of preventing misuse of funds, these companies interpreted as a sign of mistrust. Moreover, while PPE granted a partial, temporary subsidy, companies were expected to absorb the long-term financial and legal risks of hiring individuals with no prior experience in the formal labor market. More broadly, the program neglected relevant academic research on the limits and lack of demonstrable impact of generalized (rather than focused) subsidies in stimulating employment.
Policymakers also failed to learn from the private sector why it had not been hiring recent graduates—essential information for designing a high-impact program. By not engaging representatives of the private sector in an up-front collaboration, Calderón’s administration failed to gather important insights into what it needed to elicit support and missed a crucial opportunity to build a broad coalition that would boost the program’s prospects. This failure to integrate the multiple skills and perspectives needed for a successful program continued throughout the life of PPE.
Once launched, PPE was quickly rolled out at a national scale, which required the detailed incorporation into law of the program’s incentives, budgetary allocations, operating structures, and yearly targets. This large-scale, rapid rollout thus created a rigid structure that was extremely difficult to change even after policymakers realized that some of the program’s fundamental assumptions proved false. Professional civil servants in charge of program execution almost immediately identified shortcomings but were unable to do anything about them because of the legal mandate to strictly follow the program’s operating rules. Even the program’s ongoing evaluations, which clearly indicated that PPE was not meeting its goals, were not sufficient to alter its course.
By PPE’s conclusion in November 2012, it had met only 13 percent of the government’s employee registration target and 3.2 percent of the employers’ enrollment target. Because the Calderón administration prioritized the rapid rollout of PPE, it made no effort up front to develop sustained partnerships—with the private sector, with academics, or with the program’s evaluators—that could have informed the design and refinement of the initiative. PPE was designed for rapid implementation, not for learning. The program did not cultivate potentially beneficial “currencies of exchange” between stakeholders—such as linking the government’s social goals and financial investment with the expertise and shared goals of the private sector—at its inception. Once PPE was off and running at the national level and did not yield expected results, the program’s large size and scope, reflected in its operating rules and in federal budget laws, made it impossible to change.
CALIE
By contrast, the Collaborative Analysis of Labor Intervention Effectiveness (CALIE) was an innovative collaboration between policymakers and researchers that grappled with the critical issue of unemployment in South Africa. In 2011, the country’s unemployment rate was 26 percent, and youth unemployment (ages 18 to 35) was particularly dire, at 37 percent.
Seizing on the high-priority status that the federal government assigned to the issue, the Africa office of the Abdul Latif Jameel Poverty Action Lab (J-PAL), a leading development research organization, raised funds and sought proposals for collaborative projects between academics and policymakers on labor policy. Through a series of workshops and focus groups that convened government stakeholders and interested researchers from South Africa and beyond, J-PAL secured more than a dozen proposals from government-
researcher teams, including the winning proposal, submitted by two professors at the University of Cape Town and Stellenbosch University, who would become the principal investigators on the project.
The South African Department of Labor (DoL) was the central government unit tasked with reducing unemployment. One of the DoL’s unemployment-reduction programs, offered through labor centers in each province, provided counseling services to job seekers to connect them with firms that were hiring. Using the latest research on labor market dynamics, CALIE sought first to test whether “skills signaling,” such as letters of reference, could help low-income job seekers improve their employment prospects. At the design stage, researchers and policymakers focused on finding a common ground that would interest researchers, support the South African government’s mandates, and fit within the government’s operational and budgetary realities. They identified not only what was relatively well known about labor market interventions, but also critical questions that were less understood but that could have profound implications for any intervention. By negotiating their shared priorities during a collaborative (and complex) design phase, the stakeholders agreed on what they needed to learn. They were also reassured that their involvement was worth the effort and that the results would have the greatest chance of being adopted.
This design process required significant compromise from all stakeholder groups and hinged on their mutual interest in the shared problem. By explicitly addressing questions about what each group offered, required, and could compromise on, the design process identified a means of developing evidence that would be valuable for everyone involved. For example, researchers negotiated with the DoL to fit the study within the department’s resource limitations, incentive structures, and existing workflows. In exchange, the DoL agreed to randomization and data-collection protocols, which are unusual for government agencies. The partners saw these compromises as necessary not only to generate timely, relevant, and useful evidence but also to ensure a smooth, readily scalable implementation for the DoL. The collaborative process enabled them to establish a relationship of trust that would also lay the groundwork for further research.
A pilot of the intervention kicked off in 2012, and the results at the end of 2013 were positive enough to motivate the stakeholders to launch a larger-scale experiment to validate the results and also test a second intervention: an “action plan” that would help job seekers map out their employment-seeking activities. The results of this second experiment were also encouraging. Each party ultimately got what it needed from the collaboration while ensuring that the project had generated rigorous evidence and applied it to a pressing public issue.
The Limitations of the Linear Model
PPE and CALIE underscore the ways in which the generation and use of evidence can either be sidelined or serve as the basis for collaborative reforms in policy and practice. At the heart of both cases is the issue of how the various types of stakeholders in international development—namely, researchers, intermediaries, policymakers, and implementers—interact and share knowledge with one another.
We have found it useful to think of the flow of evidence into policy and practice as an “ecosystem” in which a set of archetypical stakeholder groups—including researchers, intermediaries, policymakers, implementers, funders, and beneficiaries—interacts. While not a perfect description—some organizations fall within more than one stakeholder group, and individuals often shift across stakeholder groups or play roles that span categories—this framework can help unpack the roles, incentives, and dynamics that characterize the complex relationship between “evidence” and “practice.”
Our research (which included more than 225 interviews) found that actors typically conceptualize interactions with other stakeholders as a linear process of translating academic evidence into public policy and development practice, which treats evidence as a form of abstract, generalizable knowledge that researchers produce. According to this model, researchers then pass evidence to intermediaries that (ideally) translate it into forms that policymakers and implementers can use.
A number of structural barriers impede this flow, however, starting with fundamental differences in what each group considers “evidence.” Participants perceive each step in the process as the domain of a particular stakeholder type—a view that accentuates the independence of each step, rather than the connections between them. These barriers also accentuate the predominance of each group’s norms and incentive systems, not their potential alignment. For example, evidence generation—conventionally the purview of researchers—is dominated by the structures and incentives of academic institutions, which differ significantly from those of government policymakers or implementers. These sharp distinctions in different stakeholders’ established roles and incentives reinforce the perceived need for a sequential approach to evidence “translation.”
Comparing the PPE and CALIE cases reveals important differences that suggest some of the critical limitations of the linear model. The primary difference is the level and type of collaboration between researchers and policymakers exhibited in CALIE, which allowed for the integration of evidence generation into evolving practice. The collaboration began by identifying a shared area of interest and codeveloping a dynamic research-in-policy program that addressed a salient problem for both researchers and policymakers. CALIE created a collaborative structure for identifying a pressing policy problem and for identifying researchers who were interested in that problem. CALIE extended this process to incorporate policymakers’ feedback as the research project was designed, and established a government intervention intended for adaptability and learning. Beyond building valuable goodwill, this approach allowed researchers to consider interventions within the scope of the government’s existing resources, structures, and processes. Framing the work as a government project and making adjustments that accommodated the government’s needs also developed valuable trust among researchers, policymakers, and implementers, which allowed the project to scale as positive preliminary evidence emerged.
This negotiated collaboration set the stage for how evidence that was relevant to policy concerns could be generated and integrated into current operations, cultivating buy-in among relevant policymakers and implementers as the evidence was generated. This collaboration suggests that incorporating evidence into practice is less a linear translation and more an integrated approach of evidence generation and application. As the CALIE case shows, this shift requires a change in mind-sets among the various participating stakeholders and the creation of structures that further adaptive learning. For example, the project developed cycles of interaction among the stakeholders to define the problem, develop potential solutions, and identify the assets and constraints relevant to each group in pursuing those solutions. In contrast with service delivery structures focused exclusively on rapid and efficient execution (which imply that all necessary knowledge is already in hand), this approach acknowledges limitations in current knowledge and values learning and adaptation. This approach created structures that allowed people across stakeholder groups to act on their desire to collaborate, despite existing constraints, and demonstrated how this integrative approach can generate rigorous evidence and integrate it into policy and practice.
Negotiating 'Currencies of Exchange'
These experiences suggest an alternative way of thinking about integrating evidence into practice. When stakeholders collaborate to solve a shared problem, they can negotiate a project structure that enables each group to provide its unique contributions to evidence generation and integration. Negotiated collaboration among stakeholders can make explicit the incentives and constraints that the various actors face, as well as the compromises and adaptations that may be required to maximize the likelihood of integrating evidence into practice.
Because stakeholders possess different kinds of currencies, an exchange of value can take place between stakeholder groups, providing a basis for meaningful negotiation. For example, researchers can offer their ability to frame questions and design methods to answer them, as well as the legitimacy of their work, in exchange for access to government data that will enable their research; policymakers are willing to share their data with researchers in exchange for the legitimacy that comes with rigorously produced evidence. Under current practice, such an exchange is desired but rarely discussed openly. Our research suggests that explicit acknowledgment and negotiation of potential common ground can help to establish and maintain relationships of mutual value, which are essential for the integration of evidence to occur.
This kind of negotiated collaboration has several other strengths. It increases trust among stakeholder groups, because it requires open dialogue and deeper understanding of exactly what each group can offer the others. It also recognizes that currently available evidence is often incomplete and that the generation of relevant new evidence can be integrated into program design. Implementation can then garner further insights for ongoing learning and iterative improvements, rather than being rooted in strict adherence to preestablished procedures. Altogether, the initiatives that proved most successful and resilient used this exchange of currencies.
CALIE came about because J-PAL seized on the opportunity to align its research efforts with the issue of youth unemployment, a stated priority of the South African government. A collaborative process then led to the design of rigorous research and a road map for how to integrate emerging evidence into policy and practice. By conducting a thorough evaluation by eminent researchers, J-PAL could improve program design and legitimize the government’s initiative. The government, in turn, was committed to reducing systemic unemployment in the country and saw that by supporting J-PAL’s collection and analysis of data, it could improve policy. Each actor could contribute its unique expertise to serving the shared goal of addressing the underlying issue.
The willingness of each group to compromise, where possible, on the project’s design, governance, and assignment of “credit” for success, increased the likelihood that relevant, timely, and useful evidence was generated and integrated into a program. In CALIE, J-PAL and the principal investigators negotiated with the DoL from the outset on their research design to ensure that the research question was relevant and timely for the government, so that the potential impact of the project through direct benefits and usefulness to both researchers and policymakers was maximized. J-PAL always described CALIE as a government project, crediting the DoL for the work. J-PAL also adjusted some elements of its research design to accommodate the government’s need for quick turnaround in analyzing and publishing interim results.
Recognizing specific areas of compatibility and complementarity is essential to the identification of currencies that stakeholders can exchange as a basis for negotiated collaboration. Often, such currencies are readily available to one stakeholder (in JPAL’s case, rigorous evaluation conducted by world-class researchers) and can provide tremendous value to another stakeholder (such as by granting external legitimacy to the government’s intervention).
Our research pointed to several pillars of currencies of exchange, such as the unique resources, needs, and priorities of individual stakeholders. In particular, we have identified six primary stakeholder attributes—time cycles, target audiences, typical incentives, resources, nonnegotiable needs, and evidence priorities—that contribute to the development of currencies of exchange across stakeholders. (See “Currencies of Exchange” on pages 36-7.) For each stakeholder type and attribute, we have included the research findings that emerged as most common or significant. For example, while researchers schedule their time based on publishing, tenure, and grant timelines, intermediaries structure their work primarily by identifying windows in other stakeholders’ time cycles that they deem the best opportunity for translating evidence. In turn, electoral cycles and political events govern policymakers. Examining features such as these not only helps to clarify the value that a particular stakeholder can offer others but also highlights opportunities for mutually beneficial exchanges.
However, as the PPE case shows, stakeholders often wind up not identifying, much less exchanging, their respective currencies, even when doing so would lead to mutually beneficial outcomes. Barriers arise largely because established incentives and norms within each stakeholder group have the unintended consequence of rendering each group more isolated, rather than more collaborative. The exigencies of political pressure kept the Calderón administration from seeking input in the design process from the private sector, on which the success of the program would hinge, and from looking for current academic evidence of similar policy initiatives. Once PPE launched, the rigidity of government regulations prevented emerging evidence of the program’s inherent flaws from being incorporated into midcourse corrections.
Comparing PPE and CALIE illuminates some of the barriers in the current models for incorporating evidence into practice, as well as examples of encouraging methods. In particular, these cases show that negotiated collaboration, rooted in mutual understanding of the incentives and constraints of each stakeholder, is pivotal both at the outset and throughout program implementation. When such currencies of exchange are created, a new model of evidence-based collaboration can emerge, one where program structures are designed for flexibility and learning. This alternative model creates a collaborative structure in which the diverse stakeholders participate and have their needs and contributions recognized, funding and project management mechanisms enable transparent and efficient coordination, and evidence and practice become integrated.
These trends were not unique to CALIE and PPE. In every successful context throughout our eight case studies, the systematic generation and use of evidence grew out of an ongoing process of collaboration and negotiation among stakeholders. The most effective way to develop collaborative relationships that lead to the uptake and generation of evidence is the explicit acknowledgment and negotiation of potential common ground, which establishes and maintains relationships of mutual value while also recognizing and respecting stakeholders’ nonnegotiables.
As we sought to identify common principles and requirements that allowed the various stakeholder types to engage in full-fledged negotiated collaboration, we recognized that developing a currency of exchange begins with the deceptively simple process of stakeholders’ honestly identifying their individual needs, resources, and constraints. Next, they must identify not only what is needed and who possesses it, but also how the collaboration needs to be structured so that all stakeholders will be willing to enter into a learning collaboration sustained by the mutual exchange of currencies. Two of our other case studies illustrate these points. The first shows a complex, successful collaboration, while the second highlights the limitations of not engaging in a more robust exchange of currencies.
Negotiated Collaboration in Action: Mexico’s Progresa
Developing currencies of exchange in the context of a negotiated collaboration can require substantial time and effort, but a champion can accelerate the process. In the case of Mexico’s Progresa, President Ernesto Zedillo’s career path from academia to politics gave him a more nuanced understanding of the needs, resources, and constraints of both worlds, which he brought to bear on the design of this landmark, evidence-based policy initiative.
Launched in 1997, Progresa (Make Progress) was a targeted, conditional cash-transfer program that marked a notable departure from existing approaches to social policy. A conditional cash-transfer program is one that offers subsidies to beneficiaries only if they engage in certain agreed-upon behaviors, such as sending children to school or making regular visits to doctors. In 1996, roughly two-thirds of Mexico’s federal subsidies for nutritional health programs were untargeted, generalized price subsidies. More than three-fourths of these subsidies went to urban areas, leaving rural areas—where extreme poverty was more prevalent—underserved. Food subsidies were expensive, did not reach the poor, and were routinely abused as tools of political manipulation—for example, when politicians exchanged subsidies for votes. The country’s 1995 economic crisis increased the urgency for more effective support for the poor, but within the added constraints of a reduced budget.
Zedillo, who held a PhD in economics, started his term with an appreciation of how rigorous evidence should inform policy. He was familiar with the body of academic literature that supported moving antipoverty policy away from generalized subsidies. Once in office, Zedillo adeptly traded his newfound form of currency—the ability to scale a national, evidence-based program—to recruit two leading economists to join his government: Dr. José Gómez de León, as chair of the National Council of Population, and Dr. Santiago Levy, as deputy minister of finance. Zedillo’s priority was to achieve tangible outcomes for beneficiaries; his political credibility depended on it. In return, these academics turned policymakers were given a platform to rigorously evaluate whether a conditional cash-transfer program that addressed the intergenerational transmission of poverty could be more effective and robust than the traditional food-subsidy programs.
Working with other stakeholders—including other government agencies, academic institutions, municipal and state governments, and international development institutions—Zedillo, Gómez de León, and Levy embedded evidence generation and integration into the structure of Progresa. They designed the program’s rollout to include a set of pilots and hired an independent research institute to conduct a randomized controlled trial. Even though this was the administration’s flagship program, national rollout was not planned until the third year of the administration, to ensure gradual learning of relevant concepts before scaling up.
However, while this approach held the promise of longer-term reputational payoff for Zedillo, it also created short-term political challenges: Critics, including some members of the press, thought it was outrageous for the government to conduct experiments with social programs and arbitrarily deny access to some households. Other policymakers, who were proponents of generalized subsidies that could be powerful political tools, pushed back hard. But Zedillo’s team did not buckle; they viewed the uproar as an additional reason to embed rigorous and transparent evaluation of the program’s impact into its design to insulate the program from political jockeying.
Progresa’s champions showed a commitment to furthering social impact and to serving research and political goals when they decided to prioritize those shared goals over self-interest. For instance, by Mexico’s midterm elections in July 1997, Progresa was fully ready to be implemented—with planning and funding in place. But the Zedillo administration decided against launching the program before the elections to avoid suspicion of politicizing it.
The independent evaluation of Progresa that the International Food Policy Research Institute (IFPRI) conducted found that the conditional cash-transfer program achieved positive educational outcomes for children. It also found that directing the subsidies to women enhanced the effects of the program without creating feared consequences, such as domestic violence. Zedillo’s team began to scale up the program soon thereafter; beneficiary households increased fivefold by the end of 1998, and the program reached almost 2.5 million households by the end of 2000. Perhaps most important, impact evaluation data from Progresa convinced Zedillo’s presidential successor, Vicente Fox, to maintain the program.
The Transactional Approach: Ghana’s Graduating the Ultra Poor Program
Despite its myriad benefits, the negotiated collaboration that Progresa exhibits is more the exception than the norm. Several of our case studies demonstrated the predominance of the linear approach to evidence and its costs. One of them, Graduating the Ultra Poor (GUP) in Ghana, illustrates how well-intentioned efforts to strengthen evidence can miss significant opportunities to affect practice when those efforts don’t incorporate relevant stakeholders from the start.
The GUP project was a randomized controlled trial (RCT) and pilot program conducted between 2010 and 2013. The GUP program, in conjunction with nine other programs around the world, sought to test the effectiveness of the “graduation” approach originally developed by BRAC in Bangladesh. The graduation approach seeks to break the cycle of poverty for the extremely poor by providing a carefully sequenced set of time-bound services that combine social protection, livelihoods, and financial services. Innovations for Poverty Action (IPA), a US-based research and policy nonprofit dedicated to addressing global poverty, designed the Ghana program and coimplemented it with Presbyterian Agricultural Services (PAS), a local NGO.
GUP was based on and contributed to rigorous evidence about a novel and immensely successful intervention to help alleviate poverty among the ultra-poor. But the GUP project’s specific mandate and singular focus on strengthening the global body of evidence missed the opportunity to help PAS and Ghanaian policymakers codify the evidence and learning from GUP for potential scaling up in Ghana. The decision to focus on a transactional approach to generating rigorous evidence, rather than a negotiated collaboration that sought to understand what each stakeholder—researchers, implementers, and policymakers—could offer and gain from the process, missed an opportunity for broader adoption of the graduation approach in Ghana.
For this project, IPA focused almost exclusively on using the GUP RCT to generate evidence about the spread of the graduation approach to other contexts. IPA’s primary audiences were the global funders supporting the growth of the graduation approach and other researchers and intermediaries looking to refine their understanding of its generalizable aspects. By contrast, PAS, whose structure was shaped by evidence-based implementation as a result of working with IPA, was focused almost entirely on rigorously implementing the program within Ghana. As an implementer, PAS prioritized meeting the needs of Ghanaian communities who participated in the study, with an eye toward extending and expanding the program after the conclusion of the RCT. While the Ghanaian government granted permission for the study’s implementation, its priorities, resources, and incentives were not actively integrated into the study’s design or execution.
By dividing the two areas of focus—generating generalizable evidence, on the one hand, and integrating and scaling the evidence-informed program within Ghana itself, on the other—IPA and PAS sought to operationalize evidence in different, siloed ways. Had IPA embraced the value of the second type of evidence—the requirements for integration and scale in Ghana—it could have more proactively supported and integrated PAS’ strengths, relationships, and legitimacy by including a focus on in-country scale-up as a second phase of the project. IPA also could have pressed policymakers to consider how the graduation approach and the GUP study could benefit from and advance the government’s own priorities.
The GUP project relied on the linear, rather than integrated, approach to integrating evidence. It focused first and foremost on generating rigorous evidence and relied on intermediaries to approach policymakers and implementers afterward. The case illustrates the ways in which stakeholders sometimes focus on their individual needs, audiences, and incentives at the expense of seeking a negotiated collaboration with other stakeholders from the outset, and thereby miss important opportunities for integrating evidence into practice. While the project’s RCT produced valuable evidence for the “graduation” approach to poverty alleviation, Ghanaian participants noted that the singular focus on generating academically rigorous evidence (the program’s stated goal) did not accord comparable importance to other kinds of evidence, such as feedback from field operations about the GUP project’s organizational implications. Further efforts in the area of this latter concern could have nurtured local capacity building and helped scale the program.
Integrating Evidence into Practice
Ensuring that international development efforts are rooted in and contribute to a body of best-available evidence requires active participation from all stakeholders. This approach is not easy: Each stakeholder involved—whether researchers, policymakers, or implementers—is governed by a set of incentives, time cycles, constraints, and other compelling factors that do not naturally lead to evidence integration. Failures at integrating evidence into policy and practice are common.
But negotiated collaboration offers promise for greater success. Given that each stakeholder holds a unique and valued currency, exchange rates also exist. And, once identified, these currencies of exchange can open many possible routes to the generation and integration of evidence. Most important, this effort’s return on investment can be measured not only in immediate gains but also in what is paid forward as a result: a system that truly integrates evidence into practice and thus helps beneficiaries more effectively.
Read more stories by Brendan Lehan, Rodrigo Canales, Tony Sheldon & Tory Grieves.