Assessing Advocacy

(Illustration by Darrel Rees) 

When the Western Energy Project, an initiative of the New Venture Fund, took leadership of a coalition to limit oil shale leasing in the Western United States, it wanted to adapt its efforts and track its progress as circumstances changed. A window of opportunity had opened, and the project’s funders believed that the next few months offered the best shot at progress. The leaders of the campaign knew they could not miss a beat.

Advocacy is by nature a risky business with no guarantee of success. Projects may play out over many years in a complex and chaotic political process with numerous competing interests. For the Western Energy Project (WEP), the central question was not whether the cause of protecting 2.5 million acres of public land was worthwhile, but whether the advocacy campaign could quickly move regulatory processes and actually make a difference in the time allotted.

Assessing the potential of advocacy is always daunting. Some influential observers suggest that systematic methods are mostly useless in gauging the potential of policy campaigns. Nevertheless, evidence is growing that foundations and nonprofits can make important strides in assessing advocacy projects by evaluating them against a framework of factors crucial to success. Although foundations themselves are restricted from engaging in certain types of advocacy projects (specifically, those “attempting to influence legislation”), the examples we provide in this article describe work conducted by 501(c)(3) public charities that are permitted to engage in multiple forms of advocacy, including attempts to influence legislation.¹

WEP, the William and Flora Hewlett Foundation, and several other organizations have joined Redstone Strategy Group in creating a new way to plan, monitor, and evaluate advocacy investments. The resulting framework provides a methodical and organized approach to managing some of philanthropy’s most complicated endeavors. The framework evaluates projects for nine conditions, from the presence of effective champions to favorable timing. It complements and enhances expert judgment when funders are comparing and choosing investments.

Although the framework was designed for grantmakers, it is a useful instrument for the organizers of advocacy campaigns as well. Its structured format helps advocates analyze the challenges of a campaign, determine which approaches might work best, and develop strategies to meet their goals.

Strengthening Intuition

Much has been written about applying intuitive approaches to assess the potential of advocacy investments. Most recently, Steven Teles and Mark Schmitt, in their article “The Elusive Craft of Evaluating Advocacy” in the summer 2011 issue of Stanford Social Innovation Review, suggested that policymaking is typically so chaotic that most methods to assess advocacy projects either before or during the process are worthless. They maintained that it is better to assess advocates rather than advocacy, and that selecting effective advocates yields better results than attempting to determine whether a particular campaign offers an attractive return on investment.

It is possible to assess and monitor most advocacy investments using more than bets on the intuition of prospective grantees and program officers.

Effective advocacy can certainly be disorganized and confusing, relying on many decisions that cannot be anticipated and requiring highly flexible assessment tools. It is also clear that savvy and skilled advocates are extremely important. Yet we have found that it is possible to assess and monitor most advocacy investments using more than bets on the intuition of prospective grantees and program officers.

Indeed, the nature of advocacy and the life experiences of a dedicated advocate can conspire to limit the reliability of unaided intuitive judgments. We are all beset by subconscious biases, and it takes a special type of expertise to overcome them. Daniel Kahneman, professor emeritus of psychology and public affairs at Princeton University’s Woodrow Wilson School and winner of the 2002 Nobel Memorial Prize in Economic Sciences, has devoted his career to studying expert intuition. He found that experts develop reliable intuition when they operate in environments that are “sufficiently regular to be predictable” and have the “opportunity to learn those regularities through prolonged practice”—think chess players.²

But as we know, advocacy work can be highly unpredictable and may unfold over many years. The advocates closest to the issue will develop unparalleled expertise on the policies, legislative allies, and communications tactics likely to be effective. That expertise, however, may not lead to an intuitive sense of when advocacy is more or less likely to be successful in other contexts.

Even the most experienced advocates and wisest decision-makers need help overcoming their subconscious biases. Consider how the concept of “anchoring” can color decision-making. An expert who has just won a hard-fought legislative victory thanks to an inspired public information campaign may anchor on that experience, consistently over-emphasizing the importance of public communications when evaluating opportunities. In general, biases drive us to pay more attention to what is most recent, most emotionally salient, and most familiar.

A bit of structure, such as the framework we have developed, can go a long way toward overcoming these biases and fairly evaluating an advocacy campaign’s chances of success. The framework is not meant to replace expert opinion, of course, but rather to supplement it. As Kahneman observes, “Simple, equally weighted formulas based on existing statistics or on common sense are often very good predictors of significant outcomes.”³ The framework for assessing advocacy investments described in this article provides grantmakers and grantees a flexible method to reduce bias and strengthen intuitive judgments about success and failure without assuming a particular path to success.

A Framework for Assessment

Although this structured approach to evaluating advocacy campaigns is new, it draws on the research, writings, and experience of practitioners and leading thinkers. At the outset, Redstone reviewed about a dozen models to determine the common themes in advocacy assessment. Many of them are summarized in Catherine Crystal Foster’s paper “Frameworks and Tools for Selecting and Reporting on Interim Policy Outcomes.”⁴ The review included well-known models, such as the one developed by John Kingdon, professor emeritus of political science at the University of Michigan, in his classic book Agendas, Alternatives, and Public Policies. It also included more recent models, such as a framework from the Urban Institute⁵ and work by Kristi Kimball of the Hewlett Foundation in collaboration with Capitol Impact consultants. The Kimball-Capitol Impact framework in particular, which posits six conditions essential for progress toward significant policy change, served as a crucial starting place for testing and developing the nine conditions described in this article.⁶

Out of this review emerged nine crucial conditions for successful advocacy. These conditions—such as a mobilized public and powerful inside champions—form the core of our Advocacy Assessment Framework. The framework was then field tested, and subsequently modified, by us in concert with our partners in a number of different advocacy campaigns.

The framework helped the WEP estimate the likelihood of success for the oil shale campaign. It has been useful in assessing projects to conserve the Wyoming Range, to spur clean electricity, and to improve California education data systems. The Center for Global Development has adopted elements of the framework to assess prospective research projects focused on advocacy. In all of these initiatives, the conditions built into the framework have helped funders and grantees make important decisions about which projects to support and what mid-course corrections to make in response to shifting political winds. The framework also served as a guide to analyzing why campaigns were successful or not.

The Nine Conditions

The Advocacy Assessment Framework is built around nine conditions widely viewed by leading researchers and practitioners as essential to a successful policy campaign:

1. Functioning venue(s) for adoption | The relevant legislative, legal, and regulatory institutions are functioning sufficiently for advocacy to be effective.
2. Open policy window | External events or trends spur demand for the solution.
3. Feasible solution | A feasible solution has been developed and shown to produce the intended benefits.
4. Dynamic master plan | A pragmatic and flexible advocacy strategy and communications plan is ready for execution.
5. Strong campaign leader(s) | Central advocates can assemble and lead the resources to execute the strategy and communications plan.
6. Influential support coalition | Allies can sway needed decision-makers and help the campaign leader to pursue the solution.
7. Mobilized public | Relevant public audiences actively support the solution and its underlying social principles.
8. Powerful inside champions | Decision-makers who can overcome the opposition support the solution and its underlying principles.
9. Clear implementation path | The implementing institution has the commitment and the ability to execute the solution.

These nine conditions can be used three ways—as a checklist, a rubric, and a quantitative estimator—each increasing in complexity and offering decision-makers additional information to aid their deliberations.

Used as a checklist, the framework helps grantmakers and campaign leaders consider the full range of influential factors in creating a successful campaign, not only those most relevant in recent experiences or prominent in the news. Through interviews, research, and discussion, each condition is evaluated for its presence or absence, and planned activities are matched to the conditions. Because campaigns are more likely to be successful when more of the conditions are present, the most important activities will be those that deliver missing conditions or preserve existing but threatened conditions. Once decision-makers have a completed checklist, they also have a snapshot of the campaign’s prospects and critical priorities. Though crude, the checklist approach is fast and straightforward. It ensures that all the conditions are considered—not only those that seem most relevant in the moment.

Used as a rubric, the framework helps grantmakers and campaign leaders avoid biases when evaluating conditions and can provide a foundation for collaborative evaluation and decision-making. The rubric allows grantmakers and advocates to take a deeper look at the conditions and score them from 1 to 5 on a spectrum from “not at all” present to “fully” present. The formal structure of a rubric increases the consistency of ratings between potential campaigns, over time and across evaluators. By going a step further and rating the conditions both with and without investment, decision-makers can also document their theory of change—how they expect the campaign to help policymakers adopt and implement sound solutions.

Used as a quantitative estimator for likelihood of success, the framework helps funders and campaign leaders judge returns on financial investments in the advocacy campaign. This use involves the most detail. A decision-maker not only describes an advocacy campaign with reference to the nine conditions, but also estimates the total cost of planned grants and activities. All conditions are then scored from 1 to 5 twice—as they stand before the work begins and as they are expected to be after the campaign—under the same rubric. In addition, the potential benefits of the policy sought by the campaign are estimated in comparison to the overall goal (for example, gigawatts of renewable energy capacity installed by 2015). At this point, the advocate applies a custom mathematical formula—based on the context of the planned campaign—to estimate statistics such as likelihood of success, contribution, and return on investment. Although the resulting estimates can be used to compare opportunities, they are not intended to be applied mechanically and without introspection. Rather, they add one more level of discipline to intuition and apply the same consistency to the relationships between conditions as the rubrics apply to each condition. This gives the decision-maker another point of data for consistent comparison—data that may be used alongside other qualitative considerations and compared with actual outcomes.

A Flexible Estimation Approach

Any formula that provides quantitative estimates of uncertain values, such as the likelihood of achieving policy change, is a decision-making aid, not a scientific truth. Its most important function is to apply consistently the decision-makers’ best understanding of how conditions will interact to influence the likelihood of success. The details will vary depending on the advocates’ context and hypotheses. Nonetheless, estimates obtained from a formula have proven valuable in helping funders and advocates make hard choices among their many options for policy campaigns.

When decision-makers seek quantitative estimates of the likelihood of success of a campaign, their organization’s potential contribution to that success, and their philanthropic return on investment, we apply a flexible formula that reflects the best available models and analysis. The formula is derived from two ideas: 1) There are a few significant stages that a campaign must clear before benefits are realized from policy change; and 2) some conditions can be substantially improved through near-term advocacy, whereas others should be accepted as largely outside advocacy’s influence.

We divide policy change into three stages: agenda-setting, adoption, and implementation. The first three condition scores in the framework (1-3) are averaged to estimate the likelihood that a new policy solution will find a place on the policymakers’ agenda. The next five condition scores (4-8) are averaged to estimate the likelihood that the solution will be adopted into law. The final condition score (9) is used to estimate the likelihood that the policy will be implemented to deliver potential benefits.

The formula then multiplies the likelihoods of success for each unfinished stage. A campaign to champion a new policy idea will need to pass through all three stages, so all three likelihoods are multiplied. A campaign to take a solution on the agenda through passage and implementation, on the other hand, will use the product of the likelihoods for only the adoption and implementation stages. Rather than simply averaging all of the conditions together, this method acknowledges that strength in one area can’t always counteract weakness in another. For example, the best policy solution is useless when government institutions are too ineffectual to implement it.

To estimate the funder’s and advocates’ contribution to a campaign’s success, we zero in on the second idea: Some conditions can be improved through near-term advocacy, whereas others should be accepted as outside advocacy’s influence. We differentiate between factors that the campaign will target and those that it will not. Six of the conditions (3-8) are typically designated “campaign conditions” and can often be improved by advocates in the near term. By focusing on these conditions, grantmakers learn early in the process where changes in their investments might improve the likelihood of success.

The three remaining conditions (1, 2, and 9) are typically labeled “context conditions” and reflect forces that advocates have little ability to influence via the campaign, especially in the near term. When the context is challenging—as a result of gridlocked institutions, lack of attention by decision-makers, or faulty implementation—even the most successful advocacy campaign is unlikely to produce the intended social benefits. Yet when a policy window opens in a strong institutional environment, the situation is ripe for a strong campaign to deliver policy change.

This framework’s combination of conditions was designed for campaigns that contribute to policy adoption. Although it can be useful when funders are evaluating efforts to put new solutions on the agenda and implement policies already adopted, a different balance of conditions is more effective in those cases.⁷

Contribution is estimated by comparing the estimated likelihood of success if the campaign conditions improve via advocacy to the likelihood of success without intervention. The difference is a campaign’s contribution. A significant contribution might be as high as 90 percent; a minimal contribution could be as low as 10 percent.

With an estimate of contribution in hand, funders can roughly assess the philanthropic return on investment for a campaign. This calculation multiplies the potential social benefit of a successful policy by the percentage-point increase in the likelihood of success (the contribution), and then divides the product (contributed benefits) by the cost of the advocacy campaign to estimate return on investment (contributed benefits per dollar invested). Introducing benefit and cost estimates helps level the playing field between big, costly campaigns targeted at significant policy change and small, targeted efforts to make positive but incremental change.

Quantified to this extent, the framework helps program officers and grantees translate their knowledge into consistent estimates that form the basis for discussion and modification. The estimates are not final answers, and they generally have large margins of error. Nonetheless, they serve as starting points that synthesize many pieces of data—both publicly available information and that provided informally by experts. They identify strengths and weaknesses that can be used to set strategy, monitor progress, and assess results. And in cases where differences in results are striking, they can usefully influence decisions.

Stages of Assessment

The Advocacy Assessment Framework is designed to be useful at every stage of an advocacy campaign. At the beginning, it helps grantmakers judge which advocacy approach is the most promising and how their financial support can be most effective. Once the campaign is under way, funders and advocates may use it to monitor progress and make any needed adjustments. At the conclusion of the campaign, the framework offers an objective instrument to assess what worked and what didn’t.

• Evaluating pathways to advocacy success, helping funders and grantees pick a strategy that complements the context and collaborators’ investments.
• Incorporating contribution to identify opportunities that deliver the most bang for the buck once the strategy has been decided.
• Monitoring progress to respond nimbly to events that require a change in strategy.
• Assessing results to test assumptions and learn about how campaigns were truly effective in educating policymakers and the public.

The next few sections describe how this framework and resulting estimates have guided the design and execution of recent campaigns.

Evaluating Pathways

The first step in the process is to choose a pathway for the campaign. Choosing from among all the possible campaign pathways to advance a program’s goal is one of the most important, and difficult, steps toward creating a well-developed strategy. Several paths may lead to the same result; each may have different strengths and weaknesses. For example, if a program’s goal is recovery of an endangered fish species, it could pursue water rights litigation under the Endangered Species Act, regulations to modify the operation of dams, or legislation that appropriates more funding for recovery.

Each pathway will have different context conditions and campaign conditions. By using the framework to assess each of the pathways, we will see which has the greatest likelihood of success. Pathway assessment proved extremely helpful to the Hewlett Foundation when it selected the top five states where it would give support for broad-based advocacy of clean electricity policies, rallying voices beyond the environmental NGO community. The aim was to provide advocacy support in influential states with high potential for reductions in greenhouse gas emissions. One of the requirements was that the states were likely to be home to important decision-makers in deliberations on the Clean Air Act.

The framework may be used to evaluate pathways:

• As a checklist. Scan the pathways to see which offers more of the nine conditions, and flesh out a few top alternatives.
• As a rubric. Assess each of the top alternatives for the nine conditions, reprioritizing on the basis of highest average score if necessary.
• As a quantitative estimator. Estimate likelihood of success for each of the top alternatives, again reprioritizing if necessary. Test the sensitivity of the relative likelihoods of success to the conditions most liable to change over the course of the campaign.

Incorporating Contribution

Determining the likelihood of success of a pathway is only the first step toward evaluating it. The framework also can be used to estimate the extent to which the grantmaker’s investment would contribute to the success of the campaign. The greater the likelihood that its investment would make a difference, the more reason for a grantmaker to contribute. Both factors enter into the calculation of estimated return on investment, which may be an important factor in a funder’s deliberations.

For example, one state became the top choice for broad-based grantmaker support when the Hewlett Foundation analyzed the prospects of advocacy for clean electricity in 10 states and one region. From interviews with experts in the field, we estimated the likelihood of increasing the top state’s renewable energy requirement to be about 90 percent with Hewlett Foundation grants for educating and building broad-based support for clean energy. The contribution made by engaging broad-based groups was estimated at 50 percent.⁸ That suggested that the foundation’s dollars would have a large impact on the state’s policies.

Similarly, efforts to inform the state’s public utility commission and city council deliberations on rules governing a utility’s energy mix were given likelihoods of success of about 70 percent with the Hewlett Foundation’s support. The contribution was forecast at 25 percent. Those figures helped give this state the highest expected return on investment among the candidates and pushed it to the top of the list.

Only by strengthening relatively weak conditions can a grantee make a significant difference in the outcome of a project. If a grantee is working on improving conditions that are already strong, we will see little change in the resulting likelihood of success, and the estimated impact of the campaign will be much lower.

The framework may be used to incorporate contribution:

• As a checklist. Compare current conditions for each pathway to the goals of the advocacy campaign and prioritize opportunities to deliver missing conditions.
• As a rubric. Rate each of the nine conditions twice for each pathway. First, rate the conditions as they would be without the proposed campaign. Then rate them on the basis of what the campaign is expected to deliver. Prioritize opportunities where the change is greatest, all else being equal.
• As a quantitative estimator. Estimate likelihood of success for both sets of rubric ratings. Use differences in the results to estimate contribution and expected return; prioritize the opportunities that offer the highest returns.

Monitoring Progress

The policy environment often shifts as a strategy moves forward. Politicians leave office, the economy takes a sudden plunge, or new issues capture the public imagination. Advocates must be ready to respond quickly, and to do so they must closely monitor the progress of an initiative. This structured approach provides a formal process for tracking a strategy and allows grantmakers to monitor the trajectory of various investment opportunities, realigning their funding as necessary.

As WEP began the campaign, it used the framework to compare the project’s likelihood of success before it became involved and to forecast results now that it had entered the fray. The calculations showed the likelihood of success jumping from approximately one in five to approximately one in two with WEP’s involvement. This improvement was important, because it meant WEP was delivering resources where needed and not just duplicating others’ efforts.

WEP is using the framework to estimate its likelihood of success through several stages of its efforts to restrict oil shale leasing up to release of the Final Environmental Impact Statements and a Record of Decision (ROD).

For example, by using the framework to track their progress, WEP realized it was targeting inside champions without the authority to deliver the policy change it sought. In response, WEP reassessed how it could educate the decision-makers who mattered. As Director Laurel Angell said, “I have found myself thinking about the campaign in the same terms and structure as the [framework] does.” As a result, “There were a few times when the scores were a bit lower, and it was helpful to ask why that was, take a step back, and evaluate it: What could be going better?”

The framework may be used to monitor progress:

• As a checklist. Track the nine conditions over time with a “traffic light” dashboard. If conditions deteriorate unexpectedly or are not improving as expected, consider a change in strategy or a change in pathway.
• As a rubric. When crucial events occur, reassess the nine conditions against the rubric. If assessments change dramatically—for the better or worse—consider a change in strategy to eliminate duplicated effort or seize a missed opportunity.
• As a quantitative estimator. When critical events occur, re-estimate likelihood of success and contribution, and revise estimates of return on investment. Consider reallocating the portfolio or redesigning campaign strategies to improve returns.

Assessing Results

Winning significant policy reforms is never easy. Funders and advocates know they must learn the lessons of past projects if they are to succeed in future efforts. The Advocacy Assessment Framework can help by analyzing completed projects for what worked, what didn’t, and why.

An assessment of an effort to protect land in the Wyoming Range from oil and gas drilling suggests that continued NGO support was essential to securing the final outcome, despite large variations in the likelihood of success. All parties involved in the advocacy campaign ultimately regarded the 2009 Wyoming Range Legacy Act, which protected 1.2 million acres of remote Wyoming mountains from further oil and gas exploration, as a tremendous victory for conservation.

The work began in 2005 when the US Forest Service opened up 44,000 acres in the Wyoming Range for oil and gas leases. A broad coalition of conservationists, sportsmen, unions, outfitters, and landowners was put together, spearheaded by Trout Unlimited and the Wilderness Society. The campaign gained momentum but saw two significant events that reduced its likelihood of success: first, its prime inside champion, US Senator Craig Thomas, died in 2007; and second, an era of high gas prices led to more public support for drilling. The campaign overcame these obstacles by finding a new inside champion to introduce the bill (the new US Senator John Barrasso, with support from Senator Thomas’s widow and Wyoming Governor Dave Freudenthal) and a strong campaign with the broad support coalition that led to a floor vote and passage of the bill in 2009.

This example shows that the continued NGO support was crucial to secure the final outcomes (their contribution was high, estimated around 60 percent), and also that the likelihood of success of a given campaign can be expected to vary over time. Decisions on continued investment should weigh potential changes in the political landscape or likelihood of success conditions. In this case, continued investment in the face of varying estimates of likelihood of success led to a very successful overall outcome.

The framework may be used to assess results:

• As a checklist. Evaluate the campaign’s final outcome and compare it with the initial expectations about which conditions would persist. Assess what the campaign could have done differently to respond to changing conditions.
• As a rubric. Review campaign activities against changes in the conditions over time. Identify the activities that seemed to improve conditions—accounting for the work of other actors and external events—and those that were less cost effective.
• As a quantitative estimator. Compare likelihood of success estimates over time and across campaigns to improve the way estimates are made, especially the relative weights on the conditions. If outcomes consistently differ from estimates, review how those estimates were made to improve accuracy in the future.

Conclusion

Savvy grantmakers know that advocating policy change is often the most productive path when an issue is monumental and complex, such as global warming or poverty.

Savvy grantmakers know that advocating policy change is often the most productive path when an issue is monumental and complex, such as global warming, poverty, or K-12 education. But philanthropic advocacy is also an extremely difficult and chaotic process fraught with risk. Absent explicit efforts to assess that risk, it may take many expensive years to determine whether advocacy campaigns are likely to succeed.

The Advocacy Assessment Framework offers grantmakers a clear-eyed, analytical method to assess whether their efforts to solve big problems will make a real difference. At the same time, it gives advocates an objective way to identify the best strategies to influence public policy.

The framework shows promise in bolstering the intuition and expertise of prospective grantees and other experts as funders synthesize information to compare campaigns, estimate the expected return of an investment, and predict an organization’s potential contribution to success. The early applications described in this article suggest that it can help grantmakers determine whether they are choosing the right topics and investing in the right grantees to reach their goals.

The authors thank Tom Steinbach, Barbara Chow, Kristi Kimball, Erin Rogers, Michael Scott, Chris Shearer, and Val Hovland for their generous contributions of time and advice.

Redstone has adapted the framework described in this paper for institutions working on a wide variety of social issues and at every phase of the policy design, advocacy, and implementation process. More about the approach to assessing advocacy described in this article can be found at: www.redstonestrategy.com/advocacysupport

Read more stories by Jason Blau, Ivan Barkhorn & Nathan Huttner.