Evaluation with the End in Mind

How should life-limited organizations think about evaluation? These kinds of organizations expend all their resources over a constrained period of time—versus limited contributions in perpetuity—with the aim of making a larger, more immediate social or environmental impact. Most organizations use evaluation to determine outcomes, refine current efforts, and set future direction. But what purpose does evaluation serve when the clock is ticking down, and there is a heightened sense of urgency to accomplish your mission before the doors close?

These are real questions for life-limited organizations. While there is a growing literature based on the function and logistics of operating a life-limited organization, less is known about how to utilize evaluation for these types of organizations. As staff of a life-limited organization, we have needed to re-position evaluation as a tool to drive progress toward our end goals and to measure the enduring impact of our efforts. Following are recommendations for achieving this, based on our own work.

Evaluation and the life-limited organization

Minnesota was one of the first states to sue major tobacco companies for deceiving its citizens about the harmful nature of their products. The state settled prior to the Tobacco Master Settlement Agreement, in which the remaining 46 states reached agreements with the four leading tobacco companies. Our organization, ClearWay Minnesota, was created in 1998 with 3 percent ($202 million) of the funds Minnesota won as part of its settlement. It was established as an independent nonprofit tobacco control organization, whose mission was to enhance life for all Minnesotans by reducing tobacco use and exposure to secondhand smoke through research, action, and collaboration. The architects of the organization believed dedicating significant dollars within a set amount of time would have the strongest impact, and so we were chartered with 25 years to accomplish our mission.

Initially, we approached evaluation like most organizations do, putting in place program evaluations to demonstrate outcomes, quality improvement, and transparency. Many of our programs were the first of their kind, so we invested in independent and robust evaluations with a strong ethic of publishing findings to advance and inform the field, as well as improve the programs. In due course, we developed an organizational logic model and an online system that mapped evaluation findings against our model of change. We also evaluated our success in achieving the stated goals and metrics of our multi-year strategic plans. Finally, we developed surveillance systems to track tobacco-related behaviors and attitudes over time.

As the organization matured, however, we became increasingly aware of our limited life, and recognized a need to adopt an organizational framework that would help guide us through a productive tenure and successful closure. While we spent the first 10 years developing, establishing, and demonstrating the success of various strategies for addressing tobacco control, in 2009, we took stock of our remaining resources and decided to focus our programs and funds on a few specific goal areas. Over a 12-month planning process, we systematically aligned our strategic plan, budgets, work plans, and evaluation approach with these goal areas. The image below illustrates how we integrated our vision, mission, goals, and strategic plans from the most aspirational to the most tactical. Through this process, we came to understand that evaluation would continue to play an important, though changed, role for the organization.

(Image from ClearWay Minnesota’s Legacy Guide)

Based on this work, here are three evaluation recommendations for life-limited organizations:.

1. Set “legacy goals” that are part of a long-term evaluation framework.

To be effective, an organization needs to formulate what we call “legacy goals”—goals that reflect what the organization will accomplish by the end of its limited lifespan—and then track progress toward those goals. Unlike most organizations, life-limited ones do not have the luxury of unlimited time and resources to re-set goals as initiatives evolve. Being mindful of both the reality and urgency of work in the short term, and the rapidly approaching finish line helps organizations keep their vision and focus sharp.

ClearWay Minnesota established three goals we wanted to accomplish by 2023:

• Reduce the prevalence of smoking among adults to less than 9 percent
• Reduce secondhand smoke exposure among nonsmoking adults to less than 5 percent
• Advance the science of eliminating tobacco-related health disparities

Making these goals the new focal point for our organization meant making a significant organizational shift. It took some time to get comfortable with the idea that our programs and accompanying evaluation would focus on our three longest-term outcomes. For example, we ended a very detailed evaluation of our strategic plan, which included multiple metrics for each of our original goal areas, and replaced it with a recurring annual “Legacy Evaluation Update,” which honed in on our progress toward the three legacy goals. We swapped the question, “How will we know we’ve succeeded in our strategic plan?” and instead began asking, “How will this effort impact our legacy goals?” In doing so, we were able to shift our measurement resources and strategic attention in more efficient ways.

2. Be selective about programmatic evaluations.

Of course, our focus on legacy goals did not entirely displace programmatic evaluations. In fact, it sharpened our criteria around what we should evaluate, knowing that we no longer had the same luxury of time and resources we did in earlier years. This meant being selective and evaluating only the most innovative programs with the greatest potential for creating new approaches, building capacity, developing best or promising practices, and informing the field. It also meant designing timely evaluations to allocate increasingly scarce resources and ensure the dissemination of the information in our lifetime.

Over the past two years, for example, we have worked to advance several policy priorities, including raising the price of tobacco products, restricting flavored tobacco products, and raising the legal sales age for tobacco to 21. We have successfully advanced most of these policy options at the local level, but we had to prioritize which of the local policy changes to evaluate. Given that our menthol sale restriction policies in three Minnesota cities were among the first of their kind nationally, we determined that their evaluation would be novel and innovative, and have the potential to inform future state and national efforts. At the same time, we made the difficult decision not to evaluate the impact of the first local policies passed in Minnesota that raised the legal tobacco age from 18 to 21, knowing that other states and cities already have robust evaluation efforts in this area underway.  

3. Measure the enduring impact of your work.

We see incredible value in measuring the enduring impact of an organization’s work. Many policy and system changes generate financial or human gains long after they are implemented. Being able to quantify the impact of specific policies or programs 5, 10, or 20 years later reinforces the importance of the strategy choice and investment of dollars in the first place, and helps inform future projects.

ClearWay Minnesota is conducting simulation studies in our final years to quantify the impact of declining tobacco use by estimating lives saved by fewer tobacco-related diseases and deaths, as well as associated reductions in medical costs. The simulation model is also helping determine the impact of policies and strategies—such as smoke-free laws and improved access to cessation services—that have contributed to successes in tobacco control. We are also investing in two final, statewide survey research projects: the Minnesota Adult Tobacco Survey and the Tribal Tobacco Use Project. Both aim to help us better understand tobacco use and quitting both among the population at large and within one community where we worked extensively. These studies will help us tell our story and hopefully position ClearWay Minnesota as a public investment model that others can duplicate. These long-term evaluation studies will also serve our partner organizations. Although our work is ending by 2023, the tobacco industry is not, and leaving our partners and stakeholders with knowledge to help them meet future challenges and ultimately eliminate the harm of tobacco is a part of the legacy we are working to create.

Designing evaluations that address legacy goals, using limited evaluation resources selectively, and measuring the enduring legacy of organizations like ours can all help determine the efficacy of the life-limited funding approach. But they can also be useful in the evaluation of limited-term initiatives within organizations that live on in perpetuity. Either way, we hope our collective learnings can provide a clearer roadmap for the next generation of limited-life organizations.

Read more stories by Ann W. St. Claire & Barbara A. Schillo.