Across the United States, government officials, social entrepreneurs, nonprofit leaders, philanthropists, and impact investors talk about the need to pursue not just program outputs but “positive outcomes”—results that make a genuine difference in beneficiaries’ lives. They talk about adopting models that are designed to focus attention on such outcomes: public-private partnerships, collective impact projects, pay-for-success (PFS) initiatives. And they talk about rigorously using data to track progress toward a given set of outcomes.
But all of this talk presupposes that everyone agrees on the meaning of “positive outcome.” In fact, there is no single shared definition of that term. And perhaps no such definition is possible: Determining whether a program is actually changing people’s lives is fiendishly complex work. It’s much easier to measure outputs. As a result, the following kinds of scenarios are all too common:
- An organization measures how many previously homeless people it has housed—without measuring whether they stayed off the streets, whether they became healthier, or whether they reduced their use of services.
- An organization measures how many children it has placed into prekindergarten education—without measuring whether their skills improved at graduation.
- An organization measures how many teenagers it has helped to find jobs—without measuring whether those jobs provide a living wage or whether they create a path to economic security.
But even if defining outcomes in social sector work is no simple matter, practitioners must resist the temptation to ignore this challenge. Our firm, Third Sector Capital Partners, has served as an advisor and coordinator for an initiative that illustrates one model for clarifying which outcomes matter most.
In December 2014, Cuyahoga County, Ohio—which includes the city of Cleveland—launched a PFS initiative called the Partnering for Family Success Program. This initiative aims to reduce the length of stay in out-of-home foster care placement for children whose families are experiencing homelessness, and it provides housing in combination with an intensive 12-to-15-month treatment program. The program serves 135 families, and it will last five years. Several philanthropic organizations are funding the program, and Cuyahoga County will reimburse those funders if (but only if) the program achieves certain clearly defined outcomes.
The lead implementing partner for the program is FrontLine Service, a local nonprofit social services agency. Cuyahoga County collaborated with FrontLine to define “positive outcomes” for the program and to develop a method for measuring whether the program has achieved those outcomes. The work of defining and monitoring outcomes is essential not only to supporting the PFS funding mechanism but also to enabling FrontLine to manage the program. Over the course of this initiative, FrontLine has been receiving outcomes data from county agencies and other partners, and it uses those data to drive continuous learning and improvement.
What to Ask
Through our work on the Partnering for Family Success Program, we have learned that the best method for defining positive outcomes starts with asking the right questions. We urge social sector leaders to answer a set of six questions that correspond to the classic questions—“5 Ws and an H”—that journalists pose when they report a news story.
What are we trying to achieve? The less specific the goal, the harder it is for communities to agree on whether they have achieved success. In the Cuyahoga County program, the initial goal focused on addressing homelessness in general. Then, following a collaborative process that involved analyzing county administrative data and soliciting input from community members, the goal of reducing homelessness among families that interact with the child welfare system came to the fore. Ultimately, program leaders selected a more specific goal to be the main positive outcome: reducing the number of days spent in out-of-home foster care by children whose caregivers are experiencing homelessness.
Who are we trying to serve? Identifying the most appropriate beneficiaries of a program can help program leaders to determine which outcomes are most relevant and most achievable. Defining the beneficiary population of the Cuyahoga County program as homeless families with children in foster care was a breakthrough that allowed the county to focus on the connection between homelessness and out-of-home placement. That insight in turn led the county to consider interventions that involve multiple government agencies.
Why are we pursuing one particular approach? During the decade before it launched the new program, Cuyahoga County had sought to reduce the number of children in out-of-home foster care and to reduce its homeless population. Although the county made significant progress on both of those goals, data showed that some families persistently struggled with access to housing and also interacted persistently with the county’s Child and Family Services agency. So the county based its design of the new program on the need to address the relationship between those two service areas.
When will we know that we have achieved success? Program outcomes occur within different time frames. It’s possible to measure some of them within days or weeks of initiating a program; others take years to measure properly. Initially, the Cuyahoga County program used several interim performance metrics: the number of primary caregivers that it has housed, for example, or the number of caregivers that it has reunited with their children. But ultimately, in order to serve the goal of keeping families together over time, the county decided to use a longer-term metric—reduction in the number of days that children spend in foster care over five years—as its primary measure of success.
How will we know that we have achieved success? Methods of outcome evaluation are, of course, the subject of an ongoing debate. Is it best to conduct a randomized controlled trial (RCT)—the so-called gold standard of evaluation—even though that method entails considerable expense and requires a denial of services to families that end up in a control group? Or is it enough simply to measure outcomes against a historical baseline? For the Cuyahoga County program, leaders chose to use an RCT. Their goal in doing so was twofold: to ensure that the county could rigorously attribute outcomes to the program, and to create an evidence base that would yield lessons for other communities.
Where are we seeking to generate positive outcomes? Deciding where to intervene in the lives of beneficiaries often depends on access to data or on the availability of resources (such as an infrastructure of suitable service providers), and constraints in those areas can limit a community’s ability to achieve certain outcomes. Cuyahoga County, for example, would have preferred to measure outcomes for all housing-insecure families in its jurisdiction, but its current systems allow it to capture data only on families that enter a homeless shelter. It’s also arguable that the most appropriate point of intervention occurs before a family experiences housing instability. Yet the county lacks the ability to identify families that would benefit from early support of that kind. For these reasons, the Cuyahoga County program is designed to intervene at a point where—given the information and the resources available to the community—it can measurably improve lives.
In addition to asking and answering these questions, community stakeholders—particularly governments—must ask a seventh question: So what? W hat a re the policy implications, and the implications for overall systems change, of defining, measuring, and achieving a given set of positive outcomes? For those who want to achieve impact on a large scale for populations in need, this question is perhaps the most important one of all.
What Happens Next
The process by which we worked with Cuyahoga County to answer questions about outcomes took a great deal of time, and required deep partnerships with a wide array of service providers and funders. In any event, the county’s commitment to answering those questions before undertaking a multiyear initiative has resulted in a more deliberate—and potentially more transformative—use of taxpayer dollars.
The Partnering for Family Success Program is nearing its midpoint. Its legacy will become clear only as its five-year duration comes to a close. If FrontLine is able to reduce the number of days that children in homeless families spend in out-of-home foster care, then Cuyahoga County might expand the programming that FrontLine has developed, or it might extend the PFS model to other program areas. One thing is certain: If FrontLine meets its outcome goals but the county ends up returning to the traditional model of program delivery—a model in which a government pays providers only on the basis of how many people they serve—then the success of this initiative will be a hollow victory. The ultimate purpose of focusing on outcomes, after all, is to change how government and social sector organizations do business.
The Cuyahoga County program offers one example of how communities across the United States are taking a new approach to confronting seemingly intractable challenges. This approach involves stepping back to address crucial questions about outcomes, using data in an ongoing way to measure performance against those outcomes, and designing feedback loops that help drive performance improvements. In many cases, to be sure, the outcomes that stakeholders initially establish for a project will prove to be less-than-perfect indicators of success. All of us in the social sector have a long way to go on this journey. Yet the willingness of more and more stakeholders to ask questions about outcomes, and then to combine that process with data-driven decision making, is heartening. Over time, we believe, more and more communities will adopt this approach, and positive outcomes—however stakeholders choose to define them—will follow for those who are most in need.