(Illustration by Martin Haake)
In 2003, the Greater Ithaca Activity Center (GIAC), located in upstate New York, received a letter from a funder that expressed concern about the outcome measures that GIAC had presented. The funder suggested that GIAC—a multi-purpose community center that has served culturally diverse young people and families since 1972—could achieve greater impact by narrowing the scope of its programs and services. About this interaction between GIAC and the funder, one observer later said: “Programs took center stage; the relationship between people was nowhere to be found.” Soon afterward, GIAC staff members began working with a facilitator on a yearlong effort to articulate the difference that their organization was making in its community. Together, they and the facilitator concluded that the funder’s preferred outcome measurement model failed to capture what was most essential about the work that GIAC had been doing. To account for that work adequately, GIAC would need to adopt an evaluation framework that treats relationship-building not merely as instrumental to programmatic outcomes but as valuable in its own right.1
Today, more than a decade later, nonprofit organizations still struggle to represent their work in the context of prevailing outcome measurement models. In a 2010 survey of 415 US-based nonprofits conducted by Johns Hopkins University, 80 percent of nonprofit leaders called for the development of better tools to measure qualitative impact.2 Underlying the concern of these leaders is a sense that existing evaluation models overlook something essential about how nonprofits serve individuals and communities. “I think [outcome measurement] is a worthwhile thing,” the executive director of a mental-health counseling service told the authors of a 2009 study. “I am not complaining about the time it takes; it is a tool that needs to be continuously refined and made better, but it needs to get closer to the actual activity.”3
Our research on nonprofit organizations has sought to develop a better understanding of this “actual activity.” This research has led us to conclude that standard outcome measurement models are limited by their assumption that programs, and only programs, are what lead to participant outcomes. (In this article, we apply the term “participant” to every type of person who potentially benefits from interaction with nonprofit staff members. Depending on the institutional setting, staff members might use other terms to refer to a participant—“client,” “resident,” “constituent,” “patient,” “beneficiary,” and so forth.)
Anyone who has ever set foot in a nonprofit organization understands that the work done by staff members extends beyond the simple task of program delivery. Nonprofit workers spend time getting to know the people they serve. They take time to adapt services to meet particular needs. They work with participants to identify outcomes that will be relevant and meaningful to those participants. They connect participants to resources that are available outside their own organization. Not only is all of this work instrumental to achieving program outcomes, but it can also lead to beneficial outcomes that program-centric models are ill equipped to anticipate.
To understand nonprofit performance fully, we need to broaden the lens through which we view the work that staff members do to achieve outcomes for participants. Programs and program outcomes matter a great deal, to be sure. But an outcome measurement model that relies exclusively on “the program” as its unit of analysis will miss a good portion of the work that staff members do. Not surprisingly, many nonprofit staff members come to view outcome measurement as a burdensome funding requirement, rather than as a practical tool to help improve their work.
In making this argument, we do not wish to criticize outcome evaluation in general. The field of evaluation is incredibly diverse, and it includes many approaches that do recognize factors other than programmatic intervention. Yet most nonprofits don’t have the resources to employ their own evaluator, and so they depend on popular outcome measurement guides or on whatever model their funders happen to require. And these default models, in the main, continue to focus pivotally on program activity.
The Legacy of Program Evaluation
Why did the standard outcome measurement framework for nonprofits come to rely on the program as its primary unit of analysis? Evaluation as an identifiable field of practice emerged during the 1960s as part of the US War on Poverty.4 After the federal government launched a variety of social policy innovations, public officials and other funders recruited social scientists to address the issue of effectiveness: Were these policy interventions, in fact, producing the intended outcomes? Initially, the main concern of social scientists was to determine whether there was a causal relationship between a given policy and a given measurable impact. Over time, however, evaluators began to develop methods that would go beyond what was essentially a “black box” approach. Their goal was to understand not just whether an intervention produced a desired effect, but how: What were the intervening causal mechanisms that resulted in that outcome?
Evaluators fastened on the programmatic intervention as a way to organize their analysis of those causal mechanisms. They also adopted the concept of “program theory,” which posited that every intervention rests on a set of assumptions—sometimes explicit but often implicit—about why a given program would lead to a given outcome. By specifying a program theory for each intervention, evaluators hoped both to identify causal mechanisms and to understand why the intervention did or did not succeed. Also during this period, evaluators drew on insight from the field of policy implementation studies. A principal theme in that field involves the way that implementer discretion can undermine the pursuit of policy objectives. As a consequence, evaluators have tended to regard any deviation from the design of a program as a threat to its effectiveness.
The programmatic legacy is evident in the outcome measurement models that many funders currently require from their nonprofit grantees. That legacy, in our view, has several problematic consequences. First, these models portray staff members mainly as program implementers, thereby overlooking the way that they partner with their target population in pursuing change. Second, these models do not account for staff member discretion in tailoring programs to meet participant needs or to match the context of participants’ lives. Third, these models treat participants as targets of intervention and therefore ignore their agency in the change. And fourth, these models neglect the role that outside organizations, programs, and services play in achieving intended results.
A number of evaluation scholars have challenged the prevailing focus on program implementation. Michael Patton argues that in certain complex settings, the framing of desired outcomes needs to emerge not from predetermined program goals, but from iterative dialogue and deliberation.5 Madine VanderPlaat points out that the instrumental nature of program-centric evaluation fails to accommodate empowerment-oriented initiatives that treat active engagement and mutual support among participants as critical drivers of change.6 And Huey-Tsyh Chen suggests that a program theory should consider not only the hypothetical causal mechanisms of a program, but also the actions that staff members undertake to run the program and the contributions that partner agencies and participants themselves make to achieve outcomes. Together, these elements make up not just a program theory, but “an action theory,” as Chen calls it.7
The next step for funders and nonprofit leaders, we believe, is to develop a more complete understanding of frontline work—of the “actual activity” that staff members do, apart from program activity. Research across a range of disciplines emphasizes the importance of this work. The social policy scholars Lisbeth B. Schorr and Frank Farrow, for instance, argue that funders and nonprofit leaders should seek out evidence that would enable them to learn from “interventions that are complex, interactive, and relationship-based; that can be adapted to a variety of cultures and populations as well as new and changing contexts; and [that] require significant frontline flexibility and sensitivity.”8 So far, however, frontline work has received little attention in the debate over how best to measure nonprofit effectiveness.
The time has come to enlarge the frame of reference that we use in assessing nonprofit performance. By looking closely at frontline work, we can create a more accurate picture of the efforts that enable staff members and participants to achieve transformational change.
Varieties of Frontline Work
In our research, we have found that nonprofit staff members commonly engage in four types of frontline work. (All of the examples and quotations used below come from interviews that we have conducted with nonprofit professionals as part of our ongoing research.)
Relational work | Staff members build relationships with participants in order to understand their problems and aspirations. Relationship-building can result in significant outcomes (self-confidence, a sense of efficacy) that are independent of program activity. Relational work can also help anchor and organize a program by ensuring that it meets a real need and that it matches participants’ existing capacities or goals. Indeed, research in mental health, elder care, youth development, and other fields shows that frontline relationships are often more central to outcomes than are program techniques. The same principle applies to nonprofits that engage in grassroots organizing—an activity in which success, almost by definition, pivots on the quality of relationships.
In interviews, frontline staff members refer frequently to the role of relational work in their daily practice. In fact, this kind of work often must occur before a staff member and a participant can do any other work together. Relating to participants requires skill and discernment, and in many cases it requires a staff member to cross certain formal boundaries. “There is a lot of discussion in the legal field: … ‘How close do you allow your clients to become?’ Some lawyers never talk about their personal lives with clients, ever, period. I don’t take that tack,” said one immigration attorney.
To build enduring relationships, frontline professionals listen carefully and avoid falling back on pat responses. In a nonprofit that provides services to refugees and immigrants, one staff member developed a technique in which she would ask clients to write a list of their goals; at the same time, she would write down what she understood to be their goals, and then they would compare what each of them had written. When clients saw that her list aligned with their own, they felt that she had truly listened to them. That practice helped to build trust, and it laid a foundation for other types of work.
Adjustment work | Staff members often adjust their approach in response to a participant’s goals, needs, and immediate circumstances. “When you work with trauma, you have to be able to tolerate ambivalence,” said a frontline professional who helps people who suffer from PTSD. This professional described a fairly typical client interaction: “This guy has come back and left, he has come back and left, he wants to explore it [his issue] and he does not want to explore it.” The best solution in that case, according to this frontline worker, involves “letting him [the client] go at it at his own pace.”
Adjustment work also reflects an understanding that change is not linear. What might seem like a step backward can be, in fact, a critical step in a long-term transformation process. Frontline workers must be able to judge whether a particular “step backward” will promote a desired transformation or whether it will undermine a participant’s growing self-confidence. In interviews, many frontline workers shared vignettes in which adjustment follows a trial-and-error pattern. One frontline staff member, who helps residents in a homeless shelter in their job search efforts, explained how he debriefs clients after they go on job interviews: “I say to them, ‘You know what? A ‘no’ to you is more power. Don’t take it as a rejection. Just get confident. You didn’t lose anything; they’re losing you, because you’re a good worker.’ So you have to build the self-esteem, and get them back to another interview.” Through this kind of debriefing, the staff member and the client can experiment with different strategies for matching the client’s aptitudes with employer needs.
Adjustment work is not a matter of “anything goes.” Nor is it always effective. But it’s real, and it’s important, and it requires experience and judgment. The frequency with which the frontline professionals whom we have interviewed mention adjustment work is striking, and it calls into question the fixed nature of program-centric outcome measurement models.
Codetermination work | Staff members work with participants to sort through various possible courses of action. Codetermination work starts with the conviction that participants have the right to claim ownership over their path toward transformation. “When all is said and done, it’s really up to them,” a frontline worker in a faith-related employment program said. “We can support them, but unless they make a decision to change, their lives are not going to improve.”
Codetermination work can be highly demanding. It requires complicated situational judgments about how to reinforce a participant’s sense of agency. The impact of such work can be quite subtle—as much a shift in perspective as a change in approach. Consider this interchange between an employment counselor and his client:
“These things I’m asking you, they’re not for me. You don’t come dressed up like this to see me, to please me. It’s for you. If you look good, you feel better. How do you feel when you finish showering, and you [dress] up, and you come to see me? How do you feel?”
“Well, everybody gives me a compliment.”
“That’s what I’m talking about. You do it for you.”
“Oh, I thought I was doing it for you.”
“Oh, no, you don’t do it for me.”
Frontline work of this kind involves using techniques to ensure that participants embrace their ownership of the change process. One immigration attorney, for example, described how she provides clients with her supervisor’s contact details. Those clients reflexively defer to her—she is a lawyer from the dominant racial group—but she tells them that they are hiring her and so they can also fire her.
Linking work | Staff members connect participants with resources from other parts of their own agency, from other nonprofit organizations, and from local community groups. Individuals and families often have multiple needs that go beyond what any one organization or any one program can meet. In addition, their needs may change over time. Frontline staff members therefore achieve success not just by running their own programs, but also by serving as a portal to other programs and services. When this kind of linking work is absent, the impact of a given program is less likely to be sustainable.
Consider the example of employment programs, which often require frontline workers to complement job training activities with referrals to organizations that provide various products and services: work clothing, dental work, tattoo removal, transitional housing, and so forth. Success in this area comes from being able to piece together disparate resources. A frontline staff member who works with people who have ADHD, dyslexia, and other mental or emotional disabilities told us that his job covers “whatever it is that a client needs to get back into the workforce and become [a] productive [member] of society.”
In interviews, staff members often refer to the amount of time that they spend on linking activities—getting to know colleagues at agencies that offer related programs, attending partnership meetings, working out the terms of complex collaborative arrangements. A common lament of these professionals is that they generally don’t get credit for that time. “I do a lot of finding resources for clients: ‘You need to go here. You might find food here.’ … I do a lot of case management. I don’t always document it,” said a frontline professional who works with people who suffer from depression, among other problems. “Talking to them about what depressed them is not effective while their basic needs are not met,” this staff member noted.
Nonetheless, a willingness to perform linking work can matter a great deal for participant outcomes. One frontline professional, for example, makes a point of joining her clients when they visit a vocational counselor. “Sometimes it’s hard for members and clients to articulate what they think their barriers are,” she said. “So if I can articulate what I see, … it gives the vocational counselor a better picture of how best to serve [each client].” In another case, this staff member noted, a colleague didn’t engage in that kind of linking work with a client. As a result, important information about the client got lost in the hand-off to another agency. “This process should’ve taken, at the most, two months. It’s probably going to take a year,” the staff member said.
In sum, when we judge the effectiveness of an organization solely by its programmatic outcomes, we risk misunderstanding the factors that determine the success or failure of that organization. When we ignore the relational work that staff members do, we neglect the way that high-quality relationships may compensate for a poorly designed or under-resourced program. When we do not consider the adjustment work that staff members do, we lose access to valuable information about the potential mismatch between the logic of a program and participants’ actual path to transformation. When we do not account for codetermination work, we fail to see how programmatic “failure” may reflect real progress for clients whose choices do not conform to program logic. And when we do not pay attention to the linking work that staff members do, we underestimate how dependent client outcomes are on resources that lie outside any one organization.
Principles of Outcome Measurement
To overcome the limits of models that focus solely on program performance, we need to develop models that encompass all of the work that takes place between frontline staff members and participants. Our research has led us to formulate four principles of a more comprehensive outcome measurement framework—principles that reflect the various forms of frontline work.
Honor relationships | An outcome measurement framework should take into account the pivotal role that relational work plays in the transformation process. Healthy and mutually supportive relationships can be valuable outcomes in their own right, especially when participants have suffered from a lack of meaningful relationships in their lives. Equally important, achieving programmatic outcomes often depends on an organization’s ability to enable high-quality relationships. Such relationships include those that connect frontline staff members with participants, those that connect participants with one another, and those that connect participants with members of their social network. A critical component of these relationship-building efforts is the attitude that staff members bring to their work. It has long been known that in many human service activities—teaching, therapy, and social work, for example—frontline workers’ attitudes toward participants are a significant driver of change.9
Allow variation | The ebb and flow of participants’ lives, the necessarily improvisational nature of frontline adjustment work, and the desire to facilitate creative problem-solving all point to a need for outcome measurement models that give space for variation. Take the example of substance-abuse treatment. Although abstinence might be the desired long-term outcome for people with a substance-abuse problem, the short-term outcomes that signal progress toward that goal can vary. Outcome measurement models that allow for this kind of variability enable treatment professionals to discover patterns of individual change and to adjust their services accordingly. In some cases, long-term outcomes will vary as well. In drug rehabilitation, for instance, “harm reduction” can serve as an alternative goal for participants who are unable or unwilling to practice abstinence.
Respect agency | Too often, standard outcome measurement models treat clients as if they were merely inputs to programs. Instead of treating participants as objects of intervention, however, an outcome evaluation framework should acknowledge the codetermination work that they pursue in collaboration with staff members. As we have noted, supporting participants in their progress toward greater self-determination is a critical yet often invisible aspect of frontline work. In some nonprofit organizations, participants actually exercise significant agency not only in the design and delivery of the services that they receive, but also in organizational governance. Some existing outcome measurement models do give scope to participants’ “cooperation” with, or “acceptance” of, an intervention. But outcome measurement models should account for the self-determination of participants as well.
Support collaboration | In a traditional outcome measurement model, nonprofit leaders generally have no way to assess the value of the linking work that staff members perform in order to connect participants with resources that exist outside the boundaries of a given program. Nonprofit workers, as we have seen, frequently must collaborate with colleagues both inside and outside their organization. What’s more, the linking work conducted by frontline staff members mirrors the work that participants themselves must do as they piece together various services to address a particular problem. An outcome measurement framework should focus on how a program fits into the life of a participant—not how the participant fits into the program. By recognizing that a program is one factor among many, we can better understand how services work (or fail to work) within a larger ecosystem.
If we fail to situate programmatic activity within the context of frontline work, we cannot reliably measure nonprofit effectiveness. We have identified four reasons why this is so. First, non-programmatic work is instrumental in achieving program outcomes. By motivating participants or adjusting each program to particular needs, staff members help ensure that a program will actually work for those who take part in it. Second, non-programmatic work involves important outcomes that are separate from program outcomes. For participants, these outcomes might include a decrease in feelings of isolation or an increase in feelings of efficacy or self-confidence. Third, non-programmatic work is a necessary prerequisite to defining appropriate outcomes. If frontline workers don’t listen carefully to the issues that participants bring to them, and if they don’t systematically explore how various resources might help to confront those issues, then they cannot properly determine the most suitable outcomes for participants. Fourth, non-programmatic work enables staff members to appreciate the kinds of micro-transformations that the blunt instrument of a program-based measurement model will often fail to capture. For one client, a job secured might be just another step in a long journey; for another, it might be a life-changing moment.
Emerging Alternatives
In our research on the evaluation of nonprofit effectiveness, we have encountered several organizations that have undertaken innovative efforts to capture the role that frontline work plays in driving participant outcomes. Here are a few examples that point the way toward an outcome measurement model that honors the full breadth of the work that contributes to transformative change in individuals and communities.
Measuring relational work | Roca, an organization based in Chelsea, Mass., that helps young people move out of violence and poverty, recognizes the significance of relational work. The organization works with 17-to 24-year-old men who are involved in the criminal justice system, who have no employment history, and who face a high likelihood of re-incarceration. The leaders of Roca put a premium on the transformational relationships than can emerge between young people and frontline staff members, and for that reason they have developed a performance measurement system that tracks important relationship milestones. Such relationships, they have found, are what enable staff members to know when and how to challenge a young client as a means of supporting behavior change. Staff members at Roca also recognize that relapse and even “failure” are a part of the transformational process, and they focus on addressing problems of that kind in the context of a solid relationship.10
Accounting for adjustment work | The Full Frame Initiative (FFI), based in Greenfield, Mass., partners with government agencies, philanthropic organizations, and human service nonprofits that are working to break both the cycle of poverty and the cycle of violence. FFI has identified a set of principles and practices that it calls the Full Frame Approach. This model, which FFI leaders believe is characteristic of effective organizations that work with people who have multiple challenges, raises several problems for traditional evaluation methods. For one thing, it is highly tailored to the specific situation and trajectory of each participant. For another, it allows program activity to evolve over time and within a community context. For still another, it avoids the tendency to see participants as “clients in a program,” and instead treats each program as part of an ecosystem of resources. FFI has created individual assessment tools, and it is developing new methods that will enable other organizations to track and evaluate their work. The core FFI method focuses on tracking changes in five crucial aspects of life (FFI calls them the Five Domains of Wellbeing): social connectedness, safety, stability, mastery, and meaningful access to relevant mainstream resources.11
Recognizing codetermination work | Liberty Hill Foundation supports nonprofit organizations in Los Angeles that promote grassroots social change. Its grantees work simultaneously to build leadership capacity within marginalized communities and to change the policies and practices that contribute to that marginalization. In partnership with its grantees (and with help from an outside consultant), the foundation has developed an outcome measurement model that accounts for the codetermination work that grantee organizations pursue with their target populations. The goal is to overcome certain limitations of the traditional outcome measurement framework. First, that framework tends not to recognize the way that members of a target population help define and carry out the work that grantee organizations do. Second, the traditional focus on short-term program outcomes encourages organizations to work mainly with more-experienced members of a target population—a practice that undermines a core mission of these organizations, which is to cultivate new leaders.12 The Liberty Hill model, therefore, considers two kinds of outcomes: one that focuses on external policy change, and one that emphasizes leadership development.
Acknowledging linking work | The Women’s Funding Network, a philanthropic collaborative based in San Francisco, supports initiatives to meet the needs of women and girls. Over the past several years, the organization has developed an evaluative framework called Making the Case. The purpose of the framework is to help network members and their grantees to design, implement, and evaluate social change initiatives. Currently, 40 funders around the world are using Making the Case with their grantees. The framework uses five indicators of social change to measure organizational effectiveness: “a shift in definition” (how people frame issues); “a shift in behavior” (how people act within a community); “a shift in engagement” (how people pursue change); “a shift in policy”; and “maintaining past gains.” A notable feature of Making the Case is that it takes into account the importance of what we call linking work. Within the “shift in engagement” category, for example, Making the Case tracks alliance building. Here the framework calls on grantees to show whether they have engaged in activities such as forming coalitions, bringing public- and private-sector organizations together, and encouraging institutions to share information. The framework then asks grantees to describe the outcomes for each activity.13
People at these four organizations are doing their part to transform the practice of outcome measurement. But leaders and scholars in the nonprofit world must devote more systematic attention to the challenge of developing evaluation models that fully account for frontline work. The point here is not to burden frontline staff members with additional data-collection demands or to justify micro-management of their efforts. Instead, the purpose is to recognize the full scope of the work that they do. We need to understand the conditions that affect this work, the kinds of skills that it requires, and the range of outcomes that result from it. The new framework that we envision retains the focus on performance that has long been central to nonprofit evaluation: Did this activity help to improve the lives of its intended beneficiaries? But it challenges the assumption that programs alone enable improvement.14 Evaluators and evaluation researchers have spent several decades refining techniques for measuring program performance. Now it’s time to make a similar investment in understanding how frontline work contributes to participant outcomes.
Support SSIR’s coverage of cross-sector solutions to global challenges.
Help us further the reach of innovative ideas. Donate today.
Read more stories by Lehn M. Benjamin & David C. Campbell.