Why Measure?

“Everywhere you turn in the world of philanthropy and nonprofits these days, people are talking about accountability,” stated the lead sentence of a recent New York Times article that reported on the need to measure nonprofit effectiveness. ¹ Indeed, the buzzwords of “transparency” and “accountability” contribute to making performance measurement one of the hottest topics in philanthropy.

Proponents of measurement believe that philanthropists need to apply the same discerning eye to charities that they would to a stock investment, and that one way to do so is for the charities to measure the success of their programs. ²

Against this backdrop, we began a recent research study not just with the belief that performance measurement could make a meaningful contribution to the effectiveness of nonprofit organizations, but also with the hypothesis that others would share this view. We did not set out to test this hypothesis because we felt its truth was self-evident. Why wouldn’t donors be supportive of performance measurement? Instead, our goal was to contribute to the design of these metrics by asking donors to describe their ideal performance measurement tool. In short, we began with an acknowledged bias in favor of the wider use of performance measurement.

We decided to focus our research on individual donors, as opposed to institutional funders such as foundations, for two reasons. First, we followed the “Willie Sutton rule.” (When asked why he robbed banks, Willie Sutton once replied, “Because that’s where the money is.”) In 2001, individual donors accounted for 76 percent of total charitable giving in the United States, for a total of $161 billion.³ Second, we expected the need for metrics to be stronger among individuals. While institutional funders often have one or more evaluation professional, individuals are left to their own devices to assess the effectiveness of different organizations. We further narrowed our focus to relatively sizable donors – generally more than $50,000 in giving per year – in order to engage with philanthropists who make gifts likely to have a sizable impact on a nonprofit’s budget.

Ultimately, we interviewed 22 individual donors. Collectively, these donors give roughly $50 million per year to charity. Almost all the interviewees were businesspeople who earned their wealth in finance and investing. Overwhelmingly, they were middle-aged male caucasians living in the Boston or New York City areas.⁴

As we planned our research, we believed these interviewees occupied the “sweet spot” for performance measurement. They had become successful in the world of finance, where analysis is critical, where objective and transparent performance measurement happens on a daily – or even minute-byminute – basis, and where decisions are based upon concrete data. Performance measurement is in their blood, we reasoned. We set out to learn the precise form in which they wanted to see these metrics carried to the nonprofit sector.

We never got that far. Only four of the 22 interviewees were strongly interested in getting better data on the performance of nonprofit organizations. Much to our surprise, the rest expressed skepticism – or even outright disapproval of the concept.

Why this aversion to performance measurement, particularly given the group’s professional background? We identified five major findings that explain their opposition. Each of these findings has implications for what providers of metrics should do in order to overcome a donor’s misunderstandings and objections.

No Rewards Yet for High-Performing Nonprofits

Generally speaking, performance measurement seeks to reflect the achievements of an organization through the use of quantitative indicators across a variety of dimensions, including financial, staff, operational, and impact. For instance, Jumpstart, a group that organizes college students to tutor preschoolers in low-income neighborhoods, uses performance measurement to track various levels of service delivery.⁵ It tracks everything from growth in the number of volunteers (from 15 to almost 1,600) to change in children’s skills levels (students that started the program behind their peers caught up despite being younger on average). Nonprofits like Jumpstart use performance metrics to measure their own progress against a variety of goals; they also try to use them as a selling point to donors.

Proponents of the use of metrics in the nonprofit world typically make three main arguments. First, metrics can be a useful tool for management by enabling nonprofit executives to identify improvement opportunities. Second, metrics could enable donors to understand the relative effectiveness of different organizations, and therefore shift their dollars to the best performers. Finally, by providing greater transparency and accountability, metrics might increase overall donor confidence in the sector and thus increase the amount of total giving. “I have no doubt,” says Joel Fleishman, a professor at Duke University School of Law and former president of the Atlantic Philanthropies, “that both the second and third of these would occur if the information is available and reliable.”⁶ Dick Spangler, former president of the University of North Carolina, shares this sense of optimism. “The common sense quotient among American donors is extremely high,” he says, “and this could be enhanced by reports that increased the transparency of the financial [and other] affairs of nonprofit organizations. Transparency should build confidence and increase giving.”

Edward Skloot, executive director of the $675 million New York-based Surdna Foundation, offers four explanations for the emergence of the measurement trend: “First, the bursting of the stock market bubble of the late 1990s reduced the amount of donor capital, thereby encouraging donors to be more discerning in their giving. Second, the emergence of ‘venture philanthropy’ has contributed to greater use of measurement tools previously reserved for the private sector. Third, the nonprofit field has seen an influx of new faces, bringing with them management tools in wide use elsewhere. Finally, government officials and journalists have discovered the sector and are turning their attention to it – including assessing its performance.”

But do individual donors apply such a keen eye to those organizations to which they decide to give? And will they perhaps do so in the future as more information becomes available?

It would stand to reason that more effective nonprofits should be able to attract more funding to increase the scale and scope of their activities. Unfortunately, reality doesn’t follow that logic. Jumpstart, for instance, has seen how little its track record is valued by donors. “The connection between Jumpstart’s success at demonstrating impact and its ability to fundraise is at best tenuous,” complained Aaron Lieberman, the co-founder. “We double every single year, we get better impact measurements, and still no one ever comes back to us and says, ‘Hey, you guys are doing so great, we want to give more.’”

Why has Jumpstart’s success at measuring performance gone largely unrewarded? Is the absence of a link between metrics and fundraising success specific to this organization, or is it a more general problem? Why might funders not be interested in applying metrics to their giving decisions, and what are the implications for organizations promoting the use of metrics? The nearly 60 interviews we conducted with individual philanthropists, institutional funders, nonprofit executives, and others suggest answers to these questions. The objections to performance measurement highlight the fact that performance measurement proponents need to go beyond the theoretical value of measurement; they need to change fundamentally the way people think about and give to nonprofits.

Finding No. 1: Donors Do Not See a Need for Performance Measurement

Over 115 years ago, pioneering rags-to-riches steel magnateturned- philanthropist Andrew Carnegie lamented that his fellow millionaires were squandering their money on unworthy charitable causes. “It is ever to be remembered that one of the chief obstacles which the philanthropist meets in his efforts to do real and permanent good in this world,” Carnegie said, “is the practice of indiscriminate giving.”⁷

Millionaires in Carnegie’s time may have been indifferent as to whether they were giving to the indigent or to public libraries; we found that several of today’s individual high net worth donors were indifferent to the effectiveness of the particular organizations they fund. “Virtually all giving goes to a good cause,” says philanthropist Russ Carson, head of the private equity firm Welsh, Carson, Anderson & Stowe. Seth Klarman, head of the Baupost Group, agrees, noting that in the absence of fraud, the act of giving is substantially more important than attempting to differentiate between the most efficient organizations and other groups. “Unless you’re being taken,” he says, “society’s interests are advanced.”

It is no wonder, then, that the argument we heard most often from donors against the use of metrics was the idea that measurement tools were fundamentally unnecessary. In order to be interested in measurement, donors would need to believe that there is a substantive difference among organizations. In other words, it is important to track performance if and only if you expect to find that one organization is better than another. By and large, our interviewees did not expect this to be the case.

In fact, this select group of business philanthropists that we interviewed acknowledged that the standards for their philanthropic decision making diverge from their daily work. “Once I’ve gotten beyond an assurance of efficiency – that the organization is not running a deficit – and as long as the staff can articulate that they are meeting their goal, I don’t apply the same rigor,” admitted a managing director at Morgan Stanley.⁸

Donors’ second reason for why performance measurement is unnecessary is their reliance on personal connections in making giving decisions. Metrics serve no purpose, the donors argue, where they have a trusting relationship with either a nonprofit executive or a board member. “The reality of the giving is that it has nothing to do with results measurement; it has everything to do with having a personal connection,” said Joanna Jacobson, a former corporate executive who now leads Strategic Grant Partners (SGP), which brings together philanthropists to attack the root causes of intractable problems. The head of a major private equity firm concurs: “There is a huge reliance on the board or on the trustees.”

Although we believe better assessment will improve the ability of the nonprofit sector to serve clients – by helping shift donor capital from less-efficient to more-efficient organizations and by strengthening donor confidence, bringing more overall dollars into the sector – these arguments centered on “efficiency” simply don’t resonate with donors, especially when matched against the impact of a strong relationship to an executive or board member with a good story to tell.

Developers of performance measurement systems must translate the “efficiency” argument into an appeal to the psychological motivations of donors. Spurring individual donors to look at performance measurements before they write a check requires an appeal to their desire for greater social impact. One way to do this is to link improvements in performance measurement to social impact. Metrics alone may be a tough sell, so an organization seeking funding should complement metrics with anecdotes about the impact and success of an organization or a particular program. Such an effort must go hand in hand with any efforts to further promote the use of a performance measurement system.

For instance, Big Brothers of Massachusetts Bay (BBMB) has persuaded individual donors to give more because it was able to link performance metrics with impact on children’s lives. “Some donors give based upon an emotional connection – their nephew was a Big Brother, for instance,” says John Pearson, president and CEO of BBMB. “But what we’ve found in terms of other donor segments, especially some large donors in the Boston market who work in equity investing or venture capital, is that they respond better when we can demonstrate accountability. We booked an appointment with one donor with whom we expected to get 20 to 30 minutes. But he was so intrigued by the information we were able to provide through our metrics that he spent an hour and a half with us.” The donor asked questions about BBMB’s volunteer yield rate and Pearson was able to provide him with the numbers broken down by marketing channels – the number of volunteers attracted by direct mail or Web site, for instance. “Our monitoring and evaluation of our outcomes impressed him and others like him who invest in companies and understand the importance of good management,” Pearson says.

Because they know how each level of their organization is functioning, BBMB officials can give concrete answers to donor questions like “What will you accomplish for children if I gave you $10,000?” BBMB officials are able to tell donors, for instance, for $50,000, they can reach 150 new volunteers. And when there may be a particular shortage of African American male volunteers, they can tell their donors that to reach this urban segment, it costs $550 to pull in each of the necessary volunteers. “By showing that good management of the program improves volunteer yield, improves cost effectiveness, and, after one year with a volunteer Big Brother or Big Sister, a 52 percent reduction in truancy, we can draw a fairly direct line of dots going from funding of the program to the benefits to the individual child,” says Pearson.

Finding No. 2: Donors Do Not Have Time for Performance Measurement

Many interviewees had a simple explanation for their lack of interest in metrics. They simply do not have the time or energy to look that closely at their grantees’ activity. Our conversations suggested that these donors see their philanthropic activities as an escape from their intensive work environments. Even simply reading and attempting to interpret metrics can be a time-consuming exercise that is difficult to stomach in light of the extensive professional and personal commitments many donors already face.

“I have a day job. … There just aren’t enough hours in the day,” admits Carson. “I’m used to making fast decisions and I’m perfectly happy to make decisions based on incomplete information.” Professional fundraisers make the same observation. “One of the problems is that people don’t spend enough time on their philanthropy,” said Bruce Temkin, director of development for the National Council of Jewish Women. “It’s the rare donor that does. They know where their 401(k) is at any given time, but they don’t know how much they give.” Adds Lieberman, “They really don’t like to be hardheaded, because that’s what they do in their day-to-day life.”

Measurement providers need to engage a broader campaign to educate donors about the time and effort that is required of a philanthropist committed to real social impact. In some ways, this echoes Carnegie’s philosophy that the “duty” of the wealthy is “administering surplus wealth for the good of the people.” Carnegie saw millionaires as “stewards,” whose responsibility was to use their experience and judgment to distribute wealth for the most public good. “The duty of the millionaire,” he exhorted, “is to resolve to cease giving to objects that are not proved clearly to his satisfaction to be deserving. … Greater good for the human race is to be achieved by inducing them to cease impulsive and injurious giving. As a rule, the sins of millionaires in this respect are not those of omission, but of commission because they will not take time to think and chiefly because it is much easier to give than to refuse.”

Then, just as today, individual donors should be reminded that philanthropy is important and serious work. It does require time and thought. Nonprofit leaders can help by educating donors about differences in efficiency of operations and service delivery between high- and low-performing organizations. For example, a nonprofit could conduct an analysis comparing its performance to that of other organizations with a similar mission and programs based on a set of clearly defined and consistent metrics. Creating a comparison table of these metrics could inform donors as to which organization is performing more efficiently and having greater impact. This would also convey the more fundamental message that society could be better improved by directing money toward better-performing nonprofits that would otherwise go to poorly performing ones.

Finding No. 3: Donors Do Not Have Confidence in Performance Measurement

Some donors in our study noted their frustration with the current giving model and argued in favor of objective measurement and even comparisons. Said real estate broker and philanthropist Julien Studley, “A rating system, with all of its faults, may have some benefit.” However, many more of the donors we spoke to were concerned about the difficulty of quantifying a nonprofit organization’s impact – especially since nonprofits often seek general and qualitative outcomes such as building better communities and improving people’s lives. The significance of such outcomes is difficult to capture accurately in a finite set of metrics. Ron Daniel, former managing partner of consulting firm McKinsey & Company and the recently retired treasurer of Harvard University, argues that metrics “have the appeal of providing measurement, but they often don’t convey the essence of the phenomenon which you are trying to judge.”

In general, donors seemed skeptical of the ability of purely quantitative metrics – especially financial data – to reflect performance, particularly in regard to quality of service delivery. Klarman summarizes the concern of other donors when he derides most metrics as “specious precision.” Carter Bales, head of the Wicks Group, a private equity firm, worries especially about the completeness of such information. “What are the unknown unknowns? What did the writer leave out of the report?” Bales fears that any performance measurement system would focus on areas that are measurable, rather than those that are important – creating what he calls “the fallacy of focus.” He feels that metrics will undoubtedly include and overemphasize easily measurable items like financial efficiency, such as overhead ratio.

An organization that has a slightly higher overhead ratio is not necessarily spending its funds less wisely. It may be helpful to vary the method of evaluation by the stage of the organization. For example, if the goal is to accelerate the program for effective replication, metrics may be more relevant, while if the goal is to prove the success of an unproven idea, then metrics may be less relevant.

For performance measurement to be accepted, providers should find a way to incorporate qualitative depictions of organizational performance and impact, rather than simply capturing financial ratios. Options include expert evaluations, peer review, client satisfaction surveys, client testimonials, board reviews, and media coverage. For example, peer review circles could allow leaders in each field to offer their opinion about a set of organizations, which might divulge new and important information to a donor. Peer assessment could also be used as part of a performance system. Jumpstart’s Lieberman is open to ranking nonprofits, so long as the rankings incorporate peer review, much like U.S. News and World Report’s college rankings include the view of school presidents.

The co-director of one foundation believes both data and anecdotal information highlighting the more qualitative aspects of an organization’s programs are needed in order to make a funding decision. Anecdotes about the lives of clients served, or snapshots of how their lives have been changed by a nonprofit’s service, can make a powerful and lasting impression on a donor.

Finding No. 4: Donors Do Not Want to See Nonprofit Resources Dedicated to Performance Measurement

The philanthropists we interviewed feel that measurement may be a poor use of scarce resources – including time and money. Except for organizations where the metrics are easily tracked, “I don’t think it’s worth spending the time and money on it,” said Klarman. “Charities are small organizations with a million issues.” Many organizations, he notes, pursue goals that defy easy (or inexpensive) measurement. Venture capitalist Laurie Thomsen observes the same sentiment with many donors. “I don’t think people want [nonprofits] spending money on measurement,” she says. Moreover, at least one interviewee felt that aggressive performance measurement was likely to threaten the relationship between the donor and the nonprofit, at the expense of the organization. “There have been attempts on the part of individual philanthropists to monitor performance, and they have ended in misery,” says one donor.

To justify the costs of measuring performance, then, it is critical to assuage donor concerns about the demands on nonprofits by making these organizations allies in the drive for more performance measurement. Nonprofit organizations need to be enlisted as partners. Their endorsement is needed to help donors see that nonprofits stand to recoup the time and resources used in performance measurement. Several nonprofit executives we interviewed explained their willingness, and even eagerness, to be evaluated so long as the metrics meet three criteria (sidebar, p. 48). First, the metrics must provide insight that is useful internally, to the managers of the organization. Second, there must be coordination among funders (including the government) to reduce the varied and often conflicting demands for data that pour in from donors. Lastly, there must be a source of funding to enable nonprofits to collect and manage the data required.

Finding No. 5: Donors Look to Institutional Funders to Engage in Performance Measurement on Their Behalf

Even those individual donors who expressed their belief in the need for greater nonprofit accountability noted that so-called “umbrella givers” or intermediaries already exist to conduct measurement and provide a screening mechanism for donors. Individual philanthropists feel strongly that institutional funders such as foundations and venture philanthropy organizations have the resources and expertise – human capital, in particular – to more efficiently and effectively assess the performance of nonprofit organizations. Understandably, many individuals are willing to “outsource” performance metrics to institutional funders that can act, in the words of one donor, as a “subcontractor.” Many donors prefer to work through these intermediaries, rather than conducting their own duplicative assessments.

Donors further perceive that foundations feel more accountable and are thus more careful about their giving decisions because they are giving money away on behalf of their funders or board. According to Fleishman, “It is their job to care about this, to make sure that the opportunities make sense.”

And indeed, many of our interviewees placed their confidence in the expertise and judgment of foundations. Our interviewees looked to organizations like the Boston Foundation or Combined Jewish Philanthropies in Boston, and to the New York Community Trust or the Robin Hood Foundation in New York for reliable information or to distribute their philanthropic dollars for them. Institutional funders like these can offer donors a “seal of approval.” Donors currently hear about whether institutional funders are conducting performance measurement through various channels – the funder’s newsletters, for instance, as well as through newspaper articles and other media coverage. Institutional funders can also continue to improve their communication of performance measurement information by making it even easier for individual donors to access this information. The metrics must make sense to the donors and shouldn’t be limited to the “latest and greatest” figures, but rather ones that are reliable and practical. Brevity is also critical, as donors have limited time. Institutional funders can also provide more detailed information on their Web sites, increasing the reach and frequency of any print and electronic publications. Doing so should allow individual donors to easily ascertain whether or not an institutional funder has given its seal of approval to a nonprofit and, if so, to what extent performance metrics were considered.

Two Steps Forward, One Step Back

The message from the individual donors we interviewed was loud, clear, and consistent: at best, a lack of enthusiasm for performance measurement; at worst, outright disdain for the idea. Though our sample was not statistically significant, both the view and the reasons offered were consistent enough to raise serious questions for proponents of greater use of metrics in the sector — especially those seeking to offer standardized measurement tools for use by donors.

Should this be the death knell for metrics? Should proponents of metrics close up shop and direct their energies elsewhere? We believe the answer to these questions is “no.”

While there are many donors who say they are opposed to measurement, we believe that by using our recommendations, we can marry the rational and the emotional motivations for donors to contribute to the sector. Our hope is that a better understanding of individual donors’ current opposition to metrics will inform the approach to performance measurement systems going forward. Following our recommendations may help those developing performance measurement systems gain ground in this effort to change attitudes and practices in the nonprofit sector.

1 Christensen, J. “Exploring New Ideas for Making Finances Clearer and Scandals Rarer,” The New York Times (Nov. 17, 2003).

2 See, for instance, Meehan, W.; Kilmer, D.; and O’Flanagan, M. “Investing in Society,” Stanford Social Innovation Review (spring 2004).

3 Giving USA Annual Report for 2001, American Association of Fundraising Counsel.

4 We also interviewed 14 representatives of institutional funders, 10 executives at nonprofit organizations, and 13 other active participants in the nonprofit field (e.g., academics, writers, consultants).

5 Grossman, A. “Jumpstart,” Harvard Business School case 9-301-037 (May 17, 2002).

6 Unless otherwise specified, all quotes are from interviews conducted in person or by phone by one or both authors.

7 Carnegie, A. “The Best Fields for Philanthropy,” North American Review (1898).

8 Unattributed quotes are from those who requested anonymity.

KATIE CUNNINGHAM is the director of national development of Teach for America.
MARC RICKS is a senior policy advisor for the city of New York. They conducted this research as students at the Harvard Business School, with the guidance of professor Allen Grossman and Andre Dua, a consultant at McKinsey & Company.

Read more stories by Katie Cunningham & Marc Ricks.