In Numbers We Trust

(Photo by iStock/naihei)

It’s a bluebird day, high in the Himalaya, sometime in the late ’90s. The air is bracing, but the morning sun is warm on my skin. The North Ridge of Mt. Everest glistens on the skyline, and there is a village in the distance so perfectly placed that it seems an essential part of the landscape. Everything’s beautiful, and my friend Bill and I are standing in the sand outside our Land Cruiser, yelling at each other.

Bill was the founder of an organization with a cool idea about how people in these high mountains could thrive on their own terms in the face of modernity. We’d spent the better part of three weeks roaming the region. I’d gotten to know—and admire—the senior team members, and I’d seen their work in a bunch of different villages. We’d hiked behind yaks into stunning hidden valleys, downed endless cups of butter tea, and talked into the night with grizzled village elders. We’d come upon a snow leopard kill at the base of a mile-high mountain wall and stood in awe looking at ancient murals in remote monasteries. It was one of the best trips of my life.

On the last day, we stopped to bask in the light and scenery. I was stoked, and Bill knew it. He turned to me and asked: “So what do you think? Are you going to fund us?”

Taken aback, I sputtered. “I dunno, Bill, I haven’t seen any numbers yet.”

Things went downhill from there. I had the qualitative stuff. I understood the work and context. I’d heard the stories, and I’d met many of the protagonists. Now I needed to fill out my understanding with the quantitative stuff, with all the numbers that communicate the quality of implementation and what changed as a result of their activities. Bill was more than a little annoyed. After all, I’d been privileged to see and do, why didn’t I believe him? Didn’t I trust him?

Well, sure. I understood the work and believed it had great potential, but the project was already a few years in, and it seemed like Bill should have some implementation and impact numbers—first and foremost for himself and his organization. Bill didn’t take well to hearing this from some guy who’d never built anything himself, and he didn’t hold back. I didn’t either, and so there went a perfectly good bluebird day.

Things didn’t get better in subsequent conversations. It was back at the dawn of “venture philanthropy,” there was a wave of new funders whose reach exceeded their grasp, and perhaps doers were more irritable about funders than usual. Eventually, Bill got so pissed off that he sent a letter to my board “terminating the funding relationship.” I was mortified. The board was mostly puzzled.

The whole thing sounds kind of silly now, but at the time it was quite painful. Bill remains, to this day one of the most creative and accomplished people I’ve known (and of course that’s not his real name). This wasn’t our first trip together, and he’d become a friend and a mentor of sorts. I take friendships seriously and felt it keenly when it all went to hell.

We eventually patched things up—we even funded him, in the end—but my obsession with metrics (and Mulago’s) has its roots in this squabble, and in what it revealed. For me, one of the most important roles that metrics—numbers—can play is to nurture and maintain solid and satisfying relationships. There’s nobody I like and admire more than those who’ve taken on the work of making the world a better place. It’s a joy to spend time with them, and I’ve always hated the idea that my role as a funder precludes friendships with those we fund (or don’t, for that matter). But here’s the thing: A healthy friendship requires a shared reality, and in the case of social entrepreneurs, that means a shared reality in terms of the work, and that requires both qualitative and quantitative knowledge. You need to see the work, understand the ideas, and hear the stories, but the achievement of a shared reality requires metrics that capture delivery and impact.

Bill and I didn’t get to a shared reality—in the end, all we could do was argue. More than ever, I see my job as a funder as getting to that shared reality, and it’s on me to do the necessary work. I don’t get to roam around for weeks in the Himalaya anymore (at least not while working)—we have a small team at Mulago, and while we think it’s essential to see the work and the team on-site, we have 60+ portfolio organizations and 40 social entrepreneur Fellows at any point in time. No matter how hard we work and travel, we’re lucky if we can manage three days on-site, much less three weeks. Numbers matter more than ever.

And there is an even more fundamental reason we need numbers: We are human. Never mind sharing reality; we have a hard time just grasping it. Our intuitions are unreliable, and we’re prone to a bedeviling bunch of cognitive biases. First off there’s optimism bias, without which founders would never found anything. And then there’s confirmation bias—the tendency to pay more attention to those things that confirm your beliefs—and motivated reasoning—the tendency to argue yourself toward a pre-ordained conclusion. Those three biases alone stack the deck for considerable overestimation of impact, and they’re only the tip of the iceberg. It’s chaos in there! The very things that make you human lead you to distort reality, but numbers help you make the most of your good intentions. There’s a reason why the effort to extend rigorous evaluation to the social sector led to a Nobel Prize.

If you’re trying to save the world you need numbers—metrics that are selected carefully and gathered reliably. Not too many, just enough to know accurately:

1. Delivery: what you did.
2. Behavior: what people did differently as a result.
3. Impact: what material change came about as a result of that behavior.

If you don’t have those, you’re flying blind. You don’t know what you’ve accomplished, and you don’t have the information you need to get better at what you do.

And this brings us to the “trust-based philanthropy” movement, which represents, among other things, a reaction to oppressive and unproductive reporting requirements. I get it—some of the thoughtless and/or clueless stuff I hear about makes my eyes bleed. However, this justifiable indignation too often morphs into a weird, retrograde hostility to numbers. I see a lot of it: In a trust-based philanthropy event at UNGA, somebody shouts “the hell with metrics” to general applause; on a phone call, an influential leader tells me that local organizations shouldn’t be expected to measure impact and that they should replace numbers with video clips and stories; and, the admirable Trust-Based Philanthropy Project says that we shouldn’t use “jargon that may alienate or exclude certain organizations,” including such outlandish things as “scalability,” “theory of change,” and of course “quantifiable metrics,” which I guess would preclude the use of something so exotic as numbers.

Among other things, this is more than a little condescending. This work is complex, yes, but it’s not quantum physics. Anyone clever enough to start an organization and do good work can and should master these concepts, including the necessary and productive use of “quantifiable metrics.” We acknowledge that organizations must master complicated accounting methods to track how money is used, but we somehow fail to extend those reasonable expectations to quantitative methods tracking delivery and impact. As a Kenyan founder friend said to me after “the hell with metrics” outburst, “What, they think we’re too dumb to measure things?”

An emphasis on numbers is critical if we’re serious about creating a new generation of effective leaders who do not come from rich countries. Numbers are the closest thing we have to a universal language, and as such they contribute to a more level playing field between doers and funders. Numbers work the same in Zambia and California: They privilege effort over origin and clear thinking over charisma. We have work to do to make measurement methods more accessible, and as funders, we need to ensure that leaders have the resources they need to master and apply those methods.

Mulago has been doing most of what is described as “trust-based” philanthropy long before it was a thing: For two decades we’ve done unrestricted, long-term funding, and instead of requiring proposals, we do our own homework to understand an organization’s model, strategy, and delivery. It’s heartening to see an approach we’ve advocated for years become part of the mainstream, and so it puzzles me that some trust-based philanthropy proponents talk as though reporting requirements themselves are inherently wrong. One influential foundation says it “is still asking recipients for progress reports but no longer prescribes what’s in them,” which presumably means that if the funded organizations don’t feel like talking about impact, they need not worry themselves further. To me, that seems like dereliction of duty. As funders, it’s our responsibility to assure real change for the better in the lives of those we all are trying to serve. It is our job to make sure that the road so notoriously paved with good intentions actually leads to a better life. It is our job to make sure that predictable human frailties don’t lead to a distortion of what has and has not been accomplished. Nobody’s infallible, and those who’ve been denied equal opportunities in life deserve to have funders looking out for them too.

And if doers are measuring the things it takes to do right by the people they serve, then reporting requirements become more like sharing requirements. If funders are smart about what to ask for and doers are assiduous about what they gather, then there won’t be much daylight between what the one wants and the other already has.

I’m still sad about that ruined day in the Himalaya, but not about where it led me. I treasure the stories, the ideas, the people, and the places that are part of this work, but they can’t be fully understood and appreciated without numbers. The work isn’t fun without trust, but trust comes from a shared reality, and a shared reality can only happen when the numbers anchor the stories, and the stories make the numbers sing.

Read more stories by Kevin Starr.