Moving Beyond Vanity Metrics

“Right now, our metrics bias us towards recruiting the most new members, not the right new members.” That’s what one senior executive at a social change organization told us recently, and she’s not alone in her frustration. “Vanity metrics”—raw numbers totaling the number of times people view a webpage, for example, or the number of email subscribers—are widely used, and by definition, impressive and easy to access. But they offer questionable value to member-focused organizations that rely on people power to achieve impact.

What these organizations need are targeted metrics about which members are most engaged and where the potential for greater engagement lies—metrics that will help them make better decisions, to achieve greater impact.

Many social change leaders know this. And yet, they have not left vanity metrics behind. Even those who embrace “lean startup” principles and regularly make data-driven decisions can find the prospect of gathering and analyzing more targeted, helpful information daunting. Sometimes, they even feel as if their hands are tied. Many existing measurement tools aren’t suited to the task. Funders often demand “topline” reports.

Attempting to provide a solution, we researched the issue, asking our colleagues and others in the field of member-based nonprofits some tough questions about measuring engagement. We detailed our findings in a new report by the Mobilisation Lab at Greenpeace and Citizen Engagement Laboratory entitled, “Beyond Vanity Metrics: Toward a Better Measurement of Member Engagement.”

We didn’t find a “silver bullet” metric that will work for every organization. But we did surface four insights that we believe nonprofit leaders should consider as they strive to get the most out of their data-gathering activities:

1: Rates trump aggregates.

An aggregate figure may tell you how many supporters you have. But a rate—such as the number of actions per 10,000 members, for example, or actions per mailing—can help you understand how active and engaged those supporters are. Rates are also more difficult to manipulate compared to aggregate numbers, so they are less likely to deceive those who study them. (Sending many emails to increase the total number of actions, for example, would decrease actions per mailing. A rate metric would reveal that; an aggregate measure of total actions would not.)

Fortunately, rates are relatively easy to track since it’s already common practice to look at rates for individual emails. So for many organizations, starting to use rates as a metric shouldn’t be too difficult.

2: Tracking cohorts reveals valuable trends.

Many organizations consider all members as one unit, but breaking up that unit into smaller groups of people who share a common characteristic over a certain period of time—a cohort—can reveal valuable information. Cohort analysis helps organizations understand the changing composition of their membership and monitor member lifecycles. Armed with that understanding, an organization can focus on growing particularly valuable cohorts.

The “MeRA” quotient is among the best new examples of cohort analysis in the digital campaigning sector. The corporate accountability campaigning group SumOfUs uses this method to track the number of existing “members returning for action” on a monthly and quarterly basis. Another prominent organization we know tracks the composition of various cohorts defined by how many actions members have taken.

One caveat: This approach typically requires data and analytics expertise, so it may be difficult and time-consuming for organizations that don’t have access to necessary staff or technology.

3: Scoring reveals levels of engagement.

Consider the “Klout Score.” (Klout measures how influential an individual is by factoring data points about how people respond to your activities on various social media platforms into a numeric score.) Now imagine a Klout Score for member engagement. This approach assigns points for various actions and activities, weighted by importance, to produce a numeric score reflecting an individual’s overall engagement.

Understanding how active or engaged an individual is gives organizations the information we need to build stronger relationships. Communicating with members at levels that match their commitment translates to greater investments of time and money—and less effort recruiting new members. (Airline frequent flyer programs, for better or worse, represent great examples of relationship management based on activity or scores.)

Many modern databases (or “CRMs”) can be customized to track engagement scores, assuming practitioner know-how and/or a sufficient technology budget. Of course, any such score is only as good as the activity data an organization can collect and its judgment in ranking or weighting activities.

4: Simplicity can be your friend.

A raw, aggregate number can be useful in scenarios where it clearly aligns with an organization’s (or team’s) goals—for example, tracking total volume if there is an explicit connection between volume and the desired outcome. One organization we know tracks the total number of people who attend labor strike rallies; that aggregate is an important metric because the nonprofit can draw a straight line from rallies to greater media influence, negotiating power, and ultimately bigger, faster victories.

The upside of aggregates, as we’ve noted, is that they are easy for everyone to understand. Remember, though, that aggregates can easily become “vanity metrics” with all of their associated pitfalls. Review the aggregates you track regularly to be sure that they are still relevant, and not “torqueing” your perspective.

The challenge of understanding what’s working and what needs work in campaigning and advocacy existed long before the Internet era. While there is surely a mix of art and science involved with all successful campaigns, we are lucky to live in an age where rapidly improving technology allows us to engage more people than ever before, in more-innovative ways than ever before. The promise we seek as leaders in the digital age is to leverage these voices to have social impact at scale. Shifting culture within organizations to regularly discuss and evolve important metrics is one of the most powerful things we can do toward that end.

Read more stories by Jackie Mahendra & Michael Silberman.