By 2013, according to a study by the Pew Research Center, more than 170 US-based nonprofit news outlets had established a presence online. These organizations cover everything from hyper-local issues to matters of global concern. Significantly, more than 70 percent of them came into being in or after 2008.1 This boom represents a new path for media, and it raises a new set of questions for those who seek to understand the impact of these outlets. Whether one is an editor who needs to gauge the real-world ripples of an investigative journalism project or a funder who needs to evaluate the case for supporting such work, access to accurate and meaningful metrics is critical to navigating this nascent industry.
It’s an industry that has risen from the ashes of traditional newspaper publishing. An initial crop of a dozen or so nonprofit outlets—including our organization, Grist—sprang up in the late 1990s, joining well-established predecessors such as National Public Radio and the Center for Investigative Reporting. Just a few years later, the post-millennial implosion of the newspaper business and the explosion of social media led to a sea change in how journalists create and disseminate their work.
(Illustration by Curt Merlo)
Philanthropic support has played a critical role in this transformation. Among outlets that took part in the Pew survey, 74 percent reported that they had secured grant funds to launch or maintain their operations. And the scale of such funding has accelerated. Between 2009 and 2011, foundation support for media grew by 21 percent, compared with a 5.8 percent increase in overall domestic grantmaking, according to a report by the Foundation Center.2 During that period, more than 1,000 foundations made a total of $1.86 billion in media-related grants.
The rapid growth of foundation-supported media makes the question of impact keenly relevant to journalists and philanthropists alike. How can those who operate and fund these organizations measure the full impact of journalistic work? What are the best methods for determining the connection between published content and real-world change? What, fundamentally, is the role of a journalist in the 21st century? Across the United States, efforts are under way to address these questions—efforts that range from newsroom experiments to ambitious research projects. The result is a conversation that can be staggeringly complex and vaguely navel-gazing, and so far it has revealed exactly one truth: There is no easy answer.
Questions, Questions
For nonprofit media, the act of measuring impact is not nearly as straightforward as it is for other nonprofit organizations: There are no trees planted, no cans of soup distributed, no lawsuits won. Existing resources for nonprofits have little to say about media. The IRIS catalog, for example—a project of the Global Impact Investors Network that offers a bevy of options for evaluating work in areas such as banking, health care, and conservation—doesn’t include a media category. What’s more, the numbers held in high esteem by old media—circulation figures and advertising dollars, in particular—have minimal relevance in this new world. Operating somewhere between the mission-driven world of traditional nonprofits and the profit-driven realm of traditional media companies, nonprofit media are a fence-straddling lot.
Many observers have noted a misalignment between traditional metrics and new-media needs. In 2013, the John S. and James L. Knight Foundation issued a report that offered this assessment: “The near-universal perception is that standard metrics … used by nonprofit news organizations are simplistic and often misleading.”3 The view inside those organizations isn’t much different. “The large majority of [nonprofit media outlets surveyed by Knight] feel completely lost when it comes to measuring their impact,” says Jonathan Sotsky, director of strategy and assessment at the foundation.
So what’s a news outlet to do? As a starting point, many of them rely on the same metrics that other Web-based organizations use. These metrics include page views (which count, as the name suggests, the number of times that visitors request a single Web page), unique visitors (a tally of each device that accesses a site over a given time period), and time on site (the length of time that visitors keep a particular site open on their browsers). Data of this kind are relatively easy to access, thanks to widely available tools like Google Analytics. They are wonderfully tangible. But they are flawed. Time on site is especially problematic; a better term for it might be “time on site while intending to read an article but wandering away to put the kettle on, then taking a phone call from Aunt Midge, then—wait, what was I doing?”
Standard metrics tell only part of the story. Yes, it’s vital to know how many people a news outlet is reaching and which links those people are clicking. But other questions are equally important, if not more so: Are people actually reading or watching the content that they access? Are they sharing or commenting on it? Does their engagement with the content spur offline conversation and action? Does that engagement ultimately lead to shifts in public opinion or policy? The answers to those questions are much harder to determine, but they are essential to understanding the impact of a media organization.
Standard metrics tell only part of the story. Yes, it’s vital to know how many people a news outlet is reaching and which links those people are clicking. But other questions are equally important, if not more so: Are people actually reading or watching the content that they access? Are they sharing or commenting on it? Does their engagement with the content spur offline conversation and action? Does that engagement ultimately lead to shifts in public opinion or policy? The answers to those questions are much harder to determine, but they are essential to understanding the impact of a media organization.
Trial and Error
Our experience at Grist offers an instructive example of what it has meant to be a nonprofit news outlet in this brave new millennium. Initially, in the absence of other options, we relied on the existing online metrics to chart our progress. We were thrilled to be able to point to hard numbers: We’ve grown from an audience of 100 unique visitors to an audience of 10,000! 100,000! 250,000! (Today our total monthly audience, including unique visitors and those who interact with us via social media, is close to 2.5 million.)
As our readership mushroomed, we began to focus on another factor that signaled progress toward our goal of shaping the national environmental conversation: influence. We started tracking indicators such as media mentions, awards, testimonials, public-speaking invitations, and interactions with notable decision makers. During our first decade, this suite of metrics offered strong evidence—to our team, to our board, and to our financial supporters—that Grist was having an impact. We were reaching a growing number of people, they were clicking on our links, and influencers were discussing and acting on the ideas and stories that we put into the world.
Given the social mission that underlay our journalism, however, we yearned for more information about how our work was resonating with readers and translating into real change. The occasional anecdote made its way to us—a Grist-inspired debate that took place behind closed doors at the US Environmental Protection Agency, a shift in the farming practices of a Native-American tribe, a clean energy referendum in a US city—and we treasured these bits of qualitative data. In many ways, they told us more about our impact than hard numbers could ever do. But we needed more reliable ways to evaluate how readers were engaging with our content, both online and offline.
We created a metric that we called—tongue firmly in cheek—the Very Special Index of Goodness. This complex amalgam, designed to improve our understanding of reader engagement, combined external and internal data to yield a single number that we could track over time. The intentions behind this tool were as earnest as its name was wry, and we weren’t the only ones who were thinking along such lines: In 2010, the online arm of the Philadelphia Inquirer released a reader engagement formula of its own: Σ (Ci + Di + Ri + Li + Bi + Ii + Pi). That formula took into account several factors—clicks, duration, “recency,” loyalty, brand, interaction, and participation—and, like the one that we had concocted, resulted in a single number.4
For us, the limits of this single-number approach quickly became apparent. It reminded us of the assertion in The Hitchhiker’s Guide to the Galaxy—the cult novel by Douglas Adams—that the “Answer to the Ultimate Question of Life, the Universe, and Everything” is 42. So we shifted course and focused anew on qualitative methods for measuring engagement. We now conduct online surveys and carefully track the flow of social media, and what we’ve found has pleased staff members and financial supporters alike: In surveys, up to 70 percent of readers say that they recently took action on the basis of Grist content. We aren’t an advocacy organization, but our storytelling has clearly inspired change on the ground.
After more than a decade of trial and error, we arrived at a set of metrics that work for us. For these metrics, we use terms now familiar to most people who work in nonprofit media. We measure reach, which covers the size of our audience—the number of people who access our content either at our site or elsewhere online. We measure impact and engagement, which involve reader activity both online (in the form of likes, shares, and comments) and offline (in the form of behavior change). And we measure influence, which encompasses media citations, policy changes, and other elements that make up the environmental conversation.
The meaning of these terms, like the field of nonprofit media as a whole, is fluid. As yet, people are not using them consistently. In their work on this topic, for example, Anya Schiffrin and Ethan Zuckerman define “influence” in a way that resembles our use of the term “impact”—and vice versa. (See “Surveying the Field.”) But the core idea is the same in each case: How users respond to your content is distinct from how your content affects the larger media or policy environment, and both of those variables are distinct from how many people simply read or view your content.
Although we sometimes felt alone in our explorations, other players in this field were also experimenting with ways to evaluate the connection between content and social impact. Over the past several years, a national conversation on this topic has started to develop—one that includes practitioners in nonprofit media, funders who support them, and a growing cadre of researchers. Recently, we spoke with several influential figures who are contributing to that conversation.
The Search for Solutions
Jessica Clark has been thinking about how to chart media impact since 2004. She first ventured into the fray in a moment of journalistic upheaval: “In the wake of the [2000 US presidential] election and the Iraq War, there was a wave of new media projects that expanded the possibilities for different kinds of journalism,” says Clark, who is now the research director at Media Impact Funders, a network of more than 50 funding institutions. Amid those developments, she notes, journalists were being asked to leave objectivity behind and to express opinions about the news. Over the next several years, Clark explored that trend while serving as editor of In These Times, a progressive magazine, and as director of the Future of Public Media Project at the Center for Social Media at American University. She then co-authored a book, Beyond the Echo Chamber: How a Networked Progressive Media Can Reshape American Politics (2010).
When the book came out, Clark and her co-author, Tracy Van Slyke, opted out of a conventional book tour. Instead, they organized a series of “impact summits” that took place in seven US cities. Drawing on insights gathered at these events, Clark and Van Slyke developed a report titled “Investing in Impact.” The report included strong advice for funders, journalists, and other stakeholders: “Shifts in technology and user habits mean that old assumptions about what constitutes impact must be reconsidered. Simply reporting on an issue or community is no longer the final outcome in an era of multi-platform, participatory communication.”5
It isn’t just technology that has changed, Clark argues. By partnering with foundations, nonprofit news outlets have carved out a new business model. And funders, having entered what Clark calls “uncharted territory,” are raising questions about the industry in which they are investing. They are eager for insights on “how to understand the impact dynamics of emerging platforms, how to build rigorous case studies that track the movement of coverage across platforms and contexts, and how the increased ability of users to participate in production shifts the impact equation,” she explains. More to the point, funders are also investing in serious efforts to address these questions.
Major players such as the Knight Foundation, the Bill & Melinda Gates Foundation, and the Ford Foundation have directed significant funding to this area. In 2013, Gates and Knight created the Media Impact Project (MIP), a $3.25 million initiative that is housed at the Annenberg School for Communication and Journalism at the University of Southern California. (The project now also receives funding from the Open Society Foundations.) MIP bills itself as nothing less than a “hub for best practices, innovation and thought leadership in media metrics.”6
New Challenges, New Tools
Dana Chinn, who runs MIP, is a media analytics strategist who serves as a lecturer at the USC Annenberg School. Previously, she worked at organizations such as the Gannett newspaper chain and the Los Angeles Times. According to Chinn, the nonprofit media industry could learn a lot from industries such as e-commerce and technology. “Analytics are essential to any business, and they are integrated into the operations and management philosophy of most companies,” she says. “If the very survival of the news industry is at stake here, shouldn’t we be taking the same approach?”
MIP is now collaborating with nonprofit and for-profit news organizations that include The Seattle Times, Southern California Public Radio (KPCC), and a trailblazing outlet called The Texas Tribune. Together, these partners are testing ways to improve their capacity to gather and analyze impact data. Among other projects, MIP served as a consultant to Participant Media (an entertainment company founded and led by philanthropist Jeff Skoll) on the creation of the Participant Index, a tool that measures the effectiveness of films, TV shows, and online videos that feature social causes.7 “We’re not going to get the 100 percent answer” to the impact question, says Chinn. “But we can get one level above where we’ve been in the past, which is throwing up our hands and saying, ‘It can’t be done.’” A signature project of MIP is the Media Impact Project Measurement System, a data repository that will combine public and proprietary sources of information. The system will likely be operational by the fall of 2015. As the repository grows, Chinn says, MIP and its partners will be able to analyze impact over time and across different media types.
A similar effort is in progress on the other side of the country. Researchers at the Tow Center for Digital Journalism at the Columbia University Graduate School of Journalism, with funding from the Tow Foundation and the Knight Foundation, have created a tool called NewsLynx. The tool collects quantitative and qualitative data in one central place. It aggregates data from sources such as Facebook, Google Analytics, and Twitter; it offers a way to track anecdotal evidence; and it provides a system for monitoring links and discussion threads related to news content. By using keywords and alerts that apply to a specific organization, topic, or piece of coverage, users can create a custom dashboard that offers a full-spectrum report on the impact of their work.
Over the past year, about a dozen US news organizations—from small-city newspapers to national outlets—have participated in a pilot test of the NewsLynx tool. (Those beta testers include organizations that work with MIP, and the head of the Tow Center sits on the MIP advisory board. It is, as Chinn notes, “a small news-metrics world.”) “People had never been able to get easy access to things like share counts of an article over time,” says Brian Abelson, co-creator of NewsLynx. (Abelson, a former fellow at the Tow Center, now works at Enigma, a data analytics company.) This tool provides a fix for that problem, he explains: “Now anyone can keep track, with very little effort, of how many times an article has been shared, when it was shared most [widely], and how that information lines up with how many people visited the article over time.” Abelson and Michael Keller, a data journalist at Al Jazeera America who helped create NewsLynx, recently produced a research paper on the project. They concluded that the increasing flow of open source data will provide newsrooms with an unprecedented amount of information about media impact.8
But the prospect of navigating a mighty river of data is a mixed blessing. In 2014, Grist undertook an experiment—funded, like NewsLynx, by the Knight Foundation—in which we developed a prototype open-source tool that measures “attention minutes.” Pioneered by for-profit media sites such as Medium and Upworthy, this metric tracks how far users actually make it into an article or a video. Our use of this metric has yielded data that give us new insight into how readers engage with our content. In the past, we might have assumed that two articles with the same number of page views had performed equally well. Now, by looking at how long each article held readers’ attention, we can see that one piece may have gripped readers more deeply than the other. We can then apply that information on what makes an article “sticky” to other items of content. It’s a promising tool, but there’s a catch: It delivers more data than we can feasibly store and regularly digest. As a next step, we are working to partner with an organization that can help us manage and analyze this rich lode of data. In the meantime, we have news to cover and a site to produce.
Resource constraints, of course, are a common challenge for nonprofit newsrooms. But another obstacle to the widespread adoption of data-tracking tools is the fact that most news organizations operate in a self-imposed silo. “Everyone is slightly different and interested in slightly different things,” Abelson says. “So how do we build something that can accommodate all those needs, while still being coherent and workable and easy to start using?”
A Shared Language
Lindsay Green-Barber arrived at the Center for Investigative Reporting (CIR) in 2013. She had recently completed a dissertation on the use of communications for political mobilization in Ecuador. Now, under a fellowship awarded by the American Council of Learned Societies, she took on the newly created role of media impact analyst at CIR. Her first assignment: to define what “impact” actually means to the organization.
Green-Barber spent two months surveying various stakeholders about that question. She then spent a year creating and refining systems that allow CIR journalists and other staff members to track data related to audience feedback, requests for interviews, and social media activity. “Rather than think about analytics and metrics being the end measure of success, we started thinking about them as part of the broader picture,” she says. Green-Barber also used her understanding of social movements to help CIR expand its notion of success to encompass more than just a shift in law or policy. “An investigation of a vulnerable community is not going to lead a lawmaker to ‘do a 180,’” she says. “If you’re looking just at legal change, you’ll miss a lot of other important change.”
Indeed, the simple act of informing and engaging readers can be among the most important forms of impact that a media outlet can pursue. “The fact that a user not only visits a site but visits it regularly, and engages through sharing or commenting, means that [the user has] an emotional connection to the organization,” says Elise Hu, a culture and technology reporter for NPR. (Hu cofounded The Texas Tribune, and serves both as an advisor to the Knight Foundation and as a member of the Grist board.) “That emotional connection will lead to other actions.”
The sense that there’s more to life than policy change led Green-Barber to identify three types of impact for CIR to track: macro, which includes legal and regulatory changes; meso, which includes social shifts, such as a change in public opinion; and micro, which includes changes at an individual level, such as increased knowledge. Using this framework, she collaborated with MIP, the Tow Center, and other organizations to create a taxonomy of impact. This tool, known as the Offline Impact Indicators Glossary, “is giving people a methodology to look at things they’ve been thinking of as unmeasurable or unknowable,” Green-Barber says. The glossary is broad in scope, encompassing everything from the reversal of a legal decision to an increase in social capital.9 These aren’t the sorts of things that can be measured by Google Analytics, but they are critical to understanding the full impact of journalism.
Abelson, a collaborator on the glossary project, hopes that it will help news organizations develop both a shared language and a habit of sharing data. “This work has to be done on an inter-newsroom level,” he says. “More newsrooms have to be willing to share information in a more transparent way.”
The Conversation Continues
“If there is one thing that seemingly all media organizations can agree on, it is that impact is not any one thing,” Green-Barber wrote last year in a report for CIR.10
That’s not just a Zen koan. For nonprofit media, metrics pose an especially knotty challenge because they must serve multiple purposes. They must offer meaningful evidence for foundations and other impact-oriented investors. They must make sense to advertisers who still think in terms of CPI (cost per impression) and other traditional standards. (Not all nonprofit media organizations rely on income from advertising as part of their revenue stream, but many do.) They must convey organizational progress to board members and other internal stakeholders. Ideally, moreover, they will offer information that’s relevant to journalists and others in the newsroom.
During a period that overlapped with Green-Barber’s stint at CIR, Grist also dedicated a position to studying the question of impact. Our self-dubbed “actionable metrics engineer” was able to track data and unravel mysteries in ways that even the most well-meaning editor would never find time to do. One of his most important conclusions was that the topline numbers that we track—the ones that help make the external case for Grist—didn’t always resonate with individual team members. Today, like many other outlets, we are working to resolve that tension between external and internal needs.
But the core problem that nonprofit outlets face may not lend itself to resolution. After all, any metrics that work today might cease to be relevant tomorrow. News organizations must therefore be flexible and innovative when it comes to measuring impact. Philanthropists, meanwhile, must understand that impact metrics in this field might never be as black-and-white as those in other sectors. “The best we can do is find out which organizations are doing interesting things in this area and which practices can be replicated,” says Sotsky.
The real solution to this challenge most likely will not arrive in the form of cutting-edge tools or complicated formulas. In fact, it might resemble what journalists already do best. “Impact analysis is like reporting: You have to cover the five Ws [who, what, when, where, and why],” says Clark. “If I were an editor and I were assigning a story on what happened with your site, I would want to know the numbers. They are important to measure, and they make your newsroom smarter. But measuring impact is not the only way to think about it. You can also share data, information, or strategic intelligence about a project. What you are doing is storytelling.”
Support SSIR’s coverage of cross-sector solutions to global challenges.
Help us further the reach of innovative ideas. Donate today.
Read more stories by Chip Giller & Katharine Wroth.