Can We Measure Media Impact? Surveying the Field

In October 2014, the nonprofit news organization ProPublica released “Deadly Force, in Black and White,” a study of police killings of civilians in the United States. By examining data collected by the US Federal Bureau of Investigation between 2010 and 2012, ProPublica found that young black men were 21 times more likely to be killed by police than were young white men. In most cases, white policemen were responsible for the killings. The average age of the black victims was 30. Police reports typically cited “resisting arrest” or “fleeing arrest” as the reason for such shootings. Often, however, the police didn’t provide a reason and instead described the circumstances of a given shooting as “undetermined.”¹

Complete with graphics and links to official documents, “Deadly Force, in Black and White” is a great piece of original journalism. Media outlets quoted it, and civil rights organizations cited it in support of police reform. But in November, just a month after ProPublica released the report, an event occurred that highlighted the limited ability of media coverage to affect facts on the ground: A grand jury in Ferguson, Mo., declined to indict police officer Darren Wilson in the shooting of an unarmed black man named Michael Brown. When it comes to changing the world, it seems, powerful forces such as systemic racism often matter more than careful reporting and hard data.

(Illustration by Curt Merlo) 

ProPublica, like a growing number of other media organizations today, relies extensively on donor funding to support its work. As a consequence, these media outlets face increasing pressure to demonstrate that journalism can make a difference in the world. Donors are seeking ways to measure the impact of the media projects that they fund, and media organizations in turn are working to track the real-world effects of what they publish—partly in the hope that proving their worth will help enable their survival.

The days when media companies could survive by relying solely on advertising and subscription revenues are over. Disruptive technologies, among other factors, have steadily eroded the business models that traditionally supported US news organizations. One alternative source of funding, of course, is the public sector. But Americans have always been uneasy with government support for media. The United States—unlike Germany and the United Kingdom, for example—has never developed robust taxpayer-supported public media institutions. In the absence of both commercial and public sources of revenue, more and more media organizations are willing to accept novel funding arrangements from the philanthropic sector.

Private philanthropy, along with government agencies, has long supported struggling newspapers and community radio stations in developing countries. Donors in this category include the Ford Foundation, the Open Society Foundations, the US Agency for International Development (USAID), and a host of European governments and international aid organizations. USAID, for example, has funded community radio stations in Mali, and the Open Society Foundations has supported media initiatives led by Burmese exiles.

But we are now adjusting to a reality in which major US and European news organizations depend on philanthropy. The Guardian newspaper receives funding from the Bill & Melinda Gates Foundation for its Global Development Web page, and NPR gets support from that foundation for its coverage of topics such as education. In addition, as traditional media outlets cut their investigative budgets, donor-supported organizations like the Center for Public Integrity, the Global Investigative Journalism Network, ProPublica, and SCOOP have emerged to fill that gap. Philanthropic organizations, in fact, have fueled an explosion of independent investigative reporting in recent years. The Omidyar Network, for instance, has funded a Web-based outlet called Sahara Reporters, which focuses on exposing corruption in Nigeria.

Some of the most influential new donors on the scene today—including the founders of the Gates and Omidyar philanthropies—come not from a journalism background but from a business tradition in which management by metrics is commonplace. Partly under their influence, a movement has emerged to find ways to track the effects of donor-supported journalism. All around the world, media outlets are learning that some funders are uncomfortable with supporting journalism merely as a “public good.” They want to see proof of impact.

The Media Metrics Quandary

The task of “proving impact” doesn’t come naturally to most journalists. They reject a utilitarian view of their worth, preferring to believe that news is a public good that merits support for its own sake. They view themselves not as campaigners for a cause but as fair and impartial observers. At the same time, they like to think that they can change the world simply by “getting the story out.” Aron Pilhofer, executive editor of digital at The Guardian, summarized the prevailing view in a much-quoted blog post: “The metrics newsrooms have traditionally used tended to be fairly imprecise: Did a law change? Did the bad guy go to jail? Were dangers revealed? Were lives saved? Or least significant of all, did it win an award?”² In any event, journalists tend to be wary of adopting universal metrics. They know that each media organization has a different audience that it wants to reach and different ideas about what constitutes “impact.”

There is skepticism on the donor side, too. Some donors have taken the stance that media impact is impossible to gauge: There are too many variables to measure, and the time scale for evaluation is too short, they argue. “Media organizations need to make the case that their work could lead to change, but I am very skeptical about what I see as a growing impact-industrial complex,” says one grantmaker from a prominent US foundation. The debate about whether and how to ask grantees to measure impact has created a fault line in the world of donors. In some ways, this dispute echoes the ongoing debate over strategic philanthropy. Here, too, critics argue that basing donor decisions on outcome-related evidence forces grantees to focus on conducting evaluations to prove the worth of their work—at the expense of actually doing that work.³

Journalists and donors both note that the media is only one part of a larger ecosystem. The multitude of variables that affect any process of social change makes it hard to isolate—let alone measure—the impact of journalistic efforts. People sometimes credit the media with helping to cement opposition to the Vietnam War, or with inspiring the protests that led to the Arab Spring, for example. But was it reporting on the Vietnam War that undermined public support for it, or was it the fact that middle-class college students didn’t want to fight in that conflict? Was the Arab Spring a “Facebook revolution,” or was it a predictable response to deteriorating economic conditions and widespread youth unemployment?

For years, economists and political scientists have studied the effects that media coverage has on areas such as government accountability, public corruption, and voting behavior.⁴ Their research shows that news coverage does affect a wide range of outcomes, including government spending decisions and governmental responses to natural disasters. At the same time, scholars warn that it’s hard to trace a direct connection between, say, a single newspaper story and an identifiable real-world effect. “If we were to monitor the effect of 100 news stories on people’s behavior and noticed no difference, we could conclude that there is no media impact. But then story 101 may cause people to take to the streets in protest. Perhaps that one story hit a nerve, or perhaps stories 99, 100, and 101 packed into a snowball that spurred the mobilization,” says Paul F. Lagunes, assistant professor of international and public affairs at Columbia University, who studies the effectiveness of anti-corruption programs. “Then there’s the question of unobserved change. A news story may modify the way we view the world without spurring us to take immediate action.”

What’s more, the metrics that media organizations typically use were not designed to measure social impact. Most of these metrics originated in the advertising industry. They estimate the size of an audience for broadcast and print outlets, or they count the number of visitors to a website. But knowing that an article reached millions of readers is only one part of answering the larger question of whether the article had an effect on voters and policymakers. After all, conveying information to a large number of people is not always the best way to promote social change. Minky Worden, director of global initiatives at Human Rights Watch, notes that efforts based on mass action often have limited efficacy. “Boycotts are kind of old-fashioned now,” she says. “Targeted sanctions, reaching policymakers at a key moment, or creative campaigns to change specific laws and policies can be more effective.”

Measure for Measure

Despite these concerns and caveats, several organizations are taking steps to develop usable standards for measuring media effects. Among those groups are the Gates Foundation, the John S. and James L. Knight Foundation, the Nieman Journalism Lab at Harvard University, the Norman Lear Center at the University of Southern California Annenberg School for Communication and Journalism, the Pew Research Center, and the Tow Center for Digital Journalism at the Columbia University Graduate School of Journalism. In looking at the approaches that these groups are adopting, we distinguish broadly between reach (how many people engage with a given body of media content), influence (how that content affects public dialogue), and impact (how the content helps drive policy change or movement building).

Reach | There’s no guarantee that a story read by millions of people will have more impact than one that reaches only a few hundred readers. But it’s easier to posit impact when a story finds a substantial audience. Numerous metrics exist to help news organizations gauge the audience for their content. In addition to the metrics commonly used by Web advertisers—page views, unique visitors, and so on—some outlets consider “attention minutes,” a variable that measures the amount of time that readers spend with an article or view. The leaders of Upworthy, a website that develops and markets articles that it deems “meaningful,” label attention minutes as their “primary metric.” More specifically, they look at two variables: “total attention on site” and “total attention per piece.”⁵ Many organizations also track the “social sharing” of a story by noting how often people cite it on Facebook, Twitter, and other social networks. Sharing of this kind expands the reach of a story beyond the base of a site’s regular readers.

One problem with focusing on reach is that it tends to favor offbeat or feel-good stories over more substantial reporting and analysis. Another problem is that reach-oriented metrics are ones that people can easily manipulate. If the number of clicks on a given article is the measure of its reach (and if advertising revenue derives from reach data), then those who manage websites will devise ways to increase that number. Hence the emergence of “clickbait” stories that carry attention-grabbing headlines and of “bots” that click on stories automatically.

Efforts to enable advanced approaches to measuring reach are now under way. A leading example is NewsLynx, a project hosted at the Columbia University Graduate School of Journalism. NewsLynx aims to help news organizations and their funders map how stories spread across the Web. “We found that people were getting a bunch of notifications from Google Alert and then manually entering them into a spreadsheet. We are trying to make the drudge work of being an impact analyst easier,” says Brian Abelson, who helped launch NewsLynx. (For more information on that initiative, see “Reading Between the Lines.”)

Influence | How do media stories affect readers’ attitudes? How do they shape the public dialogue as a whole? We know that in some cases media coverage can change how people and organizations approach a given issue. Josh Greenberg, an associate professor of communication at Carleton University in Ottawa, Ontario, studied the way that coverage of labor conditions in the shoe factories that supply Nike led to a change in how people think about solutions to the sweatshop problem. He showed that media outlets such as The Washington Post shifted their coverage away from the conditions in those factories and toward the role of individual buying choices. As a result, the discussion of how to solve the problem began to focus less on improving or enforcing regulations than on, say, encouraging consumers to buy fair labor footwear.⁶

In the past, news organizations had to rely on focus groups and survey research to understand how audiences absorbed their content. But with the rise of the Web, the toolkit for measuring influence has expanded dramatically. Hyperlinks, the glue of the Web, provide a clear proxy for influence. Media professionals sometimes claim that a link isn’t an endorsement. But there are clear indications that when an author links to a story, she signals that the story did influence her (positively or negatively) and that it therefore has helped shape the broader discussion of a given topic. Consider a study led by Yochai Benkler, a professor at Harvard Law School. Benkler and his colleagues used links to trace how Wikipedia contributors, grassroots activists, and technology bloggers influenced the debate over the Stop Online Piracy Act (also known as SOPA-PIPA), a bill in the US Congress that proposed significant changes to the laws that protect copyright holders on the Internet.⁷

Media Cloud, a joint project of the MIT Center for Civic Media and the Berkman Center for Internet & Society at Harvard University, offers a new way to discern patterns of influence. (One of us, Ethan Zuckerman, is a principal investigator for Media Cloud. So is Yochai Benkler.) It is an open source tool that monitors 50,000 social and journalistic channels, and it allows researchers to study two important media-based processes: agenda setting and framing. By measuring the volume of stories on a given topic, as compared with the coverage of other topics, Media Cloud can show how effective politicians, activists, and other parties are at putting an issue “on the agenda” for public debate. And by tracking the language that people use to talk about a topic, Media Cloud can highlight the various “frames” that get attached to a news event. A frame, in short, is a way of interpreting an event that supports one social or political agenda over another. A story about the killing of Michael Brown, for instance, might lead to discussions of topics such as urban poverty, racial bias, and militarized policing. By tracking published stories and clustering those that use similar language, Media Cloud helps identify which frames appear in reporting and which news outlets have introduced new frames into the public debate.

A difficult challenge in this research involves evaluating the role that media-based efforts play in setting an agenda or framing an issue. The #blacklivesmatter movement is a case in point. The killing of Brown, along with other episodes in which young black men died at the hands of police, led people to use that hashtag on Twitter in order to frame these tragedies as part of a larger narrative about police treatment of people of color. But violent protests against those deaths led to a reframing of the entire story as one that ended in “rioting.” Both events on the ground and people’s interpretation of those events, in other words, can help set the agenda for an issue.

Impact | Just because journalists have exposed people to information doesn’t mean that people will take action or demand policy changes in response to that information. Jonathan Stray, a fellow at the Tow Center for Digital Journalism, calls this challenge—that of translating media coverage into social impact—“the last mile problem.”⁸ In certain cases, however, it’s possible to link a specific journalistic project to a discernible policy outcome. Even an article that doesn’t gain wide circulation can cause change if those who do read it are willing and able to act on what they read. (The late Robert L. Bartley, who was editor of the Wall Street Journal opinion page for many years, once said, “It takes 75 editorials to pass a law.”⁹) Here are some examples of journalism that resulted in real-world social impact.

• In 2012, Bloomberg Businessweek published an article that built on research by New Zealand scholars who had documented the forced labor of Indonesian workers on Korean-owned vessels. These vessels were fishing for catch to be exported by New Zealand companies.¹⁰ The article caused so much outrage that the New Zealand government quickly enacted legislation that made it a crime, punishable by imprisonment, to exploit migrant workers. The government also passed a law that will require all vessels that fish in New Zealand waters to abide by the country’s labor, health, and safety regulations. After the article came out, moreover, retailers such as Safeway, Wal-Mart, and Whole Foods launched investigations into their supply chains. Some US buyers canceled contracts with New Zealand fish suppliers.¹¹

• In 2010, ProPublica published a series of articles that tracked which US doctors had received payments from pharmaceutical companies. Over a two-year period, according to the series, those companies made payments to roughly 17,000 physicians, and these payments totaled more than $2.5 billion. ProPublica provided an app that people could use to find out whether drug companies had given money to their doctors.¹² More than 180 outlets picked up the story. “This scrutiny,” according to ProPublica, “has prompted tightening of disclosure rules.”¹³ The University of Colorado, Denver overhauled the conflict-of-interest policies that apply to its teaching hospitals, for example, and Stanford University took disciplinary action against five of its faculty members.

• In 2014, the Center for Public Integrity published a series on how Luxembourg had enabled major companies such as Coach, Disney, FedEx, Ikea, Koch Industries, and Pepsi to evade taxes by registering in that country so that they could take advantage of loopholes in various treaties.¹⁴ The story splashed across front pages throughout Europe. Within days, there were calls for Jean-Claude Juncker—a former prime minister of Luxembourg, who had just been chosen to head the European Commission—to resign from his new post. The European Union, meanwhile, moved to pass a law that would ban these kinds of sweetheart deals.¹⁵

Some investigative reporting teams have developed systems to track the real-world impact of their work. They use these systems to monitor editorials that cite their investigations or policy changes that come in the wake of their reporting. Each year, ProPublica publishes updates on the status of issues that it has covered in its major reporting projects. And Participant Media, an entertainment company that produces films with social and political themes, has launched the Participant Index, an attempt to capture outcomes that go beyond policy change. The company conducts surveys to determine whether people who have seen specific Participant films have then taken related actions—signing a petition, making a donation, or joining an organization, for example. Participant has invited newsrooms and advocacy organizations to use its method as well.¹⁶

The Impact of Measurement

Nonprofit newsrooms and the organizations that fund them will continue to hone their use of new and existing metrics. Tools that can measure not just “reach” but also “influence” and “impact” are in their infancy. Yet they are becoming ever more sophisticated, and our ability to apply them has advanced dramatically.

Today, we know a lot more than we did previously about news consumption habits and about the way that ideas spread, and this knowledge can help us understand how social change occurs. Research shows that change usually takes place over the span of decades and that media coverage has an impact only when other social forces are also at work. Take the case of female foot-binding in China. For hundreds of years, the philosopher Anthony Appiah notes, Chinese writers had called attention to the dangers of that practice. But only when young members of the country’s elite became ashamed of foot-binding did Chinese authorities begin to issue decrees against it.¹⁷

Some kinds of impact are easy to measure—the passage of a law, the ousting of a corrupt politician. Other forms of social change (like the demise of foot-binding in China) involve a transformation of cultural norms. Almost inevitably, that kind of change is a long and complex process. Even episodes of seemingly rapid change, such as the shift in Americans’ attitudes toward gay and lesbian marriage rights, typically follow an extended period of political activism and cultural ferment.

In that context, consider again the ProPublica report titled “Deadly Force, in Black and White.” It’s probably unrealistic to expect that such a report would lead to an indictment of the police officer who shot and killed Michael Brown. But that measure of impact is not the only one that matters. By bringing attention to the disproportionate use of force against black men, the ProPublica investigation may help shift public attitudes about that subject, and it may open a dialogue both about police practice and about the persistence of racial bias in the United States.

As media organizations become ever more dependent on funding that doesn’t come from advertising or subscriptions—funding that comes with pressure to demonstrate real-world impact—they and their supporters should heed the risks that this trend involves. Funders, for their part, must avoid the trap of supporting only groups that are able to deploy the latest technology. A small journalism NGO in Africa, for example, is unlikely to have the staff, skills, or resources to use sophisticated impact measurement tools. Would we want a world in which that kind of group cannot get funding? After all, until a journalist actually covers a story, we can’t know whether the story will make a difference.

Media organizations, meanwhile, must watch out for threats to newsroom independence. The increasing focus on measurable impact may become an excuse to decide that only some kinds of coverage are worth supporting. If newsrooms limit their reporting to stories that can have immediate effects or quantifiable results, they might be unwilling to cover large, persistent—yet vitally important—social problems. Ultimately, the impact that journalists can have on society will erode if they must serve the whims of funders. That is true whether the funders in question are government officials, advertisers, corporate owners, or well-intentioned philanthropists.

Read more stories by Ethan Zuckerman & Anya Schiffrin.