How the experiment was conductedOn of the challenges with research on media impact is the lack of control researchers have over the publication of news itself. You can’t exactly randomize the news. The need to maintain editorial independence and the timeliness of reporting means there have been relatively few large-scale, controlled experiments attempting to experimentally measure the effect of news media. The research team found a way around this by actually collaborating with news outlets, using the first three years of the five-year study to build trust. “Frankly, when we began talks with the researchers in 2012, what they required seemed impossible,” Media Consortium executive director Jo Ellen Green Kaiser wrote on MetricShift in 2016. However, the researchers began attending industry conferences and worked with the Media Consortium to test the experiment with a small group of participants. In the end, the study recruited 48 news outlets who agreed, in small groups, to publish at the same time on a broad policy area agreed-to beforehand and typical of the the outlets regular coverage. The 11 policy areas were race, immigration, jobs, abortion, climate, food policy, water, education policy, refugees, domestic energy production and reproductive rights. The media outlets retained full control over what was published, as well as the option to opt-out at any time. For their part, the researchers picked a two-week publication window when they expected a slow news cycle and randomized the publication date to either the first week or the second week. They then were able to measure and compare changes in the volume and composition of conversation on Twitter between the publication week and the control week. The experiment was run 35 times, and this group-publishing was found to have increased conversation about the general topic by 63 percent over a week, relative to the average day’s volume. The organizations were mostly small publications, with a median 50,000 subscribers, along with a few better-known ones like Mother Jones. Some members of the Media Consortium who participated were Truthout, In These Times, Bitch Media, The Progressive, Earth Island Journal, Feministing, Generation Progress, Ms. Magazine, Yes! magazine and Making Contact.
What do the findings mean?For the Media Consortium, this research shows that “even small independent news outlets can have a dramatic effect on the content of national conversation,” Kaiser wrote in a blog post. Actually, she says, independent media has some built in advantages when it comes to having a impact on the national discourse. Independent outlets boast loyal communities of readers and supporters who are eager to push national conversations on social media, a willingness to collaborate openly that amplifies the effect of news, and news coverage that can face down serious topics because there’s less imperative to chase pageviews. Defining impact as influence on a national conversation is new, with trustworthy measurement having been a big hurdle to overcome. There are some limitations to the study. Kathleen Hall Jamieson, director of the Annenberg Public Policy Center at the University of Pennsylvania, told the New York Times that without the researchers disclosing more information (names of individual stories and publishers were withheld for reasons of editorial integrity), there would be questions about its general applicability. Is the volume of conversation on Twitter a true proxy for the national conversation, for example, and how meaningful is the discourse there? Nevertheless, this research may shift the discussion about media impact. “In fact, too often, philanthropic foundations refuse to support these outlets because they are ‘too small’ and ‘don’t have enough impact,’” Kaiser wrote.
After sending 153,861 tweets, I definitely *want* to believe that journalists at small outlets have "substantial power to affect national conversations about politics & policy" https://t.co/XpOLZTtmws Highlights need for diversity in media sources & amplifying trusted journalism. pic.twitter.com/bGCC01Lr4X — Alex Howard (@digiphile) November 10, 2017