Thursday, March 22nd, 2007 at 12:49 am | In Media, Politics, News |

With more polling data coming out on the 2008 campaign coming out every day, I though it’d be useful to post summary of what it takes to accurately report on poll results:

A news report about a public opinion poll needs to have several qualities in order to serve both the public interest and that of the news organization.[1]

The report must be interesting enough to attract general readers and simple enough for general readers to understand. Polls make attractive topics for journalists because the most common results show either 1) a fairly clear winner and loser or 2) a close race. Both make for interesting, easy to write stories. It is rarer to see a journalist tackle in-depth analysis of a poll that takes on multiple aspects of an issue, such as the recent poll of Iraqis done by ABC News, USA Today, the BBC and ARD[2]. If a journalist were to write a story on that poll, he would likely pick out one or two poll questions to focus on.

This simplification is not necessarily a problem. In fact, it can be desirable. If a journalist were to include details on every question in a lengthy poll with careful attention to potential wording bias, sampling size, and methodology, few people would ever read the story. When writing for a general audience, it is better for a journalist to glean the most important facts and relay them with enough detail to assure readers that the poll is credible. While their job is to simplify complex data, it is essential that journalists be educated in statistics and polling methodology. With knowledge of how polling works, a journalist can determine whether the results are valid and whether the story should be on polling results alone or, in rare cases, the motives or shoddy practices of those conducting the poll.

Beyond being straightforward, the report should provide enough detail about the polling methodology for the reader to assess the validity of the results. There are several technical questions that the article should answer:

1) What is the margin of error and confidence interval?

2) What is the sample size?

3) Who is the population being sampled?

4) During what time span was the poll conducted?

If the poll’s results fall within the margin of error, the article should explicitly say that there is a statistical tie. Reporting that one candidate is up 51 percent to 49 percent when there is a 3 percent margin of error is irresponsible journalism.

I do not think it is necessary to publish a full statistics lesson with every article on poll results, but it would be helpful to have an explanation of key statistical terms for those who have little experience with polling data. Including such a glossary may prove to be unfeasible for print publications, but it would take little effort to include a link with every polling article on a news organization’s Web site. A good example of a statistical guide is available on the Web site[3] of Robert Niles, a journalist in Pasadena, California.

News reports should also reveal question wording. Although they might present reporters with similar headlines, asking, “If the election were held today, would you vote for Candidate A or Candidate B?” may provide different results than asking, “Which candidate do you prefer, Candidate A or Candidate B?” For online articles, media organizations should link to the full results and methodology of a poll if they are available. The onus for responsible poll reporting does not fall solely on journalists. Polling firms need to be sure they provide all the necessary data for reporters to write accurate stories.

One critique of news reporting on polls states that it places undue focus on the “horse race” aspect of political campaigns. According to Stephen Hess of the Brookings Institution, there is too much focus on polling numbers and not enough on policy positions. Hess wrote, “Horse race stories are more fun. Substance stories are more important.”[4] Simply covering who is ahead does little to serve the public interest, however, polling data on public opinion during a campaign is still newsworthy, if only to shed light on the campaign process and how some of the data being used by political actors. Attempting to include polling data with reporting of campaign events can actually result in media outlets unintentionally making spurious connections between public opinion and political actions.

I would like to see more journalists include not only the source of polling results (which they often do), but also a brief background on the polling organization – especially if is not a widely recognized news organization. For example, a February 2007 Milwaukee Journal Sentinel (MJS) article[5] reports the results of a poll on the 2008 presidential candidates conducted by the polling firm Strategic Vision. What the MJS article does not say is that Strategic Vision takes on mostly Republican clients. It does not take much effort to add valuable detail. David Yepsen, political columnist for the Des Moines Register, also wrote about a Strategic Vision poll in January 2007[6] and included this caveat: “Strategic Vision is a Republican-leaning firm, but its polling data are considered credible. Its numbers are mirrored by a recent Zogby poll of caucus-goers that also shows Edwards ahead with the others trailing.” This addendum acknowledges that the poll results are not in question, but it gives some background as to where the results are coming from – information that most readers would never know on their own. For these private polling firms, it would also be helpful (albeit often impossible) to know who is funding the poll. Strategic Vision often publishes “free” polling results without identifying where the funding for the poll came from.

When approaching polling data, journalists and pollsters have different priorities. According to Everett Ladd, “speed and timeliness have become highly regarded values (in journalism).”[7] This desire for quick analysis of polling results is derived from necessity. Journalists need to process polling results quickly in order to meet their deadline, compete with other news outlets and serve the news appetitive of their readers. They search for topics that make interesting journalism, such as political conflict. Increasingly, journalists have to consider the popular appeal of their stories. After all, it does not matter if a story contains astounding information if no one reads it. At the same time, I think pollsters, especially those employed with news organizations, are guilty of playing to the journalistic desire for conflict when choosing polling questions. Pollsters, on the whole, use slow, expensive methods to collect the large amounts of data needed to produce legitimate results. Just handing this data to journalists is not enough. Pollster should (and often do) provide summaries of polling results to journalists along with

Too often, journalists omit technical information about polling results when they could have easily included it, especially in an online format. I think the problem is that journalists often do not fully understand the statistics behind the results and then omit details without realizing their importance in interpreting the data. Along with a summary of results, it would be helpful for pollsters to provide a brief refresher on methodology and terminology to ease the transition from statistics to text.

The key is to balance the sometimes-complex details of the polling process that are needed to establish the validity of the results with a format that is concise and interesting enough to attract readers.

Originally written for a UW Journalism class on political communication - March 2007