The phrase “media bias” is used when someone does not agree with how a news organization presented a story. News organizations themselves accuse other news agencies of various biases, particularly with regard to where an organization falls along the liberal-conservative continuum. Fox News accuses the mainstream media, of which Fox apparently is not a part, of a liberal bias. So-called progressive organizations such as MSNBC accuse Fox News of having a conservative bias. When it comes to media bias, point an accusatory finger in any direction and you’ll likely hit a target.
The ease with which you can hit a target was evident last week in how several major news organizations covered the ratings – yes, ratings – of a cable network’s “reality” show. Consider a sample of last week’s headlines about Duck Dynasty. From NBC News: “Duck Dynasty’s ratings dip in wake of controversy.” From USA Today: “Duck Dynasty returns to big ratings drop.” From ABC News: “Duck Dynasty Returns With Lower Ratings.” From the Huffington Post: “Duck Dynasty Ratings Are Down Dramatically From Last Season.” From Salon: “Duck Dynasty season premiere sees a 28 percent ratings drop.” From the New York Daily News: “Duck Dynasty Season 5 premiere suffers drastic drop in ratings in wake of Phil Robertson’s controversial comments.” It seems that A&E’s show about a Louisiana family – the Robertsons – that manufactures duck calls might be in trouble.
The source of the trouble was an interview published in GQ magazine where the patriarch of the family, Phil Robertson, made controversial remarks about, among other topics, gay men. A&E responded in December to Robertson’s comments by briefly suspending him from the show. Significantly, the GQ article and Robertson’s suspension happened after the last episode of Duck Dynasty’s fourth season had aired. What effect Robertson’s comments would have on the show’s ratings would not be known until Season 5.
The first two episodes of the new season of Duck Dynasty aired last week. According to several major media organizations, the number of people who watched these two episodes declined precipitously – “ratings dip in wake of controversy,” “big ratings drop,” “ratings are down dramatically,” “28 percent ratings drop,” “drastic drop in ratings.” The effect of Phil Robertson’s comments – or, more accurately, his comments plus A&E’s suspension and reinstatement of him – was now clear: Viewers apparently punished the hit show by not watching it. The problem is that this did not really happen.
Any claim of declining ratings requires at least two values, a before and after value. About 8.6 million viewers watched the final two episodes of the previous season of Duck Dynasty – the before value. About 8.5 million people watched the first two episodes of the new season of Duck Dynasty – the after value. If the number of viewers before and after the Robertson controversy was about the same, how was there a drop in ratings?
The “bad scientist” explanation is that claims of declining ratings reflect a misunderstanding of how to evaluate cause-effect relationships. Comparing how many people watched the first episode of Season 5 (8.5 million) to those who watched the first episode of Season 4 (11.77 million) is questionable. Season 4 consisted of 11 episodes. The ratings across these episodes varied from 11.77 to 7.26 million viewers, averaging 9.1 (with a standard deviation of ±1.75) million viewers. Comparing the number of people who watched the start of Season 5 with those who watched the start of Season 4 is a bit like evaluating the effect of fertilizing a plant in December by comparing its height in January to its height the previous August. Even if the fertilizer had no effect, you expect a growing plant’s height to change between August and January.
The “bad journalist” explanation is that claims of a ratings dip reflect a Charles Foster Kane approach to finding whatever evidence is necessary to support the story one wants to tell. Psychologists know this as confirmation bias, people’s tendency to favor information that confirms their views. Disgusted by Robertson’s comments, journalists looked for and found evidence to support their hypothesis that viewers of Duck Dynasty were equally disgusted and would punish the show by not watching future episodes. But to make this claim, bad journalists also behaved like bad scientists by comparing the ratings from the first episode of Season 5 with the first episode of Season 4, ignoring the ratings of the intervening ten episodes.
The “bad statistician” explanation is that claims of dropping ratings reflect a misunderstanding of statistical variation. Anything that can be measured, such as the number of people who watch a television show, can and usually does vary. Variation is a normal part of nature. Duck Dynasty’s ratings vary from episode to episode. In the season before Robertson’s comments were published in GQ, the ratings declined steadily from a high of almost 12 million viewers. Even beginning students of statistics know about regression to the mean: the extreme value of a variable changes over time to more closely approximate the average. Just as a baseball player who goes 4 for 4 at bat on Opening Day will hit 1 for 4 many games if he is a .250 hitter, the ratings pattern in Season 4 could be characterized as regression to the mean. The number of viewers after the Phil Robertson controversy (Season 5) was about the same as the average of Season 4.
In the end, someone could shrug their shoulders or roll their eyes at a discussion of Duck Dynasty’s ratings. It’s just entertainment news about a show that already gets too much publicity. That may be true, but major news organizations did cover this story and they did so often on the homepage (Page 1) of their websites. Rather than an unbiased coverage of the show’s ratings, major media outlets chose to behave like bad scientists, bad journalists, or bad statisticians. It’s worth asking: If major news organizations cannot cover television ratings – yes, ratings – without bias, can we trust them to write unbiased headlines and stories about important news in politics, health, technology, and business?