The Trouble With Polls

In a recent post [The Role And Toll Of Polls], I addressed some issues with polls that relate to bias that can be introduced into them even if technical rules that ensure randomness are followed to the letter. One example was simply the words and phrases used in formulating the questions in the poll. Some articles I read this week about the results of some polls brought to mind another issue that should prompt all of us to be a bit more deliberative in interpreting media coverage about their results.

Just The Facts, Ma’am

For those of you who remember the Dragnet TV series [1950s to 1960s], Sergeant Joe Friday’s often-used line in interviewing witnesses of crimes is germane here — in response to a witness asking, “Well how much detail do you want to know?”, his answer was “Just the facts, Ma’am.”

Unfortunately, it’s not “just the facts” that the media reports about poll results. Without getting into the weeds of the statistical theory and mathematical formulae, I’d like to just present how the results of one of the polls I read about were characterized — how the article was headlined, and the clear narrative intended in the article’s content — as compared with what conclusions can realistically be drawn from the actual numbers [“just the facts”].

In case it matters, I had a Minor in Mathematics in earning my degree with a Major in Engineering, and an area of considerable concentration within that Minor, as well as within my Major [Industrial and Systems Engineering] was Probability and Statistics.

The Survey

The Senate seat being vacated by Senator Bob Corker’s [R-TN] decision not to run for re-election has become a Democrat “flip” priority. Chuck Schumer personally recruited former Tennessee Governor Phil Bredesen [a Democrat, of course] to run for the seat. Bredesen is several links up the food chain from the typical Democrat running in Tennessee [usually just placeholders because nobody thinks they’re going to win anyway]. He was a very popular Governor considered by many in both parties to have been a very effective one as well.

In the August 2 primaries, hardly anybody would argue against the assumption that Bredesen is a shoo-in for the Democrat Party nomination, or that U. S. Congresswoman Marsha Blackburn [R-TN] is a shoo-in for the Republican nomination — Bredesen because of my foregoing observations, and Blackburn because she’s a well-known and highly-respected Republican in a deep red state [in my opinion, she’s one of the sharpest knives in the House drawer].

This survey attempted to determine how Blackburn and Bredesen would fare in the November 6 General Election.

The Media Outlet Reporting The Results

The Tennessean is about as liberal as you can get in the media world, and it is an affiliate of the USA Today network, which is also clearly liberal. So, it is certainly no surprise that any characterization of the results of this [or any] poll in that publication would support the narrative of a “Blue Sweep” in the 2018 mid-term elections. They will no doubt report on many surveys between now and then, and their characterizations will all be tilted in the same way. If Blackburn wins, but by less than ten points, they will then join narrative like that following the recent 52.6 percent to 47.4 special election victory of Debbie Lesko [R-AZ] in a district that President Trump carried by more than 20 points in 2016 — “more evidence that the margin is closing and setting the stage for a huge retake of the U. S. Government by Democrats.”

The Facts Versus How The Results Were Reported

How Reported. … The headline read “Democrat leads Marsha Blackburn by 3 points in new Tennessee Senate poll.” Within the article, the following statement is inaccurate at worst, and misleading at best: “Democrat Phil Bredesen is ahead of Republican Rep. Marsha Blackburn in the race for an open Tennessee Senate seat. … Bredesen led Blackburn narrowly, 46 percent to 43 percent. … 11 percent were undecided.” Although the next sentence seems like an attempt at “balanced” reporting, [i.e., by “disclosing” a caveat], even it is misleading: “Bredesen’s lead is within the 4-point margin of error.”

Just The Facts. … These are the only conclusions that the survey revealed could reasonably be quoted as supporting with a 95% chance of being correct:

• Percent of people supporting Bredesen: 42.1 – 49.9
• Percent of people supporting Blackburn: 39.1 – 46.9
• Percent of people who are undecided: 7.1 – 14.9

    👉🏿  So the facts are that …

• Bredesen might be ahead of Blackburn by as much as 49.9 to 39.1 or as little as 49.9 to 46.9, or
• Blackburn might be ahead of Bredesen by as much as 46.9 to 42.1, and
• The percent of people who are undecided might be as little as 7.1 or as big as 14.9.

    👉🏿  And the only conclusion that can be reasonably substantiated is …

• This poll is meaningless in terms of predicting any possible outcome of this election.

Our Takeaway

Minimizing Interval and Confidence Levels in most of the poll reporting we see may avoid mesmerizing readers / listeners / viewers with details that might result in loss of their interest in the coverage. However, it also enables creation of impressions that simply are not warranted. Each of my readers can decide on his/her takeaway from this post. My takeaway is “Never trust any one media outlet’s characterization of the results of a poll; either a) take the time to do the math yourself and form your own conclusion with the logic I’ve used here, or b) access multiple sources of ‘news’ / ‘fake news’ and try to net their views into your own composite.”

…

Thanks for reading this post, and if you regularly follow my Blog, for that, too. Please consider sharing this or other posts with your friends, colleagues and associates.

 

Charles M. Jones