Are newspapers killing political polling?


1:25 pm - October 18th 2012

by Leo Barasi    


      Share on Tumblr

Each week, YouGov conduct six political polls for News International. ComRes poll for the Independent about once a fortnight; ICM, MORI and Populus do monthly political polls; Opinium seem to be polling on a weekly basis; and Survation and TNS have irregular but frequent polls.

Compare this with 10 years ago. According to Mark Pack’s list of past political polls, this week in October 2002 had just two polls. Ten years before that, there was one poll in the same week. So political polling now appears in unprecedented health.

But perversely, political polling contains the seeds of its own destruction. Here’s why.

Polls are commissioned for one of two reasons. Either the commissioner wants to know something, or they want someone else to know something.

Most polls are never made public. They’re commissioned by companies that want to know how they’re viewed, or to test ideas, or to see what people think about a question that’s important to them. These are often of little interest to anyone who’s not directly involved, though sometimes they contain some fascinating insights.

However lots of polls are made public. Some of these are by organisations trying to create a story that helps their cause. A recent example was this Populus poll, commissioned by the Tories and credulously reported by the Guardian despite a question sequence carefully designed to give Ed Miliband a bad score.

But most public political polling isn’t done by campaign groups: it’s done by newspapers, and they do it because they want a good story that sells papers. This is where the problem is.

Suppose five different polling firms each interviewed five different random samples of 2000 people from the UK population on the same day, using exactly the same top-quality methodology. Due to natural effects of sampling we would expect the results to be different. They probably wouldn’t be wildly different but there’s a good chance that at least one of them would have a quite different result in the overall figures, and when you start digging into subgroups it’s almost certain there’ll be big differences.

This variation is excellent for newspapers. A result that shows change from a previous poll is a news story. Take ComRes’ poll for the Independent, published on the day of Miliband’s speech to Labour conference, and showing Labour’s lead as just 3pts. It made a fantastic headline: far better than they would have got if they had found the same result as almost any of the preceding seven polls by other firms, which was a Labour lead of between 9-13pts.

In short, newspapers are crossing their fingers that their exclusive poll will be the outlier, the statistical freak that screams “something’s changed”.

But the irony is, the more successful political polling has become as an industry, the less value there is for newspapers to commission polls. Yet newspapers are the group that keep political polling going*.

Back when polls came out just once a week, no-one could tell whether a poll that showed a dramatic change was a true reflection of opinion or just a rogue. Now, we can be pretty confident, within a couple of days at most, whether there’s been a change. It takes seconds, when hearing about a new poll, to go onto UK Polling Report and check how it compares with others. The emperor’s clothes are threadbare at best.

In case you’ve got to this point and think I’m seriously predicting the imminent death of political polling: I’m not.

Of course newspapers will carry on commissioning polls and hoping they’re lucky enough to get an outlier. Most of their readership only read one national paper and don’t go checking up on polling websites (more’s the pity).

But as readership trends change and people’s news comes from more individually-tailored sources, those things could change. People who have even a passing interest in the results of political polls will have increasing access to a range of polls, not just their own newspaper’s poll – if they even continue to have a newspaper they see as their own. The flaw in the model may yet be exposed.

 

* Polling firms make most of their money from non-political work and typically do the political stuff for the PR value. But if it wasn’t for the newspapers promoting their polls, it’s hard to see that they’d do political polling at all.

    Share on Tumblr   submit to reddit  


About the author
Leo is a regular contributor to Liberal Conspiracy. He manages communications for a small policy organisation, and writes about polling and info from public opinion surveys at Noise of the Crowd
· Other posts by


Story Filed Under: Blog ,Media

Sorry, the comment form is closed at this time.


Reader comments


Good article.

I like to think that News International began the daily poll soon after the last General Election to track attitudes tilting towards their more rightwing agenda.

Of course it hasn’t turned out that way, which I like to imagine must stick in the craw.

1. BenM

I wasn’t aware that NI had a “rightwing agenda”. Right of centre certainly, but nowhere near as far right as the Grauniad is to the left. Left of the Telegraph too.

“Polling firms make most of their money from non-political work and typically do the political stuff for the PR value”

And by and large the big firms are absolutely shite. Yougov constantly waste my time by asking me to do dreadfully designed ‘brand’ surveys that cannot possibly be of any value to their clients. Ipsos Mori are overpriced crap that constantly recycle poorly designed surveys for their long list of public sector clients (nobody gets fired for choosing them, even when far better companies offer a better deal), and many of the others just take cash for telling their clients what they want to hear (in a powerpoint format).

4. Chaise Guevara

Thanks for an interesting article. I knew newspapers like to sieve detailed poll results to get “newsworthy” stats (does the poll say violent crime is down on last year? Just drill down to knife crime among 25-30 year olds and get the fluke stat that’s risen! Headline!) but I hadn’t considered them actually commissioning polls in the hopes of snagging an outlier. How much do these things cost?

5. Chaise Guevara

@ JC

“I wasn’t aware that NI had a “rightwing agenda”. Right of centre certainly, but nowhere near as far right as the Grauniad is to the left. Left of the Telegraph too.”

Meh, depends where you put the centre. Most people define it as “me”. You can go off public opinion, but that’s difficult when public opinion is swayed by News International in the first place.

This makes me think of Philip K Dick’s short story The Minority Report. There they have three mutants with precognitive powers who predict murders so the police can prevent them from happening. The police chief discovers that he, himself, is apparently going to commit murder so goes on the run. He later discovers that the three precog reports do not necessarily agree because each report is effected by changes initiated by the previous report, and that the knowledge of the murder not only makes it possible for the police to prevent it but also potentially allows him to change his own behaviour so he doesn’t murder anyone (as predicted in the ‘minority report’).

All of which is my roundabout way of saying that ‘predicting’ voter behaviour actually changes that behaviour – even if the poll had been 100% accurate in the first place.

A poll that predicts a huge turnout for one party might encourage voters for another party to turn out in droves, or to vote tactically for a third party with a chance of winning, or it might induce over-confidence in voters for the first party so they don’t bother to actually vote. Polls don’t give a snapshot of behaviour – they create it.

7. Chaise Guevara

@ 6 Shatterface

I think that polls tend to help out whoever they report as being in the lead, but to be honest I can’t recall where I’m getting that from. It was certainly my impression of why exit polls can’t be revealed until 10 o’clock on election night.

Incidentally, I for one would approve of all political discourse being done through P. K. Dick similes.

8. margin4error

Shatterface.

Spot on. Though that doesn’t make polls bad. The best example right now of what you suggest is Ed Miliband. He is under no internal threat as labour leader in part because labour are well ahead in polling. That in turn makes labour better able to focus on policy and the wider agenda rather than on internal rows as new oppositions often do. And that in turn strengthens public opinion on labour.

At the same time Cameron and Clegg face internal politics because their parties,are polling badly, and that in turn makes their patties look weak and fractured, which further hurts their poll ratings.

@8 margin4error

I love the idea of weak political patties.

More relevantly, more data can feed into a higher form of analysis such as Bayes Theorem and give us far better predictors of the outcome of the election. Nate Silver has a good track record on this and blogs here: http://fivethirtyeight.blogs.nytimes.com/

If the data are of a comparable, good quality then presumably the analysis will improve too.

You make some good points in the post. The ironic conclusion perhaps is that rather than having lots of opinion polls, we actually have too few. If we’re interested in small changes in public opinion (and those in the media and politics by and large are) we need a much larger volume of polls in order to produce data with the commensurate sort of accuracy.

(I’ve expanded on this point at http://www.markpack.org.uk/36013/the-pointlessness-of-political-opinion-polls/ )

PS Always good to see my spreadsheet put to use :)

Ditto.

One of the big problems with the reporting of opinion polls is that most peop[le, unless rigerously trained and actively involved, don’t understand statistics. this is made worse by the fact that most journalists come from an arts, rather than science/maths background. Consequently facts like the design of opinion polls mean that are designed to be 90% accurate goes over people’s heads, who do not realise that this means 10% will be badly wrong, and also that a margin of error of 3-5% means that only a change greater than this is significant. Consequently what happens is that the rogue polls get on the front page, and the accurate ones on page 19.


Reactions: Twitter, blogs
  1. Jason Brickley

    Are newspapers killing political polling? http://t.co/YHZqBcSx

  2. wappingate.com

    Are newspapers killing political polling? http://t.co/GH4LBrXk #notw #wappingate

  3. leftlinks

    Liberal Conspiracy – Are newspapers killing political polling? http://t.co/r4DcOARr

  4. Hermes Trismegistus

    Are newspapers killing political polling? | Liberal Conspiracy http://t.co/FmFM7DQe via @libcon #ukpoli

  5. Ready For Anything

    Are newspapers killing political polling? | Liberal Conspiracy http://t.co/FmFM7DQe via @libcon #ukpoli

  6. The pointlessness of political opinion polls

    […] Leo Barasi’s piece over on Liberal Conspiracy raises an interesting point about the frequency of political opinion polling in the UK. We now have far more polls than before giving national voting intention figures (this Parliament so far: 878, 2001-5 312 in total, 1987-92 548 in total – to give some examples). But do we have too few? […]





Sorry, the comment form is closed at this time.