THE 2015 general election was the most polled election in British history. In the run up to the election barely a day went passed without a new poll (or often many) being released in the media. In the last month of the campaign alone, some 80 polls were published in the media by 11 different polling companies. Practically all of these pre-election polls predicted a neck and neck race between the Conservative and the Labour party. Of the eight polls released on the day before the election, seven had the Conservatives and Labour party vote share within one percentage point of each other, and the eighth had a two point lead for the Labour party.

The political parties, media commentators, betting markets and academics all fully expected a hung Parliament. The first indication that the pre-election polls might be wrong came on election day when the results from an exit poll carried out by the main TV companies showed a clear victory for the Conservatives. Initially the results from the exit poll were treated with a great deal of scepticism, and Paddy Ashdown, the former leader of the Liberal Democrats, declared that he would eat his hat if the results were accurate (which he ended up doing). As the official results were gradually declared throughout the night, it became clear that the exit poll was indeed correct, and the Conservatives ended up with a comfortable victory, beating the Labour party by six percentage points and securing an overall majority of the seats.

The fact that the pre-election polls got it so badly wrong matters in a number of important respects. The failure of the pre-election polls to accurately predict the election result caused a certain amount of reputational damage: if surveys cannot be relied upon to get these basic objective facts of elections correct, or even come close to the true mark, then why should they be believed in other respects as well? The polls’ inaccuracy also had important consequences for the election campaign. The media became obsessed with the idea of a hung Parliament, and much of the coverage focused on the horse race rather than the issues at stake. The Conservative campaign also focused on the threat of a coalition between Labour and the Scottish National Party, and the prospect of England being ruled by the Scottish.¹ There is also some evidence that expectations of a hung Parliament may have influenced people’s vote choice and made them more likely to support minor parties.² Of course, on the plus side, expectations about a close race are often thought to boost turnout; the fact that the opinion polls suggested a dead heat could have motivated some people to vote who otherwise may not have done so.

As soon as the discrepancy between the pre-election polls and the final election results became apparent, the British Polling Council (BPC) and the Market Research Society (MRS) set up an independent inquiry as to why the polls had been so inaccurate. The results of this inquiry have since been published,³ as have a number of academic studies.⁴ This article outlines some of the main findings from this research, and what lessons, if any, can be learned.

 

The pre-election polls in Britain were all carried out by commercial companies, mainly on behalf of media outlets. Before discussing why these polls got it wrong, it is worth saying a few words about how they are carried out. In the UK, surveys are carried out in three main ways: by telephone, on the internet, and face-to-face. The pre-election polls were all carried out on the telephone or internet using quota samples which were then weighted to known population totals. Although telephone surveys have been used for a long time in Britain, in recent years there has been a sharp increase in the popularity of internet surveys.

Over the last ten years, there has been a dramatic expansion in access to the internet, which now covers about 80% of the population. Online polls are based on panels of pre-recruited members who are then invited to take part in a survey. The advantage of online polls is that they are quick and cheap to carry out, and over the last few years have had some success at accurately predicting a number of electoral contests, such as the 2010 UK general election, the 2011 referendum on electoral reform, the 2014 referendum on Scottish independence, and the 2016 election for the London mayor.

 

By contrast, face-to-face surveys, based on random samples of the full adult population, are far more expensive and time consuming to carry out. Although they represent the ‘gold standard’ for survey research, because of high cost they are used much less frequently. The only survey that was carried out using these methods around the election was the British Election Survey (BES), which was carried out immediately after the election. The methodology for the BES is based on a probability sample of addresses drawn from the Post Office Address File. It thus covers people who may not be on the electoral register (unlike the Indian National Election Study which uses the electoral roll as its sampling frame). An interview is attempted with a randomly selected adult at each address, and multiple visits are made to each address at different times of the day and on different days of the week in order to achieve an interview. Interviews are carried out face-to-face by trained interviewers via questionnaires loaded on to laptop computers. The response rate for the BES in 2015 was 56%, which is not particularly high in historical terms, but is nonetheless good by contemporary standards.

 

The final poll averages for the four telephone polls and seven online polls that were carried out in the last few days of the campaign are shown in Table 1. Both types of poll underestimated the Conservative vote and overestimated the Labour vote, and indicated a dead heat between the two parties. Although there was no real difference between online and phone modes in the accuracy of the final polls, over the 2010-2015 Parliament and in much of the election campaign, phone polls produced somewhat higher (1 to 2 percentage points) estimates of the Conservative vote share.⁵

By contrast, the post-election British Election Study (BES), which was carried out using face-to-face interviews based on methods of random selection, produced more accurate results and correctly identified the Conservatives as the largest party. Although the BES overestimated the vote share for both the Conservatives and Labour somewhat (and underestimated the vote share for the minor parties), importantly it did manage to accurately identify the Conservative lead over the Labour party.

 

This simple table may appear to confirm much of what we know about survey methodology: that face-to-face surveys based on random selection of the full adult population are more reliable than quota surveys and internet and telephone surveys, which do not have full coverage of the adult population. Although there is certainly some truth in this claim, the comparison presented in the table is not necessarily fair. The internet and telephone surveys were carried out before the election, whereas the face-to-face survey was only carried out after the election. There are thus important issues about how vote intention translates into voting behaviour, and how turnout can be accurately predicted that need to be taken into account. Moreover, these points notwithstanding, it is not immediately obvious why both the telephone and the internet surveys, almost unanimously, underestimated Conservative vote, whereas the face-to-face survey did not.

 

How accurate are pre-election polls? This was not the first time the pre-election polls had got it wrong. In 1992, the polls predicted a slight lead for Labour only for the Tories to win the popular vote by some seven percentage points. Since then, the polling companies have changed the way that they carry out and report their polls, but their track record has nonetheless remained patchy. Over time, the pre-election polls have tended to overstate Labour’s vote and understate the Conservative’s share of the vote (see Figure 1). In 1997 and 2001, the pre-election polls substantially over-estimated the Labour share of the vote, but since Labour went on to win the elections quite comfortably no one paid much notice. However, when the polls fail to get the outcome correct, as they did in 1992 and 2015, they understandably attract much more criticism.

In the aftermath of the election a number of hypotheses were put forward in order to try and explain the failure of the pre-election polls. These hypotheses raise some important issues about how surveys are carried out, and the implications from these findings may also have relevance for how surveys are carried out in other places too.

 

The first hypothesis was that the pre-election polls were not necessarily wrong, but that there had in fact been a late swing. Thus, the polls were accurate at the time they were conducted, but between the last poll and election day a substantial number of people changed their mind, perhaps because they were concerned about the prospect of a hung Parliament. Reports into the polling failures at the 1970 and 1992 elections both attributed a prominent role to late swing. However, there is not much evidence to support this claim. A number of polling companies re-contacted respondents after the election to see if they had changed their mind since they were last interviewed. There is some weak evidence that there may have been some slight movement towards the Conservatives, but only in the range of half a percentage point or so. Thus, late swing is not able to account for why the polls has been so inaccurate.

The second hypothesis was that respondents had not been completely honest with the pollsters before the election, and had been reluctant to admit that they would vote Conservative. Since opinion polls have historically under-reported Conservative vote share, this has led to the idea that many Conservative supporters may in fact be ‘shy Tories’. Thus, the reason why the polls were inaccurate is because Conservative supporters would either lie about who they were planning to vote for, refuse to answer the question, or say that they hadn’t made up their mind yet or that they didn’t know. The reasons why Conservatives might do this are not well developed. The phenomenon of the ‘shy Tory’ draws on ideas related to social desirability bias – and the so-called ‘spiral of silence’ where respondents are unwilling to admit that they hold unpopular or anti-social views. Thus, the reasoning goes, people don’t like to admit that they vote Conservative because they think it makes them look mean-spirited and selfish.

The obvious problem with this interpretation is that respondents didn’t appear to be particularly embarrassed about saying that they voted for Ukip, which is far more extreme than the Conservative party on a number of issues. Moreover, there was no difference between the telephone and the internet surveys. If people really did feel embarrassed about saying they were going to vote Conservative, perhaps they would feel even more embarrassed when they have to say it out loud to someone on the phone than when they can just click a button in the anonymous setting of an internet poll. Finally, the results from the post-election British Election Study were close to the actual election figures, and if people weren’t lying then it is difficult to see why they would have lied beforehand.

 

If voters did not change their mind, and were not lying – the obvious culprit for why the pre-election polls got it wrong is to do with the methodology the polls used. But how did this methodology affect the representativeness of their samples? And why did it lead all the polls to overestimate the Labour share of the vote and underestimate the Conservative share of the vote?

Part of the reason for this is to do with how the polls select their samples and how they then adjust – or weight – their data to make it more representative of the population.⁶ The telephone polls have reasonably good coverage of the adult population – though with the decline of landlines this coverage probably isn’t as good as it was – but they achieve very low response rates, which can lead to biased samples. By contrast internet polls, that rely on a panel of volunteers who are paid to complete surveys, have poor coverage of the adult population and are not at all representative of the adult population (though the exact composition of these panels are shrouded in secrecy). The accuracy of both types of poll thus depend upon the adjustments that are made to the data, either after it has been collected in the form of weighting, or before the sample has been selected in terms of quotas. The more data has to be adjusted to make it ‘look’ representative, the more scope there is for something to go wrong.

 

Mellon and Presser’s analysis shows that the pre-election polls oversampled politically engaged voters who turn out to vote at much higher rates than the average British person.⁷ To a certain extent this should come as no surprise. People who like to complete these sorts of surveys (indeed any survey) tend to be much more interested in politics and much more active in politics than those people who stay away from surveys all together.

British polls weight their samples to known population level targets on a range of different socio-demographic variables, such as age and education. This process means that the under-sampling of non-voters and the disengaged leads to the overweighting of respondents who share some demographic characteristics with non-voters. In particular, the young people that the pre-election polls sampled were much more interested in politics than young people in general, which led the pollsters to think that more young people would vote than actually did so, and since young people are more likely to vote Labour than Conservative, this biased the sample towards Labour. As Mellon and Prosser conclude, in 2015, these overweighted respondents were disproportionally Labour supporting, inflating the estimate of the Labour vote.⁸

 

This problem can be regarded as a type of availability bias, which results in part from response bias. However, it should be noted that this kind of availability bias is not just a problem with internet and telephone surveys, but can also be a problem with face-to-face surveys as well. This can in fact be neatly illustrated with the British Election Study. As already mentioned, an important aspect of the methodology used by the BES is to make repeat efforts to contact individuals at selected addresses, and multiple calls are made at different times of the day on different days of the week in order to achieve an interview. Substitutions for sampled respondents who were not reached or who declined to be interviewed are not permitted. In some cases, this meant that interviewers made more than 10 different calls to an address.

One major strength of the BES then is that it makes a great deal of effort to contact the sort of people who might not normally be available to take part in a survey. Mellon and Prosser show that the sort of people the BES managed to interview on their first attempt are different in a number of important respects to the sort of people that they managed to interview after repeated attempts, and by taking steps to reach ‘hard to contact’ groups of voters the BES managed to secure a more representative sample.⁹ For example, men who were interviewed after the first contact reported turnout rates 7.6 percentage points higher than women. However, the final sample closed this gap to 0.7 percentage points after multiple contacts were made.

 

There are a number of valuable lessons to be drawn from the investigations into why the pre-election polls got it wrong. Perhaps the strongest lesson is to reaffirm the value of methodologically rigorous probability surveys and the importance of a high response rate. If the BES had not made such a concerted effort to achieve a high response rate then it would have suffered from some of the same problems as the pre-election polls, though perhaps to a slightly lesser extent.

In recent years, academics have come to use internet surveys more and more. Indeed, for the first time ever, the BES did not include a pre-election face-to-face component as part of its design and relied upon an online panel instead. However, the use of internet surveys for academic research must surely now be seen as problematic. Not least, because as has been carefully shown, they are not based on representative samples, even after adjustment. One of the key recommendations of the polling inquiry is to include a pre-election face-to-face survey for the next BES.

But even though online and telephone polls have a number of clear methodological flaws, it is unlikely that they will be replaced by probability based surveys any time soon, particularly when it comes to their use in the media.

Given then that non probability samples are here to stay, how can they be improved? There are only really two options. The first is to try and improve the representativeness of their recruitment process – and make greater effort to sample the ‘hard to reach’. Though, in practice, there is not much that can be done to achieve this. The second is to try and develop new weighting variables to make the samples more representative (such as political interest). However, this still treats the symptoms of the error rather than treating the cause.

A more fruitful option would be to think of new ways in which online surveys can be incorporated within a random selection process. Such surveys have been carried out in a number of European countries, such as Holland. Samples are selected from the population at random – and then recruited to online surveys by face-to-face interviewers. If the selected respondents do not have access to the internet, they are provided it (and a tablet) by the polling company, thus minimizing coverage error. However, such surveys are expensive.

 

The findings from the investigations as to why the UK pre-election polls got it wrong have a number of implications for thinking about how survey research in India is conducted. One way in which the accuracy of surveys is measured is in terms of how close they get to the final vote shares of the parties. This is undoubtedly important. But another way to assess the accuracy of surveys is in terms of how close they get to the final turnout figure. One problem with the British pre-election polls was that they over-sampled the politically interested and politically active who were most ‘available’. In this respect surveys in India also tend to show much higher levels of turnout than are reported by official statistics.

One way to improve the representativeness of surveys is therefore to try and interview people who may not be likely to vote. There are various ways in which this can be done. The most important way is to get a high response rate and to make multiple visits to each selected address on different times of the day and on different days of the week, and not to allow substitution. Post-election surveys in India are often carried out in a short time frame because they prefer to carry out the interviews before the election result is known since after the result is declared, many people say they voted for the winner. However, this compressed fieldwork may make it difficult to sample ‘hard to reach’ respondents.

 

Another way to improve the representativeness of surveys is to interview people who are not on the electoral register. Election surveys in India generally use the electoral roll as a way of selecting respondents. This is fine if we are only interested in how people vote, but it does not help us to understand the differences between voters and non-voters, or what Indians in general think about politics and other issues. There is also an issue about how reliable electoral rolls are, particularly in between elections. One option might therefore be to consider alternative sampling frames that are based on the selection of households rather than individuals.

Lastly, the failure of the UK pre-election polls also highlight what is one of the key strengths of electoral research in India: methodologically rigorous samples based on random selection, conducted by face to face interviews. These surveys may be expensive and time consuming to carry out, but they are a precious resource and are of enormous value.

 

Footnotes:

1. P. Cowley and D. Kavanagh, The British General Election of 2015. Palgrave Macmillan, UK, 2016.

2. Jane Green, Edward Fieldhouse and Christopher Prosser, ‘How Coalition Government Leads to Minor Party Voting (in the Absence of Proportional Representation): A Case Study of the 2015 British General Election’, Working paper, 2015.

3. P. Baker Sturgis, N. Callegaro, M. Fisher, S. Green, J. Jennings, W. Kuha, J. Lauderdale, B. and P. Smith, ‘Report of the Inquiry into the 2015 British General Election Opinion Polls’, Market Research Society and British Polling Council, London, 2016.

4. J. Curtice, ‘The Benefits of Random Sampling Lessons from the 2015 UK General Election’, NatCen Social Research, London, 2015; J. Mellon and C. Prosser, ‘Missing Non-Voters and Misweighted Samples: Explaining the 2015 Great British Polling Miss’, Social Science Research Network, 2015; D. Rivers and A. Wells, ‘Polling Error in the 2015 UK General Election: An Analysis of YouGov’s Pre and Post-Election Polls’, YouGov UK, London, 2015.

5. P. Baker Sturgis et al., op. cit., 2016, fn. 3.

6. O. Heath, ‘Why the UK’s Pre-election Polls Got it so Wrong: is it Time to Take Probability Sampling Seriously?’ LSE British Politics and Policy blog. http://blogs.lse.ac.uk/politicsandpolicy/why-the-pre-election-polls-get-it-so-wrong-is-it-time-to-take-probability-sampling-seriously/

7. J. Mellon and C. Prosser, op. cit., 2015, fn. 4.

8. Ibid.

9. Ibid.