POLLS are an integral part of political campaigns in the United States. News headlines highlight the latest polling results, pundits dramatize daily fluctuations in horserace numbers, and candidates either trumpet a lead in the polls or selectively dismiss them when they are behind. The number of election polls in the US has also dramatically increased over time. Traugott¹ estimated a 900% increase in the number of pre-election polls between 1984 and 2000, and the volume of polls has continued to grow since then, due largely to the rise in internet polls and interactive-voice-response polls (IVR; also called robocall polls). The current US presidential election is now shaping up to be the most polled in history, with a record number of polls (more than 1000) already conducted during the nomination stage.

Despite the prominence of election polls in every campaign, there have also been spectacular polling failures in recent elections around the world – pollsters underestimated widespread Republican gains in the 2014 US congressional elections, they miscalled national elections in Israel, Britain, and Poland in 2015, and they overestimated support for Hillary Clinton in the 2016 Democratic primary election in Michigan.² Some now fear that ‘polling is teetering on the edge of disaster’.³ In this essay, we highlight some of the challenges and limitations of pre-election polls, some of which reflect global and industry-wide methodological issues and others that are specific to the American context. Ultimately, we view polls as a powerful tool for understanding elections, but we caution about their frequent misuse for election prediction and emphasize the need for methodological transparency.

 

Election polling is in a state of crisis. People are harder to reach and, when you reach them, they are less likely to cooperate. Response rates for pre-election media telephone polls now hover around 9% in the US. The issue is not limited to US telephone polls; declining response rates have plagued surveys across all modes (including mail and in-person); across all sponsors (including media organizations, academic institutions, and governments); and across most countries around the world.⁴

As traditional polling has become more expensive and difficult, there has been a rise in alternative polling methods such as online non-probability panels, IVR polls, and text message surveys on mobile devices. The combined effect of a proliferation in the amount of polling and increased variability in polling methodology makes it more difficult than ever to make sense of polling numbers.

The changing media environment has further complicated the interpretation of election polls. Major media organizations are no longer the primary gatekeepers of polling quality, either for producing polls or for interpreting polling results. While polling was traditionally conducted by a small number of news organizations and polling firms, new and less expensive polling methods have removed the barrier of entry in the polling industry. Much of the increase in the number of polls has been ushered in by a host of bloggers, independent news websites, and self-funded entrepreneurial pollsters trying to garner media attention, often using more dubious polling methods. For instance, about half of all polls conducted before the 2016 Democratic Primary in Michigan, the outcome of which pollsters overwhelmingly failed to predict, were IVR polls. The fragmentation and polarization of the media environment also mean that the public hears about different polling results from different sources, with partisan news sources often criticizing polls for their results rather than the methods used to obtain those results. It is perhaps no wonder that there is now a ‘haze of skepticism surrounding the entire industry’.⁵

 

Given this polling environment, it is important to recognize that while no poll is perfect, there is significant variability in poll quality based on the methodology used. While a detailed discussion of survey and polling methodology is beyond the scope of this essay, we briefly highlight some of the survey design factors that shape polling results.⁶

Random sampling serves as the foundation and scientific basis of public opinion polling. Rather than measuring the attitudes and beliefs of every citizen, pollsters are able to generalize, within a margin of error, from a sample of respondents to the larger population, if that sample was randomly drawn. Pollsters often joke that if you do not believe in random sampling, the next time you have a blood test, tell the doctor to take it all.

Until recently, one of the easiest and most cost-efficient methods of drawing a random sample was through Random Digit Dialing (RDD) telephone surveys, by which computers assist pollsters in dialing random landline telephone numbers until potential respondents are reached. For much of the late 20th century, the large presence of telephones in American households made this a cost-efficient way to conduct polling. Today, however, cellphones are increasingly replacing landline telephones. Nearly nine out of ten Americans use mobile phones, while nearly one half of American households, many of them young, poor, and ethnically diverse, do not have landline telephones.⁷

 

While the replacement of traditional landline telephones with cellphones might not seem problematic for random sampling in theory, federal legislation in the US prohibits computer assisted calls to cellphones. Cellphone numbers must instead be dialed by hand, which is far more expensive and time intensive than computerized dialing that only connects the interviewer once a live respondent has been reached.⁸ By the 2016 election, a vast majority of pollsters were including some percentage of cellphone respondents in their samples, but there is wide variability in the percentage of the cellphones included in the sample, how they are integrated, and how cell-phone-mostly households are handled.⁹ A further complication is that cellphones, unlike landlines, are not tied to a fixed address and it is not unusual for owners to have a different area code than where they actually live.

 

Internet polls have emerged as a less expensive and time intensive alternative. Online polls offer a number of advantages when a full list of email addresses is available for a population of interest, such as students at a school or members of a company, but obtaining a sample of the general US population presents a sampling challenge. While 84% of Americans use the internet, there is no sampling technique available to internet pollsters that functions like the random draw from telephone numbers, no master list of email addresses or a way to randomly sample IP addresses. As such, most online polls are not random samples – they tend to rely on non-probability, opt-in samples that require strong and untestable assumptions to generalize to the broader population.¹⁰ Moreover, different sampling practices and statistical adjustments (rarely publicly reported) result in wide variability in the accuracy of online polls across different vendors and survey topics.¹¹

In addition to sampling challenges, variability in polling results can reflect a variety of other design decisions – the timing of a poll, days in the field, question wording, question order, and so on. For example, during the recent US recession, asking respondents to rate their approval of President Obama after evaluating the state of the economy produced lower approval ratings than if the questions were asked in reverse order.

Given the large number of polls and high variability in methodological techniques, polling aggregations are an increasingly popular approach for making sense of polling results. Polling aggregators, like Nate Silver’s FiveThirtyEight.com, Real Clear Politics, and Huffpollster, combine and average polling results from many sources. The particular method used varies across aggregators, with some accounting for factors like polling firm performance, recentness of the poll, and other factors. Pooling across polls improves the precision of polling estimates, since the random error in one poll should cancel that out in another.

Unfortunately, it appears that some pollsters are also watching the aggregation websites and subsequently adjusting their polling numbers or failing to release their polls if their numbers look out of line with the polling average. Poll herding, whereby the polls show less variation across firms than would be expected based on statistical sampling theory, makes it more likely that the overall polling average will be wrong. Polling trends and averages may offer a better way to make sense of polling numbers, but it doesn’t mean we can entirely ignore the quality of the polls going into the mix. As Cliff Zukin observes, polling aggregations ‘are only as good as the raw material they have to work with.’¹²

 

Even if all polls employed rigorous sampling techniques, our view is that we still must be cautious in using polls for prediction, especially when those polls are conducted early in a campaign. Too often, pollsters ask about the national horserace without considering the broader institutional and electoral environment that helps provide context to polling numbers.

First, election polls in the US are often conducted at the national level, but the election outcome is decided by the Electoral College.¹³ While it is rare for a president to lose the popular vote but win the Electoral College, as happened to George W. Bush in 2000, it does mean that the campaign is hard fought only in a handful of battleground states. While polling is sometimes conducted at the state level, these polls tend to be fewer in number and lower in quality than national polls. For instance, in the year leading up to the Democratic Primary in Michigan, which was described by pollsters as a shocking historical upset, there were nearly 100 polls conducted nationally but only 18 conducted in Michigan, according to Real Clear Politics. Many of the better election polls are conducted by national media organizations, whereas state-level media organizations rarely have the resources to devote to quality polling.

Second, polls face the challenge of identifying and surveying a random sample of the correct population, that is, citizens who actually vote. Voter turnout in US elections is among the lowest among OECD countries and varies across states and elections. Turnout for the 2012 presidential election was 61%, and just 36% in the 2014 US midterm election.

 

Polling organizations have to make a guess as to which respondents are in this relatively narrow population of interest using a so-called likely voter model. Likely voter models vary widely in design and quality though, and because they are often proprietary the exact methodology used is often not transparent. Unfortunately, these models are notoriously error prone.¹⁴ Some models involve simply asking how likely the respondent is to vote – an approach that tends to wildly overestimate voter turnout when asked many months in advance of an election. Other pollsters try a more conservative approach by incorporating other factors like knowledge of polling location or turnout history, but this causes problems by leaving out new voters. If these new voters disproportionately support one candidate over another, as was the case in the last two US election cycles, this too can lead to biased predictions.

 

Political scientists and campaign practitioners have improved likely voter models through the use of voter registration records and more complex methods, but their use is more time consuming and expensive.¹⁵ Given how difficult it is to get a respondent to answer a poll these days, it is perhaps not surprising that survey firms are hesitant to throw out a respondent as a potential non-voter. Media pollsters may, unfortunately, be more interested in creating immediate headlines than obtaining the highest quality survey estimate. Polling in the 2016 US primary highlights the sometimes vast discrepancy between pollster estimates of the voting population and actual voters: a recent Survey USA poll in Florida identified 80% of their sample as likely primary voters, while actual primary turnout in Florida was a record high at 28% in 2016.

The other difficulty with likely voter models – and pre-election polls more generally – is that voters frequently change their minds prior to election day. Voters may intend to vote when asked in a poll several months before the election, but ultimately decide that it is not worth battling bad weather to go to the polling place on election day. Voters also change their mind regarding which candidate they intend to vote for. While the majority of voters are rather stable in a presidential campaign, reflecting the strength of partisan loyalties, there is enough volatility in candidate preference to be consequential to predicting close races. Cross-sectional election polls often find a smaller percentage of so-called ‘undecided voters’ in every poll; but these undecided voters are not the same in each poll. In other words, people move in and out of being undecided so that the total percentage of individuals who move their vote choice at some point during the campaign is much higher than any given polling snapshot might suggest.

 

These preference changes often occur in the closing days of the campaign, creating a bias between the polling estimate and the election result. Indeed, the standard pre-election polling question asks respondents not how they expect they will vote on election day, but instead how they would vote if the election were held today. This helps to reduce the number of undecided voters for reporting, but especially for multi-candidate races, such as primaries in the US, there can be a rational discrepancy between a voting choice made today compared to an actual ballot cast. In multiple candidate races, voters often behave strategically, taking into account not only their preferences but also the candidate’s chance of winning. For example, voters routinely change their vote choice in the primary elections as they learn that one or another candidate seems to have a better shot at beating the opposing party in the general election.

Further complicating the interpretation of the polling results in the United States is the complex voting calendar. Primary elections are held sequentially, so that the election results from one state might very well shape an individual’s assessment of a candidate’s viability and electability in subsequent states. In the general election, about one-third of voters cast their ballots early, sometimes several weeks before election day, although this is rarely accounted for in tallying and reporting polling results.

 

All of these issues mean that we simply ask too much of even the best polls and, in doing so, actively set up the industry for failure. This is not to say that polls are not useful tools for understanding voter behaviour – polls are powerful for many purposes when used correctly. The value of polls is in explaining why people vote the way they do – the meaning of elections rather than predicting the election outcome early in a campaign. As one political observer put it, ‘There isn’t a world polling problem. There’s a news media fixation on predictions problem.’¹⁶

Polls can be one piece of the prediction puzzle, but the inherent uncertainty in predicting elections can be overlooked in media coverage that focuses on horserace numbers reported to the nearest decimal place. In addition to the latest polling numbers, political science theories of voting behaviour can help inform election predictions. First, the overwhelming majority of individuals vote along party lines. For instance, in the 2012 Presidential election, 92% of partisans voted for their party’s candidate.¹⁷ It turns out that self-described independents, when categorized based on the party they feel ‘closer to’, largely behave like partisans as well, with many of the remaining pure independents not showing up to vote in US elections.

Beyond party identification, presidential approval and evaluations of the economy tend to explain much of the variation in vote choice. Finally, any salient policy issues that might divide traditional partisan coalitions can help to predict which particular partisans are more likely to abandon their party nominee in the election.¹⁸ Unfortunately, pollsters often fail to include these additional measures in their questionnaires, or journalists fail to analyze anything beyond the horse-race numbers. As pollster Gary Langer put it, a poll that asks only about the horserace ‘reduces the product to a numerical widget, with a shelf-life of 15 minutes and a value of 15 cents.’¹⁹

We should temper our expectations about the predictive power of horserace polling, recognizing the uncertainty created by campaign dynamics and the institutional features of elections. When we do use polling to measure the pulse of an election, greater emphasis should be placed on evaluating polling methodology and content – how the poll was conducted and what was asked of respondents. Polling quality should be evaluated on the basis of methodology and content, which requires greater transparency in survey design and implementation, as well as recognition from journalists and scholars about the limitations and challenges of election polling.

 

Footnotes:

1. M.W. Traugott, ‘The Accuracy of the National Preelection Polls in the 2004 Presidential Election’, Public Opinion Quarterly 69(5), 2005, pp. 642-654.

2. C. Zukin, ‘What’s the Matter with Polling?’, The New York Times, 20 June 2015.

3. M. Blumenthal, ‘Polling: Crisis or Not, We’re in a New Era’, The Huffington Post, 6 June 2016.

4. R.M. Groves, ‘Three Eras of Survey Research’, Public Opinion Quarterly, 75(5), 2011, pp. 861-871.

5. D.S. Hillygus, ‘The Evolution of Election Polling in the United States’, Public Opinion Quarterly 75(5), 2011, pp. 962-981.

6. D.S. Hillygus, ‘The Practice of Survey Research: Changes and Challenges’, in A.J. Berinsky (ed.), New Directions in Public Opinion. Routledge Press, 2014; D.S. Hillygus et. al., ‘Professional Respondents in Online Survey Panels’, in M. Callegaro et. al. (eds.), Online Panel Research: A Data Quality Perspective. John Wiley & Sons, 2014.

7. S.J. Blumberg and J.V. Luke, ‘Wireless Substitution: Early Release of Estimates From the National Health Interview Survey, January-June 2015’, National Center for Health Statistics, 2015.

8. While respondents often decline to be compensated, pollsters who dial cellphones to reach respondents must also bear the cost of reimbursing them for the minutes used to complete the calls.

9. M. Mokrzycki, S. Keeter and C. Kennedy, ‘Cell-Phone-Only Voters in the 2008 Exit Poll and Implications for Future Noncoverage Bias’, Public Opinion Quarterly 73(5), 2009, pp. 845-865.

10. D.S. Yeager, J.A. Krosnick, L. Chang, J.S. Javitz, M.S. Levendusky, A. Simpser and R. Wang, ‘Comparing the Accuracy of RDD telephone Surveys and Internet Surveys Conducted with Probability and Non-Probability Samples’, Public Opinion Quarterly 75(4), 2011, pp. 709-747.

11. C. Kennedy, A. Mercer, S. Keeter, N. Hatley, K. McGeeney, and A. Gimenez, ‘Evaluating Online Non-probability Surveys’, Pew Research Center, 2 May 2016.

12. C. Zukin, op. cit., 2015, fn. 2.

13. While US citizens cast ballots for individual candidates, they are in fact voting for electors, who then typically vote for the Presidential candidate who received the majority of the votes in their home state. The number of electors in each state is dependent upon the state’s population, resulting variation across states in the value of a vote towards the election outcome.

14. M.W. Traugott, op. cit., 2005, fn. 1.

15. T. Rogers and M. Aida, ‘Vote Self-Prediction Hardly Predicts Who Will Vote, and Is (Misleadingly) Unbiased’, American Politics Research 42(3), 2014, pp. 503-528.

16. M. Blumenthal, op. cit., 2016, fn. 3.

17. Rarely do partisans vote for the other party’s candidate in larger numbers. Even with the pivotal Reagan Democrats, still 75% of Democrats voted for the Democratic candidate.

18. D.S. Hillygus and T.G. Shields, The Persuadable Voter: Wedge Issues in Presidential Campaigns. Princeton University Press, 2009.

19. G. Langer, ‘Work, Widgets and Perfect Polls’, ABC News, 7 December 2008.