As the Brexit referendum on June 23 approaches, financial markets are becoming increasingly concerned about the outcome since it will lead to large movements in asset prices. For instance, one would expect that a vote in favour of remaining in the EU would be positive for UK asset prices, while a decision to leave would trigger huge uncertainty that would be generally negative for UK asset prices.
Considerable attention is therefore focused on surveys of voting intention that are conducted regularly. But what can one learn from them? Perhaps surprisingly, the answer is less than one might think.
Surveys must reflect accurately the views of the broader population. But views about the benefits of membership in the EU, and the likelihood of voting, vary across different groups of society. For instance, it has been argued that older voters are more negative about remaining in the EU, and more likely to vote, than younger citizens. If surveys fail to be representative of the wider electorate, they can be misleading.
Here we show that there are good reasons to be dubious about the value of these surveys, relying on a dataset collected by the Financial Times of 201 surveys conducted between 9 September 2010 and 12 April 2016. The information provided includes the fractions voting Remain, Leave, the fraction undecided and, importantly, the institution that conducted the survey.
The figure below shows that, in the full sample, surveys show that just over 40% of respondents support remaining, and just below 40% support leaving. These surveys thus typically predict that the outcome of the referendum will be for the UK to remain in the EU.
Indeed, in 115 of 201 surveys (or 57%), Remain is ahead of Leave. Since the results are rounded to the nearest percentage point, in another 15 cases (7%), the outcome is a tie.
The table below presents the results in greater detail for the full sample and for 45 surveys conducted in 2016.
|Full sample, 201 obs.
|2016, 45 obs.
These results show that the Remain side is generally ahead by a small amount. Furthermore, the fraction of undecided voters, which is almost 20%, is very large. But most importantly, there is a huge variation in results between surveys, much larger than one would expect on the basis of sample variation.1
The surveys are conducted by 15 institutions, which the FT lists as: Aschcroft Polls, BMG Research, ComRes, GQRR, Harris, ICM, Ipsos MORI, Opinium, ORB, Panelbase, Pew Research Center, Polulus, Survation, TNS, and You Gov.
If surveys were well designed, their outcomes would not depend on who conducted them. The enormous variation between surveys suggests that there might be systematic differences in the findings of the different organisations conducting the surveys. Regressing the survey results on a set of dummy variables that control for the organisation that conducted the survey, it is possible to assess how important the survey design is by looking at the R-squared. If the survey design didn’t matter, we would expect all dummy variables to be insignificant and the R-squared to be zero.
The results on data from 2016 are striking: 86% of the variation of the fraction of survey respondents that are in favour of remaining in the EU depends on which organisation conducted the survey. Similarly, 54% of the variation in the support for leaving the EU is due to which organisation conducted the survey. Turning to the result for Remain minus Leave – which will determine the outcome of the referendum – almost exactly half (51%) of the variation in the outcome is due to which organisation conducted the survey.
There are thus large differences in the findings between survey organisations. Indeed, surveys by Ipsos MORI on average lead to 13.8% higher score for the Remain side, while surveys conducted by Opinium lead to 5.2% lower scores.
It is not obvious why these differences arise. However, it has been noted that telephone polls (such as those by Ipsos Mori and Comres) led to systematically higher support for the Remain side than online surveys.2 Whether that means that they are more reliable remains unclear; we simply don’t know whether the surveys that suggest large support for Remain are more likely to be correct than those that suggest a much closer outcome.
A large number of surveys regarding the desirability of Brexit have been undertaken. These generally suggest that the outcome will be in favour of remaining in the EU. However, the organisation that conducted the survey seems to be as important as survey respondents’ voting intentions in accounting for the results. Moreover, there is a very large number of undecided voters who will, most likely, decide the outcome of the referendum.
 For instance, a back-of-the-envelope calculation suggests that the standard error of these fractions is about 1.3%.
 See “EU referendum poll tracker and odds” at www.telegraph.co.uk/news/2016/03/23/eu-referendum-poll-tracker-and-odds/ and “Brexit: Space oddity in the opinion polls,” Global Data Watch, J.P. Morgan, February 28, 2016.