The 2018 Ontario provincial election is coming to an end this Thursday. We are starting to get the final polls from many firms and it seems we are observing another case of methods disagreeing with each other. Namely, phone polls IVR, automatic phone calls) do not show the same thing as online ones.
Before looking into this though, here are, as usual, the latest projections. I have added the polls published yesterday into the mix. As you can see, for the first time in a long time, the Conservatives are ahead in the popular vote.
Trend just isn't good for the NDP. On top of that, the trend by region is also negative (NDP decreasing in the Southwest for instance). Riding by riding projections are available at the end of this article.
Online versus IVR polls
The latest Abacus, research Co. and H+K Strategies polls, all done online, show the NDP either slightly ahead or tied with the PC. On the other hand, the recent Ekos, Mainstreet (tracker) and Forum, all IVR, have the Tories between 3 and 5 points. Pollara, using a mix of online and live callers, is more similar to the online polls. Let's leave it aside for now but we'll get back to it.
The graph below shows you the polls during the campaign. For the Mainstreet tracker, I only included the data once every 3 days (since the tracker is a 3-days rolling average). I separated the polls by method. It's a messy graph but I don't believe I can do better. Also, I had to put one of the NDP in green because orange and yellow are too similar. Click on it for a bigger size.
Couple of observations:
1) Online and IVR polls have had a systematic difference during the entire campaign. IVR having the PC higher and the NDP and OLP lower than online polls. The gap has remained constant for the PC and NDP.
2) Both methods give us the same trend and dynamics. At least!
3) Because of 1), we see that online poll had the NDP and PC tied around May 19th while IVR poll only reached the same level almost a week later (and at that point, online polls naturally had the NDP ahead). The Mainstreet tracker was especially slow to pick up.
4) Right now, we can see that the online polls have a tied race while IVR polls have the PC ahead. But again, same late campaign dynamic, just different levels.
5) At least there is convergence for the OLP
6) The IVR graph is heavily influenced by Mainstreet which published way more polls. So keep that in mind.
6) The IVR graph is heavily influenced by Mainstreet which published way more polls. So keep that in mind.
So, who is right and who is wrong? In order to answer the question, I went back to my data set of polls I used to estimate the average accuracy. I compared IVR versus online polls. The answer? The average absolute deviation of both methods (with respect to the actual outcome) is almost identical. So IVR and online polls have performed similarly over the last few years. IVR polls did better in 2015 but this is mostly because they were the last ones to poll and there was a late campaign shift in favour of the Liberals.
So I see no reason to believe one more than the other. If we limit ourselves to the last Ontario election (using the numbers among all voters and not the crappy "likely voters" pollsters were trying at the time and that proved very wrong), the best two polls were online (Angus-Reid and Abacus).
Maybe we have other polls to help us determine which one is right? The latest Pollara was a mix of online+live callers (most of the sample being live callers) and the results are in line with the online polls. Live calls polls have the best accuracy in recent years, although a lot of it is due to Nanos being right on the spot during the federal election. Still, between 2014 (online better in Ontario) and Pollara, that's two arguments in favour of online polls.
We also got the latest Ipsos poll that was 2/3 online and 1/3 by phone (live caller). This poll was more in line with the IVR ones! Thus not providing any tie breaker (actually making things more complicated).
Finally, Innovative actually did two polls, one online and one by phone (live callers) with very similar results. This would confirm the online+live call group.
So at the end of the day, I'll simply do an average. But I'll be careful in case the latest polls (like published on Wednesday) are all of one type. Aggregators that tend to give a much higher weight to the very last poll may introduce a bias whereas, instead of measuring a late campaign swing, they are merely capturing the effect of the method.
One thing to keep in mind is that it might not matter as much this time around. Because not matter if you use online or IVR polls, you get a Tory majority as most likely outcome. Of course online polls would give more chances to the NDP but the PC is favourite in both cases.
Extra: riding by riding projections.
Ontario Proj 4.06.2018 Detailed by bryanbreguet on Scribd
