Well, what a Liberal wave! Even the most optimistic projections didn't have the Liberals over 180. It wasn't a very good night for me (in terms of projections, not saying anything about the result itself) and I'll have to look into it. But it doesn't look good, there is no way around it.
When I wrote my final projections analysis, I said that while the poll average was showing a Liberal minority, they were essentially three possible sources of error or uncertainty. 1) Is the last minute momentum for the Liberals (seen in the very last Forum and Nanos) as well as the collapse of the NDP real? It turned out it was, especially in Quebec 2) Will the Conservative be underestimated? They were, slightly (one of the few successes for me last night!) and 3) What about Quebec? The race was so tight that pretty much anything could happen. And what happened is a collapse of the NDP.
For now, let's look at the polls and how close they were. Some did really well, others were quite off.
I calculated the average total absolute deviation. For each party, you take the absolute deviation as the official results minus the voting intentions in the polls, you sum it over the five parties and you average. I also calculated how many parties were within the margins of error for each poll. This is indicated by the color.
As we can see, the averages (adjusted or not) did better than most of the polls except the very last two. This is quite rare that the average doesn't do significantly better than the polls and this is really because of what appears to be a last minute collapse of the NDP with its voters going to the Liberals. By the way, my adjusted average had the Liberals slightly too low, but notice there are the same (or partially the same) Nanos poll twice here. If we count it once, the unadjusted was 37.1 and I was really close. My adjustments were really to boost the CPC a little bit and to decrease the Green and Bloc. All three were going in the right direction and made the adjusted average better.
I admit I'm usually skeptical of such last minute trends but it seemed to have been the case yesterday. Notice however that Nanos didn't have the NDP 10 points behind the Liberals in Quebec in its last poll. But I'll add the provincial comparisons later.
So bottom line, the very last polls got it right. Nanos got all five parties within the margins of error, although these were pretty wide given the small sample size. Still, very impressive. Forum did also very well and I hope it'll finally stop people from always thinking they are a bad firm. Forum also got most of the provincial numbers right, but I'll add to this later.
By the way, Mainstreet did well for the average total error but terrible for the number of parties within the margins of error. This is because of the insane sample size they had.
So, should we always only use last minute polls? I don't know. It's very risky to base the analysis entirely on 1 or 2 polls with small sample sizes. But it would have worked a lot better yesterday. Although Quebec, one of the biggest surprises of the night, would still not have been right as even last minute polls had the NDP much closer to the Liberals than what happened.
So polls did pretty well yesterday, at least at the national level. However, it shows that even small deviations (like for the NDP) can have dramatic impacts on the seat projections. You get one province wrong (like Quebec) and you are off by 20 seats.
I'll add later this week the analysis by province.
