The Polls Got it (Mostly) Right
Jeff Alworth
You could make the argument that pollsters had more on the line than Barack Obama. In what was easily the most-polled election in US history, pollsters had come to a consensus by the first of the week, which meant the exercise of polling--battered a bit in the primaries--was under close scrutiny. But in a vindication of the discipline, the pollsters more or less nailed it.
According to the Pollster.com aggregate, polls were predicting a 7.6% victory for Obama nationally, just a bit off the 6.4% he actually got. They were mostly accurate in predicting state outcomes as well. In 14 swing states, they predicted correctly on 12, inverting Indiana and Missouri. The aggregate for the states polls were also pretty close; in 11 of the 14, they were within three percentage points. They were also pretty close on the Merkley-Smith race, predicting a Merkley win by a slightly larger margin than he actually got (2.1%). Have a look:
Pollster
Trend Actual Var
New Hampshire 11.9% 9.6% -2.3%
New Mexico 8.9% 14.7% 5.8%
Colorado 7.6% 8.6% 1.0%
Pennsylvania 7.2% 10.5% 3.3%
Nevada 7.1% 12.7% 5.6%
Virginia 5.6% 5.6% 0.0%
Ohio 3.1% 4.0% 0.9%
Florida 1.7% 2.5% 0.8%
Missouri 1.1% -0.2% -1.3%
North Carolina 0.4% 0.3% -0.1%
Indiana -1.2% 1.0% 2.2%
Montana -2.2% -2.6% -0.4%
Georgia -2.9% -5.3% -2.4%
Arizona -4.9% -8.5% -3.6%
National Trend 7.6% 6.4% -1.2%
Oregon Senate 5.4% 3.3% -2.1%
There were a couple of questions leading up to the election: will either the so-called "Bradley effect" or the growing use of cell-phone-only users throw off the results ? Short answers, "no" and "not so far."
The "Bradley effect" describes a type of polling bias that overestimates the strength of black candidates. Leading up to the election, Harvard released a study concluding that the effect was real, but tapered off in the mid-90s and is no longer a factor. But this was the first national election with a black candidate, so a lingering question remained (especially following the New Hampshire primary). The polls were pretty clear, though--there's absolutely no evidence of such a phenomenon. In the 14 swing states, polls underestimated Obama's support in eight--the random variation you'd expect in any election.
For the answer to the cell-phone question, I turn to Pollster's Brian Schafner:
The trend based on surveys including the CPO [cell-phone-only] population did slightly better at estimating Obama's vote but worse at gauging McCain's support. Overall, the CPO trend was slightly further off the mark than the landline trend.
Of course, there are any number of other factors at play with these different surveys (such as different likely voter screens, weighting, etc), so we can't draw any definitive conclusions from this analysis. But there is no obvious pattern from these initial results that indicate that including CPO respondents helped improve polling accuracy.
Two final notes: I have long described Pew as the gold standard for polling research. Two days before the election, they put out their final poll predicting Obama would win by six points--exactly his margin of victory. It's the second presidential election in a row that they've nailed the margin exactly.
But perhaps polling is going to become a secondary science, thanks to the arrival of Nate Silver on the scene. Bringing a brand of analysis honed in the study of baseball, he configured a predictive model for projecting outcomes that included a rich tableau of inputs, only one of which was polling. (To predict how a state might vote, he weighted it based on factors that included demographics, donor support, and idiosyncratic elements like "Walmart/Starbucks ratio.") Based on his modeling, he predicted: Obama would 348 Electoral votes (he missed Indiana but got North Carolina), win the popular vote 52.3% to 46.2%, and that the Senate would have 57 Dems (which now looks to be the number). Some polls were as accurate; none were more so. It's not particularly bold to predict that in coming elections, pundits will no longer rely solely on polling.
 |
More Recent Posts | |
Albert Kaufman |
|
Guest Column |
|
Kari Chisholm |
|
Kari Chisholm |
Final pre-census estimate: Oregon's getting a sixth congressional seat |
Albert Kaufman |
Polluted by Money - How corporate cash corrupted one of the greenest states in America |
Guest Column |
|
Albert Kaufman |
Our Democrat Representatives in Action - What's on your wish list? |
Kari Chisholm |
|
Guest Column |
|
Kari Chisholm |
|
connect with blueoregon
Nov 7, '08
It turns out Nate Silver is the genius statistician at projecting election results. Incredibly close to the actual result. Not long ago he was the anonymous "Poblano" who posted on Daily Kos. I heard so many "chicken littlers" who were screaming about the Bradley effect, and about about how it was all going to get stolen (Rachel Maddow was pathetic in her new role as "Debbie Downer.") But it turns out things developed about as they were projected to. It's remarkable in an election about Hope how many Dems were scared to death of actually hoping. I think there's a lot of unresolved political PTSD out there. Aye, Jeff!
12:36 p.m.
Nov 7, '08
Ayup. Silver's a danged National Treasure as far as I'm concerned. Kept me calm and I can't tell you how many times I had to browbeat fellow travellers who were wallclimbing tahat they needed a dose of 538.com.
That said, what if there were two "effects" in operation simultaneously?
Bradley and an "Aw shit, what the Hell" reverse Bradley.
There were lots of anecdotes and images of Confederate battle flags and pickups with rifle racks in use, White Guys in Cammo, and so on; with Obama stickers/signs in the mix.
12:37 p.m.
Nov 7, '08
Electoral-vote.com did a spectacular job, using statewide polls rather than national polls. He got only one state, Indiana, wrong. On this page, he has maps charting his final prediction and the final result. The 11 electoral votes of Indiana are the only error.
12:54 p.m.
Nov 7, '08
Alaska is a big mystery, though. Of course, Silver has his thoughts, some of which may be that Obama was called before Alaska's polls closed, perhaps it's caught in the ballots that were pulled as questionable, and perhaps Stevens will actually lose in the end.
The other big mystery is the Measure 61 call. The campaign against it said it was unbeatable, and we had to vote for Measure 57, but it went down (51 - 49%, by about 37,000 votes right now).
1:10 p.m.
Nov 7, '08
Maybe the Leg hired Riley for their potential M61 poll, which is why it was apparently off so badly...
2:11 p.m.
Nov 7, '08
I think the key was that people felt they didn't have to vote for M61 because they could vote for M57. They wanted some change in how things were handled, but maybe not as much as M61 had.
Without M57 on the ballot, M61 would have passed. I have no doubt in my mind about that.
With both on the ballot, one was sure to pass because people wanted to see some change in that area. It just took a heck of a lot of work on everyone's part to convince people why voting against M61 and for M57 was the better solution.
2:59 p.m.
Nov 7, '08
That's definitely a possibility, Jenni.
But let's be clear: the campaign said "Both Measure 57 and 61 will pass without a doubt. It's our job to make sure Measure 57 gets more votes and therefore trumps Measure 61. You, as a progressive, HAVE to vote for it whether or not you agree with it if you want to defeat 61."
And that's simply turning out not to be true (surprisingly). There are those who supported Measure 57 as good policy (including the League of Women Voters and other groups and legislators I thoroughly respect), but as pure political hardball progressives didn't end up needing to vote for it. 20/20 hindsight is easy, of course. :)
3:57 p.m.
Nov 7, '08
Bill, the percentage of people who will claim to have known Nate in his Poblano days now probably numbers 500-600% of the actual number, but count me in them. He gained blogo-noteriety in the primaries which brought him to my attention. My guess is that the NC/Indiana primary cemented his rep.
Evan, I think the analysis that 61 would pass was based on polling that showed huge support for anti-crime measures. I'm with Jenni--it would have passed without 57, and handily. Hell, it almost did.
5:38 p.m.
Nov 7, '08
Evan,
I have never seen anyone post or point to poll data on M61 dated later than January of February of 2008. It seems to me likely that such data exist but that their non-release was a strategic decision on the part of both main organized camps: pro-61 and yes-on-57-no-on-61.
For the latter, if it was known among people who would liked to have opposed both bills but felt they had to vote lesser evil yes on 57 that 61 was weaker than that early poll data indicated, more of them would have decided to vote no on both, and more would have said so in polls. Which would have made 57's passage less clear and I think led some other yes on 57 no on 61 people to vote yes on both, because they preferred M61 to nothing. That's speculation, but I bet it was the speculative possibility on which the pro-M57 people focused.
Essentially I think the data vacuum from the pro-57 side was about manipulating grudging/tactical M57 supporters to focus on worst fears.
From the Mannix side I think the speculative calculus might have looked different -- their worst case would be that data showing M57 stronger than M61 would lead to more yes on 57 votes as the best way to ensure there was one or the other from those who wanted one or the other.
A different way to put this is I think it reflected a reality of complex dynamic relationships among the four possibilities (YY, YN, NY, NN) that would be best parsed by game theory that I don't have, but in essence was produced by everyone focusing on avoiding worst cases from their point of view -- including an intentional information vacuum to keep others focused on worst cases.
Why didn't The Oregonian or some other media outlet commission and publish a poll? Not sure.
Nov 8, '08