How to Read a Poll
Jeff Alworth
The ever alert b!X yesterday alerted us to a new poll put out by Riley Research Associates. According to the poll's surprising findings (available here on .pdf):
- Bush and Kerry are tied (46% - 45%),
- Tom Potter is hammering Jim Francesconi in the mayoral race (49% - 25%),
- the ballot measure to ban gay marriage is wildly popular (61% for, 34% against), and
- incumbent Dems Wu and Hooley are enjoying comfortable leads (of 34% and 16% respectively).
Apparently Oregon is back in play for Bush again and the gay marriage ban is a done deal, right? Well, not so fast. This is one of those classic cases where the methodology--a section of research findings no one reads--turns out to be a lot more revealing than the survey's bogus results.
In order for a poll to mean anything, it has to be both valid and reliable. Reporters tend to look at the margin of error as a proxy for validity and reliability, but it has nothing to do with it--the MOE is a statistical calculation based solely on the sample size. What they should be looking at is who responded to the survey and how well they represent the population you're interested in.
In the Riley sample, we don't really know. They identify the sample as statewide and randomly-selected; they tell us 507 people participated and that trespondents were "very likely" to vote in November. Great, but who were they?
In politics, your numbers don't mean a thing if they're not representative. If I go to the Red and Black Cafe and collect responses from the first 1,000 people who come through the door, I'll have a low margin of error. But the numbers will be useless. In order to acheive statistical validity, I have to have sample a group that looks like the voters.
Zogby, which does an "interactive" poll that recently had Kerry leading by ten points, has the kind of language we want to see in a methodology section:
Reported frequencies and crosstabs are weighted using the appropriate demographic profile to provide a sample that best represents the targeted population from which the sample is drawn from. The proportions comprising the demographic profile are compiled from historical exit poll data, census data, and from Zogby International survey data.
Is the Riley poll accurate? We have no idea, and based on what they've written in their methodology, they don't either. (There are additional issues with their sample size, which you can read about here.) Polls don't only measure public opinion, they help shape it. Given that they're used so prominently by media, politicians, and voters, we should be careful about interpreting the findings. When a group like Riley offers up something as dubious as this, we should do what KGW should have done: round file it.
 |
More Recent Posts | |
Albert Kaufman |
|
Guest Column |
|
Kari Chisholm |
|
Kari Chisholm |
Final pre-census estimate: Oregon's getting a sixth congressional seat |
Albert Kaufman |
Polluted by Money - How corporate cash corrupted one of the greenest states in America |
Guest Column |
|
Albert Kaufman |
Our Democrat Representatives in Action - What's on your wish list? |
Kari Chisholm |
|
Guest Column |
|
Kari Chisholm |
|
connect with blueoregon
Sep 10, '04
yeah, polls are useless man...unless you want to find out how many voters watched the entire season of Survivor.
Sep 10, '04
Actually, the report you link to clearly says that they oversampled on republicans and undersampled on "independent" voters because they focused on "very likely" voters:
"While there are more registered Democrats in Oregon, a higher percentage of Republicans, and relatively speaking, a smaller percentage of independent voters qualified for the poll."
Since we are going to see record turnout his November here in Oregon, I am curious as to what registered voters would not qualify as "likely voters". In any case, this survey highlights the importance of GOTV once the ballots hit the mailboxes five weeks from now.
The poll also shows that independent voters overwhelmingly favor Kerry over Bush (58 to 30).
12:22 p.m.
Sep 10, '04
Nice work, Marcello. I clearly failed to read it carefully enough. Most of my general info should be more accurate than my analysis of Riley, however...
2:29 p.m.
Sep 10, '04
Great piece, Jeff. The question of "registered voters" versus "likely voters" versus whatever-the-heck "very likely" voters is, is a critical question.
"Likely" voters are a universe of people who are 4/4 voters, or even 7/8 and 8/8 voters. By definition, that means three things:
One, that excludes nearly all voters 18-21. They've only had the chance to vote in two elections.
Two, it does a good job of excluding lots of additional younger voters. Younger voters move around a lot, and the databases aren't sophisticated enough (and pollsters don't bother) to track voters from one address to another.
Three, because of the above two - and because home-owners, married people, and suburbanites move around less and are more likely to vote on a generic basis, the pool of "likely voters" tilts Republican.
In an election with heavy voter interest (and, likely, turnout) from traditionally "unlikely" and young voters, well, anyone relying on traditional likely-voter modeling is going to be surprised.
Don't forget, back in '92, George H.W. Bush was leading Bill Clinton among "likely" voters right up to election day.
For more on sample size, check out my comment over at B!X on KGW's mayoral poll.
Sep 11, '04
Totally bogus. No fly-specking analysis needed. Riley did the poll "himself" according to Jeff Mapes report.
Which means Riley didn't have a customer he could sell polling to.
Which means both parties have crossed Oregon off the battle list. (Save their money. Save our comity.)
Which means Riley is s.o.l. until he can terrify you. All the campaign parasites -- ad-selling TV stations, pollsters, consultants, p.r. wannabes, mercenary agitators, etc. -- cheat their data to lie every race is close. That's the only way they get customers. They don't care what happens to the actual people in the community, and (funny thing) it comes back to them: people don't care for cheats.
Truth is all the lawn signs and bumperstickers and energized new young voters for Kerry. The Gang Of Psychos (GOP) signs and ads are where the bigots and dinosaurs go, and there's not as many of them. Trust your eyes before Riley's lies and TV lies and terrorize.
I've been phone polled twice by Riley in the past, both times for rightwing scare topics. Actually, three times. The last time I hung up when they said they were Riley.
It's desperation when a pollster does the polling "himself." Self-representing lawyers' clients are fools, self-selling pollsters' customers are tools. Riley dusted himself.
<h1></h1>Sep 12, '04
http://zogby.com/news/ReadNews.dbm?ID=862
Kerry Holds Ten-Point Lead Over Bush in Oregon; Battleground State Not So Close, New Zogby International Survey Reveals
[full article deleted. copyright violation. -ed.]
<h1></h1>Riley could join Liars Larson in the unemployment line -- everyone avoids having their company name associated with either of them or other cells of fanatic perjorative rightists.
Sep 14, '04
People have posted a few things here that are inaccurate and misleading.
One, that excludes nearly all voters 18-21. They've only had the chance to vote in two elections.
Two, it does a good job of excluding lots of additional younger voters. Younger voters move around a lot, and the databases aren't sophisticated enough (and pollsters don't bother) to track voters from one address to another.
I don't know what Kari is reading. Riley does NOT say anywhere that likely voters are determined by the frequency of voting in past elections (probably, they are screened by one or a series of questions on interest in the outcome or self-reported likelihood of voting). On page 1, Riley explicitly states how frequently the likely voter sample reports turning out in previous elections (so, for example, 10% of the sample reports voting in 0 of 4 past elections).
Similarly, on the tables on pg. 4, you can see marginals x turnout in past elections (self-reported).
Zogby's "interactive" poll is a self-selected electronic sample -- so he has to use weights to make the results look even remotely like the population. That's the only reason Zogby has to report on how he weighted the sample. Zogby's methodology is *highly* controversial, to put it mildly.
What Riley has done is a standard RDD sample frame. I'm not sure what more information you'd want to evaluate the methodology. Riley reports the unweighted proportion Republican, Democrat, gender, age, and congressional districts. All he's missing is the number of call backs and the refusal rate (something virtually no political pollsters report on).
4:04 p.m.
Sep 14, '04
Prof, you're right. I didn't read Riley's info.
I was speaking generically about the "likely voter" method that some political pollsters use - limiting your pool to 4/4 or 7/8 voters in the voter file. I do understand that another method includes asking people a series of questions - and then develop an appropriate screen. That's the only way to do it with a random digit dial.
Sep 15, '04
There appear to be some lingering questions about the purpose of our recent (Riley Research) Oregon likely voter poll, including the use of the "Likely Voter" screener.
We conducted the poll on behalf of a number of clients who were interested in statewide voter responses on specific issues, then we threw in some additional questions for our own edification (i.e.: presidential, ballot measure, and CD questions).
We conduct these "omnibus" polls from time to time, and share the non-proprietary results with interested parties. The results can be viewed at: www.rileyresearch.com/riley_report.pdf (requires Adobe). This poll was not affiliated with nor sponsored by KGW.
As a previous writer noted, we included newly registered voters and those who have never voted before. As the methodology section of my report indicates, the "likely voter" screen was used to focus the poll on people who are most likely to actually send in their ballots come November.
Thank you for your interest and comments.
Sep 15, '04
Mike, Thanks for posting!