Lies, Damn Lies, and Writing Poll Questions
Kari Chisholm
As everyone knows, how you write a poll question can often determine the outcome. Today, the Oregonian allows its editorial slant to bias its news coverage of the city's campaign financing plan.
Take a look at the question they used (not that you can find their "news graphics" online):
The City Council will vote today on taxpayer financing of political campaigns; do you support or oppose the proposal?
And sure enough, 56% came out opposed. And no surprise, since they were asking people about "taxpayer financing of political campaigns". (Actually, 56% seems like a shockingly low level of opposition.)
In an environment where citizens (oh, excuse me, "taxpayers") aren't getting much information about the proposal (other than the hammering on the editorial pages), and when our schools and public services are underfunded, is it any surprise that they're not so sure about financing "political campaigns"? After all, everybody hates politics.
Now, if we really wanted to swing the other way, we could ask a similarly biased question. Let's try this one:
The City Council will vote today to eliminate the corrupting influence of Big Money political contributions; do you support or oppose the proposal?
I'll bet you dollars-to-donuts that this question will get at least 60% support.
Now that that's out of the way, how about a truly fair question:
The City Council will vote today on a campaign financing reform proposal. The proposal will provide public financing for candidates that demonstrate substantial support from the public, and will cost 1.3 million dollars - about one-tenth of one-percent of the city budget. Supporters say it will reduce the influence of political contributors, reduce the cost of political campaigns, and allow candidates to spend time talking to voters instead of raising money. Opponents say it simply costs too much, and public financing for political campaigns shouldn't be a priority. Do you support or oppose the proposal? (rotate the supporters' and opponents' arguments.)
It's a classic "we-say/they-say" question. It provides better information to poll respondents, it places the question in context, it's more fair to both sides, and it simulates the arguments that would be used in an actual political campaign.
Come on, Oregonian... how about a little intellectual honesty?
 |
More Recent Posts | |
Albert Kaufman |
|
Guest Column |
|
Kari Chisholm |
|
Kari Chisholm |
Final pre-census estimate: Oregon's getting a sixth congressional seat |
Albert Kaufman |
Polluted by Money - How corporate cash corrupted one of the greenest states in America |
Guest Column |
|
Albert Kaufman |
Our Democrat Representatives in Action - What's on your wish list? |
Kari Chisholm |
|
Guest Column |
|
Kari Chisholm |
|
connect with blueoregon
May 18, '05
The City Council will vote today to eliminate the corrupting influence of Big Money political contributions; do you support or oppose the proposal?
Now that is what I would call a push poll. If we want clean government, we shouldn't get there with dirty misleading polls.
10:00 a.m.
May 18, '05
Andy, I hope you didn't somehow miss the fact that Kari wasn't saying that's what the question should have been -- he was giving an example of a leading question coming from the other side, to go along with the leading question the poll actually used.
10:01 a.m.
May 18, '05
Andy... yup, that's exactly my point. The Oregonian's poll today was just as biased as the silly one I wrote.
10:02 a.m.
May 18, '05
Yeah, I would have thought that Davis and Hibbits would have known the phrase "taxpayer financing" is bound to sink polls on the plan. They're a great firm, and to use the language they used is really strange to me.
Without a ballot title, it's hard to pick a fair description, but connecting it to taxpayers paying is a born loser (and arguably unfair).
It's sort of like how voters hate Measure 37 when it is decribed as "Taxpayers must pay landowners, or governments forgoe enforcement, when restrictions reduce property value" but love it when it says "Governments must pay landowners..."
May 18, '05
This is NOT surprizing from Hibbits. His biases have been obvious for some time.
May 18, '05
I guess one could argue the fairness of the questions used in the poll (I haven't seen them all, of course), but it's worth noting that the poll makes a type I statistical error and lies just outside of statistical validity, anyway. Still, from the conversations with people I've had over the last several months, the JTTF and VOE numbers roughly fit with what I've been hearing (50/50 for both), the approval rating a little less so (been hearing more 60/40). I'll be more interested to see what the polling looks like for all of those issues a few months down the road.
May 18, '05
I like Kari's poll question. But it is much too long. Which is why I think Hibbits gets away with his biased question. It's short and too the point. It's also fairly accurate, just a little misleading.
12:22 p.m.
May 18, '05
I agree that this was a poor question. The phrase "taxpayer financing" steps right into the middle of the rhetoric wars. No matter what the result of the poll is, you're measuring respondents' orientation toward taxes, not to the City Council's proposal.
This would be slightly better for a short question:
The City Council will vote today on public funding of political campaigns; do you support or oppose the proposal?
Better still would be a question with a preamble, which is a common practice in polling--and is what Kari offered in his final version. I disagree that the question is too long. If you're doing a poll and asking a few select questions, you'll get far better data out of this query than from the Oregonian's version. (And I'm also surprised that only 56% gave a negative response to that wording.)
May 18, '05
Here's pretty much what's going on: the public is so disengaged that it can sway 20-40% depending on how you phrase a question.
Why does it matter what that disengaged, uninformed public thinks?
Polls force them to take a side, when they really don't care either way, so they swing with the question. This question is known to swing 20-30 points on the taxpayer vs. publicly owned moniker. Frame the question, get the response. She who sets the agenda wins the debate.
And only 2 of 10 folks accept polling inquiries anyway, so...
That news organizations use polls to create news (or further their editorial bias) is a strange habit, but they should at least do so honestly.
May 18, '05
Here here Kari! Excellent post.
You are asolutely right. Hibbits's question was not very good.
It looks as if he's becoming an activist as much as an analyzer.
Maybe all the t.v. appearances are getting to his head.
2:22 p.m.
May 18, '05
I don't agree. The Hibbits question could have been longer and done a one side/other side format (although those are always prone to the criticism that you have not fairly represented the two sides).
You may not like "taxpayer financed" but why is "publicly funded" better? Isn't that just misleading? Isn't Portland City Government funded by taxpayer dollars?
Hibbits tried to write as simple and direct a question as possible.
And those of you accusing him of bias have the obligation to step up and say why he'd have an interest in producing an inaccurate result for a poll commissioned for the Oregonian. Why would he purposely undermine his own credibility? What horse can he possibly have in this race other than promoting the accuracy of his own poll results?
Think harder, BlueOregon, and don't be so quick to let your ideological lenses distort your vision.
2:35 p.m.
May 18, '05
I'm not calling out Hibbits for bias in this poll. My guess is that the O wrote the question, not D&H. Adam and Tim have plenty of intellectual honesty - after all, without that, they're hosed. But, when a client demands a question be a certain way, well, that's the way it goes.
(I know there was at least one other incident where it was suggested that Tim was going beyond his own polling work in spinning a story. I'm not claiming that here. I blame his client.)
As for whether my proposed question is too long - don't be silly. Many poll questions are MUCH longer. For example, the classic two bios Candidate A vs. Candidate B question.
Paul, you ask about "taxpayer financed" vs. "publicly financed". On the one hand, you're right, there's no actual distinction. But, as Evan pointed out above, there's lots of evidence from many disparate polls that people react very differently to "taxpayer financed", "publicly financed", and "government financed". The first is almost always negative - nobody wants to pay. The last is much more positive - everybody wants a freebie.
May 18, '05
Hey Kari Chisholm & The One True b!X,
You two are right and I was wrong. I'm sick as a dog today and I must have missed 'a similarly biased' while reading in the fog I was in this morning.
May 18, '05
"as Evan pointed out above, there's lots of evidence from many disparate polls that people react very differently to "taxpayer financed", "publicly financed", and "government financed". The first is almost always negative - nobody wants to pay. The last is much more positive - everybody wants a freebie."
But there are no freebies. So isn't it as Paul said -- more honest, ie, more straightforward, to say "taxpayer financed?" Why is it a good thing to deliberately shade this into nearly a trick question?
5:47 p.m.
May 18, '05
Kari--
Not to pick nits, but in your "truly fair" question, you refer to this as "campaign finance reform." Webster's defines "reform" as "to amend or improve by change of form or removal of faults or abuses" or "to put or change into an improved form or condition." It's hardly a "truly fair" way to ask whether or not someone wants to make something better.
6:58 p.m.
May 18, '05
Well...TO nit pick, YOUR version Keri says it costs $1.3 million. Actually, we don't KNOW the cost. Or, if we accept a fixed "cap" then the promised amount of available funds --including matching funds against big spenders-- is a fantasy.
That said, I know too many good people idealistically supporting this so I'll keep my skepticism in abeyance. But, good grief, arguing the citizens of Portland would --right now-- vote FOR this, however it was worded, is, well, delusional. Whether they SHOULD is a whole other issue.
May 19, '05
After seeing this debate go on in some form in a multitude of other states, I couldn't help but weigh in. (First, disclosure: I am the ED of Public Campaign, a national group that supports full public financing systems similar to the one that has passed in Portland.)
Two points: first, polling questions that only refer to the costs of a proposal without the benefits are inherently biased. For years Gallup has asked a standard question on public financing of elections that references both that it costs money AND a benefit - candidates agree to spending limits. That standard has polled at above 50% (and often much higher) since the early 70's. In our own polling on full public financing, we add in that candidates getting public financing would NOT be able to take private contributions. That polls even higher.
But here's a second point. It is hard to get meaningful numbers in ANY short question - and one can argue endlessly about how to perfectly word such a query. So, to get a better sense of where the public is at, we have also run a kitchen sink's worth of batteries of pro and con arguments about public financing in numerous polls over the years. The striking fact is that the supermajority numbers hold up AFTER people hear arguments from both sides.
So - more fodder for this conversation...
12:45 p.m.
May 20, '05
Nick,
Dead on. The really figure out the outlines of public opinion, you'd want to ask a variety of questions with subtly nuanced changes in wording. Asking the question substituting "publicly financed" for "taxpayer financed" (say, randomly splitting the sample) would help us understand how those two phrases affect support.
Providing costs as well as benefits will help us gauge the relative weight respondents place these considerations.
And so on. None of this says that Tim's poll was bad or biased or purposely misleading. He likely did the best he could given the cost and respondent minutes.
<h2>Take the poll result for what it is: a snapshot based on what you think the public knew and on what Tim asked.</h2>