Friday, April 14, 2006

'Hey, kids, let's do a poll!'

"Hey, kids, let's put on a show!" Mickey Rooney would say to Judy Garland and others as part of the classic old film formula that was a box office bonanza.

It started with "Babes in Arms," about teens who save their vaudevillian families from ruin by putting on a show. Three more musicals with the same theme followed and made Garland a star.

"Amateurs rule" was the message.

These days, the theme seems to be, "Hey kids, let's do a poll!"

Yesterday it was St. Norbert College providing the fodder for political spin across the state, while offering up numbers that meant almost nothing.

Today it's UW-Milwaukee, with its political poll that uses an interesting methodology.

One problem is that the survey was stretched out to 15 days. If a poll is supposed to be a snapshot of opinion, it's best done in 3-4 days. The UWM poll -- and the St. Norbert poll, which stretched over 10 days -- are blurry photos at best.

But this is what should set off alarms:

In the survey of 471 Wisconsin residents who intended to vote, 44 percent indicated they would vote for Doyle, while 33 percent said they would choose Green, if the gubernatorial election were held today. But 21 percent had not made up their minds, and 107 respondents said they probably would not vote.
It is customary, when doing a poll to try to find out what's happening in an election campaign, to ask people who actually intend to vote.

If 107 people said they probably would not vote, they should have been excluded from the survey to begin with.

When almost 20% of your sample consists of non-voters, it will skew all of the results. The survey used likely voters in the governor's race matchup, which showed Doyle 11% ahead, and in asking about the amendment to ban gay marriage and civil unions, which was favored 53-37. But the other questions used the whole sample. For example, 22% said they don't know whether they like the job Doyle is doing as governor. Those are people who aren't paying attention. and probably are among the non-voters.

The problem is that the news media treat all polls much the same, although there is a wide range in their accuracy and credibility. And the campaigns add to the confusion, twisting them to their advantage.

The most accurate polls, of course, are the ones you never hear or read about, that are done by professional political pollsters for the campaigns themselves. They are used for strategic purposes, not publicity. To campaigns, the horse race numbers are less important than the issues and messages they test, to find out how to move voters in their direction. When you see a poll that's leaked by a campaign, pay attention to whether the campaign has released the entire poll or just one or two questions that make their candidate look good. To be able to judge its credibility, you need to see the context, methodology and questions they asked.

Next best are those done by professional polling firms for media outlets. You'll see many of those done for national media outlets, but few in Wisconsin. In those cases the numbers are mostly reliable. One variable is the analysis -- does it come from an objective polling firm, or from a a political reporter for the media outlet which sponsored the poll?

A friend in the business says: "National media polls are usually pretty good because the sponsors have the resources to pay to do them right. (Often they combine with other news outlets to share the cost like NY Times-CBS, ABC-WashPost, or NBC-Wall St. Journal.) But local media polls can be on varying quality because they don't have the resources to spend and it is a business where it costs more to do it right: like using trained/experienced interviewers, calling back to get representative samples instead of whomever is home, interviewing large enough samples. Unlike some other states, Wisconsin does not have a history of a reliable statewide media poll and that's why these university polls gain so much attention."

Other tips from a longtime pollster:

Be wary of people who say they are doing it without a client (like Strategic Vision, whom the GOP will cite all year). If they are doing it right, it is a pretty costly proposition to do it for publicity purposes. Either someone is sponsoring it in some indirect way, or they are cutting costs.

Be wary of polls like Zogby that take internet volunteers--the cheapest of all polls.

Xoff's bottom line: Don't believe anything you hear, and only half of what you see. And, as my grandfather used to say in bygone days, "Never bet against the Yankees, Notre Dame or the Democrats."

5 Comments:

At 1:11 AM, Blogger James Wigderson said...

Would your grandfather bet in an Indian casino that contributed to the governor?

 
At 11:53 AM, Blogger Brian Eisold said...

Hey, old white guys, let's write a hack piece.

Maybe wearing your glasses and thinking cap might have helped...and if there was still confusion, maybe it would have helped if you contacted the instructor who supervised the survey.

Looking at the paragraph that you quoted, the results were based upon the 471 residents who indicated they were going to vote. Adding that to the 107 of respondents who said they were not voting would mean that there were 578 total respondents to the survey.

Yes it would have been nice if wispolitics had the sense to include that number from the press release, but sloppy journalism isn't an excuse for poor investigation. Consulting the original press release or the survey supervisor would be much more professional than some hack piece that an amateur could have put together.

You say that the problem is the news media's interpretation of the polls. I say that the problem is it's the media AND pundits who are unwilling to do their homework that is the problem.

Saying that a group of juniors and seniors in an upper-level political research methods class are a "bunch of kids" really says a lot about your impressions of higher education in Wisconsin. The students conducting the survey are all well training in survey methodology, as well as writing, and executing such surveys. Of course, you wouldn't know that since you didn't do your homework.

You also wouldn't know about the quality of our lab, nor would you know that your implied accusations of cutting corners are wrong. For example, we did do call backs to get representative samples. We did train on the CATI program before using it, and we were monitored while conducting the survey to ensure that people were conducting the survey appropriately.

A sample of 578 people provides a realistic picture of the mood of voters. Any good statistician knows you don't need to interview thousands of people to get accurate information.

As one of the students involved in this class, I find your condescending attitude annoying, and your inability to write an accurate commentary almost comical. The only sad thing is that people listen to pundits like you.

You are right about one thing...."amateurs rule." A professional would never have published something like this without checking basic facts.

 
At 12:47 PM, Blogger Xoff said...

Actually, I knew the size of the total sample. 107 non-voters is 18.5% of 578. I rounded it off to 20%, well within your margin of error, in writing about it.


578 doesn't provide "a realistic picture of the mood of voters," as you claim, when almost 20% of them are NOT voters.

You've just proven my point.

PS: I'm a professional political consultant who's been reading polls for 20 years. But in this case I guess I was an amateur because no one was paying me.

 
At 5:16 PM, Blogger DEMKRT said...

Hey Xoff,
I'm a Doyle-backer and usually a fan of your blog, but I think this one may be a little off. Unfortunately, there seems to have been an ambiguity through the press-release. This poll did not ask those who indicated they did not intend to vote for whom they would vote. In fact, the CATI system that was used automatically skipped the question for those who responded that they wouldn't be voting in the fall. This means that the 107 responses or roughly 20% that you cited are NOT included in the survey results for that question.
If you read the questions presented through the survey, you can see that the questions are carefully crafted in order to prevent any type of bias that may occur. These interviewers were trained, and took all of the necessary precautions to ensure the soundness of this survey through the entire process. So, to agree with the earlier blog, these were not just "kids". They were upper level students who worked hard on this poll to make it as professional as possible.
Also, you cite that the poll was conducted over the period of two weeks, and that, you argue, makes the poll a blurry photo. A similar poll was conducted in the same manner in 2002 through the UWM Center for Urban Research and Initiatives was one of the state's most accurate polls in predicting the victory of Jim Doyle in that particular gubernatorial contest. So, whether or not this amount of time presents what you call a "blurry photo", it has been seen as an accurate snapshot, and who can say that this is not the case with this poll as well?
The real truth and danger here is that Doyle seems unable to capitalize on the genuinely positive feeling Wisconsinites seem to have toward the state economy, as well as many of the Democratic positions that people hold. People are overwhelmingly satisfied on the state level. People are solidly against concealed carry in this state. This should set Doyle up for a nice victory, but people don't seem to know where Doyle, or the Democrats as a whole, stand.
Please, for the love of God, stop picking at poll results that you don't care for, and concentrate on the concerns that are presented in them. This is what has gotten us Democrats in too much trouble. To avoid a Governor Green, please focus on helping the Dems fasion a positive message that Wisconsinites will automatically associate with this party. Hopefully, it can persuade all 22% undecideds into Doyle's camp.
Again, usually a big fan, and I hope my post doesn't fall on deaf ears. I just hope that people understand that just because a University of Wisconsin poll doesn't come out of a certain campus in Madison, doesn't mean that it's not a good poll.
Thanks, and GO DEMOCRATS!!!

 
At 5:58 PM, Blogger Xoff said...

At the risk of prolonging this discussion ad infinitum:

I understand, and always did, that the 107 non-voters were not included in the questions about who they would vote for. But what were they doing in the sample at all?

As to the UWM poll being evidence that two-week surveys are the wave of the future, I can only say that even a blind pig finds an acorn once in awhile.

I don't disrespect students, whether they are at UW-Madison, St. Norbert or Slippery Rock. i'm sure this exercise is a good learning experience.

But I would never rely on those polls if I were trying to get elected governor.

In truth, the most accurate polling about the last governor's race was the internal polling done by the Doyle campaign's professional pollster. But those were never made public.

My real gripe is that the media cannot distinguish between a reliable poll and an unreliable one, so treat them all as though they are equally valid. I'm sure you'll agree that's not true.

I didn't comment on the St. Norbert poll because I didn't like the results. I didn't like the methodology or the coverage. You did; we disagree.

But at least we're both Democrats. Let's shake on that and agree to disagree on the rest.

 

Post a Comment

<< Home