November 04, 2008

Bad News: Exit Polls Show Obama Way Ahead
Good News: Same Polls Showed Kerry Way Ahead 4 Years Ago

Early polls show Obama up in key states.

Every four years they claim they are going to improve exit polling data, and every four years they get it wrong.

I have a bit of familiarity with the system and how it works and, frankly, the "fixes" proposed by the number crunchers don't address the root problem. Professional pollsters tend to think that sampling errors stems from polling places. These polling places are not randomly chosen, violating the first principle of scientific sampling.

However, there may be good reason why randomness should be overlooked in exit polls. So, given that polling places are not surveyed randomly, then it is up to professionals to decide which polling stations are "representative" of the larger universe of polling places. Pollsters tend to think these problems can be boiled down to the representative nature of the polling places.

But that doesn't seem to be the problem to me. The problem comes from who is doing the polling: overwhelmingly college students.

I know this because every four years I get a call from the pollsters asking for names of bright students who would be interested in doing exit polling. In fact, I was asked to do exit polling on behalf of the VNS when I was in college. This year was no exception-- the call came about a month ago.

The sampling bias, then, cannot be fixed by changing where pollsters go to conduct surveys. College students -- even well trained ones -- will be more prone to ask certain individuals to reveal who they voted for over other individuals. Since college students are overwhelmingly liberal, they will tend to ask people who they are more comfortable asking --- liberals.

And the preemptive answer to the inevitable question: what do liberals look like? Is that you only need small biases repeated over and over again to get large sampling errors. In other words, if only one in 10 people questioned is asked to answer the poll because he's a a smelly, dirty, hippy in lieu of a cowboy with a handlebar mustache then the sampling error is huge!

I'm not saying this is intentional, or even that they are aware they are doing it, but that is the nature of bias: you cannot escape it because you don't realize you're doing it.

And there's no way of cross checking sampling here because there is no baseline by which to check! In other words, you cannot know how many dirty smelly hippies are representative of the dirty smelly hippies who voted because the only data you have profiling voters is the sample itself.

If pollsters want to fix the system, then they're going to have to hire people other than college students and the random unemployed person found at the local unemployment office.

So, don't put too much stock in the exit polls. They suck.

UPDATE: Case in point. Trust me, none of these early data should be believed. None. At some point the pollsters will have to "fix" the data bias by introducing a deflator for oversampling of Obama supporters.

By Rusty Shackleford, Ph.D. at 07:04 PM | Comments |