Strangers on a phone
So nearly one in five Americans thinks President Obama is a Muslim.
Predictably, hands began to wring. There are people who make their living wringing their hands about the State of the Nation, and sure enough, they showed up and played their part, showed their concern, and, uh, then the next thing on NPR came on. For others, it was a chance to squawk about the ignorance of the right, which led, just as predictably, to the pointing of fingers-sometimes expressed as the wringing of hands-at Fox News, talk radio, Glenn Beck, and Lex Luthor.
It didn't surprise me that the one note of skepticism was sounded by a pollster.
I've worked in polling for most of my professional life. Here's a secret: poll results tell you what people told a pollster-specifically, a call center worker reading a script-not necessarily what they believe.
Figuring out what people really believe is hard. That involves getting inside their heads and hearts, and man, if you think that's a matter of making a phone call, you have an awfully narrow view of human nature.
To people who follow politics closely, this is all a Very Serious Business. Sure, you're reading the political blogs, you're reading FiveThirtyEight-a cry for help if ever there was one-and you can name the people running in the Republican primary in some state you've never even visited. You're following Your Guys like normal people follow their favorite sports team. You'd be thrilled if some polling firm called you up, even in the middle of dinner or Glee, to ask your opinions about the health care bill, President Obama, Nancy Pelosi, and Sarah Palin. You'd answer every question thoughtfully, carefully considering whether to answer 4) agree somewhat or 5) agree strongly.
So you can't imagine that people would just say whatever, and might answer a question to express not an actual, factual belief, but a general attitude, or a broad contempt for The Other Side, whoever The Other Side is for them.
So 18% of people tell the nice call center worker that they think President Obama is a Muslim. Do they really believe that? Surely some of them do; someone is maintaining those web sites, right? But some of them are just snarling because they don't like his attitude toward Israel, or his foreign policy generally. Some are angry at his economic policy. Or they don't like the health care bill. Or they they don't like the cut of his jib. Or they just don't give a good God damn about whether the poll turns out to be a perfect little sampled mosaic and they say whatever comes to mind because screwing around with some dweeb's pie chart is more fun than just hanging up.
To the man with a hammer, everything looks like a nail. To the man with a call center and a stack of polling clients, everything looks like a poll waiting to be written. It's time for everyone to take a deep breath and think about what polling can do and what it can't. Pollsters have gotten pretty good at figuring out who people are planning to vote for, within certain parameters, like when we're close enough to the election that non-junkies are paying attention to the race. (Seriously, most people don't spend all year thinking about who ought to be the State Treasurer. They've got bills to pay, kids to pick up, a doctor who wants to run Further Tests on that thing that's Probably Nothing, and a mom who thinks they should have married someone else, and a house that's worth $100k less than the mortgage.)
We're pretty decent at measuring specific voting intent, and we're good at measuring attitudes-who and what voters like and dislike-but beliefs? That's hard. Beliefs are probabilistic by nature, and people spend years with shrinks, counselors, life coaches (::shudder::) and friends working out what they believe. The tools that tell us with 98% certainty and a 3.5% margin of error who people in Oregon plan to vote for in the gubernatorial race may not be so useful at sussing out beliefs. When it comes to voting, we can test our assumptions, compare our polls to the election results, and refine our models accordingly. But how do we test our polls about what people believe? To what do we compare our results? We refine our model based on, uh, what again?
When a good pollster tells you Schmuckface is leading Picklefeather by nine points, that's probably about right. Yeah, sure, there are "Dewey Defeats Truman" moments (I'm looking at you, Republicans of Alaska) but those are rarer and rarer. We have some specific tools that are good for some specific things. But just as you wouldn't go to a gynecologist looking for advice about that weird noise your car makes intermittently when you hit the brakes, don't look to pollsters for deeper insights into human nature than those we've proven we can unearth. We aren't psychics. And I believe, with 99.99999% confidence, that there aren't any psychics at all.
Predictably, hands began to wring. There are people who make their living wringing their hands about the State of the Nation, and sure enough, they showed up and played their part, showed their concern, and, uh, then the next thing on NPR came on. For others, it was a chance to squawk about the ignorance of the right, which led, just as predictably, to the pointing of fingers-sometimes expressed as the wringing of hands-at Fox News, talk radio, Glenn Beck, and Lex Luthor.
It didn't surprise me that the one note of skepticism was sounded by a pollster.
I've worked in polling for most of my professional life. Here's a secret: poll results tell you what people told a pollster-specifically, a call center worker reading a script-not necessarily what they believe.
Figuring out what people really believe is hard. That involves getting inside their heads and hearts, and man, if you think that's a matter of making a phone call, you have an awfully narrow view of human nature.
To people who follow politics closely, this is all a Very Serious Business. Sure, you're reading the political blogs, you're reading FiveThirtyEight-a cry for help if ever there was one-and you can name the people running in the Republican primary in some state you've never even visited. You're following Your Guys like normal people follow their favorite sports team. You'd be thrilled if some polling firm called you up, even in the middle of dinner or Glee, to ask your opinions about the health care bill, President Obama, Nancy Pelosi, and Sarah Palin. You'd answer every question thoughtfully, carefully considering whether to answer 4) agree somewhat or 5) agree strongly.
So you can't imagine that people would just say whatever, and might answer a question to express not an actual, factual belief, but a general attitude, or a broad contempt for The Other Side, whoever The Other Side is for them.
So 18% of people tell the nice call center worker that they think President Obama is a Muslim. Do they really believe that? Surely some of them do; someone is maintaining those web sites, right? But some of them are just snarling because they don't like his attitude toward Israel, or his foreign policy generally. Some are angry at his economic policy. Or they don't like the health care bill. Or they they don't like the cut of his jib. Or they just don't give a good God damn about whether the poll turns out to be a perfect little sampled mosaic and they say whatever comes to mind because screwing around with some dweeb's pie chart is more fun than just hanging up.
To the man with a hammer, everything looks like a nail. To the man with a call center and a stack of polling clients, everything looks like a poll waiting to be written. It's time for everyone to take a deep breath and think about what polling can do and what it can't. Pollsters have gotten pretty good at figuring out who people are planning to vote for, within certain parameters, like when we're close enough to the election that non-junkies are paying attention to the race. (Seriously, most people don't spend all year thinking about who ought to be the State Treasurer. They've got bills to pay, kids to pick up, a doctor who wants to run Further Tests on that thing that's Probably Nothing, and a mom who thinks they should have married someone else, and a house that's worth $100k less than the mortgage.)
We're pretty decent at measuring specific voting intent, and we're good at measuring attitudes-who and what voters like and dislike-but beliefs? That's hard. Beliefs are probabilistic by nature, and people spend years with shrinks, counselors, life coaches (::shudder::) and friends working out what they believe. The tools that tell us with 98% certainty and a 3.5% margin of error who people in Oregon plan to vote for in the gubernatorial race may not be so useful at sussing out beliefs. When it comes to voting, we can test our assumptions, compare our polls to the election results, and refine our models accordingly. But how do we test our polls about what people believe? To what do we compare our results? We refine our model based on, uh, what again?
When a good pollster tells you Schmuckface is leading Picklefeather by nine points, that's probably about right. Yeah, sure, there are "Dewey Defeats Truman" moments (I'm looking at you, Republicans of Alaska) but those are rarer and rarer. We have some specific tools that are good for some specific things. But just as you wouldn't go to a gynecologist looking for advice about that weird noise your car makes intermittently when you hit the brakes, don't look to pollsters for deeper insights into human nature than those we've proven we can unearth. We aren't psychics. And I believe, with 99.99999% confidence, that there aren't any psychics at all.