How surveys can be manipulated, part n
Every year when I start my statistics unit, I give my students a survey for purposes of collecting real-life data about them that we can analyze in the course of the class. However, I also secretly make multiple versions of the surveys, some of which have questions designed to bias responses in different ways. For example, this year half of my students received the following question:
Do you think that the government should spend tax dollars to support Amtrak even though the company is not commercially viable and is far inferior to other countries' rail systems? [] Yes, [] No
While the other half received this question:
Do you think that the government should continue to support Amtrak to help link communities together and provide cheaper, more environmentally sound alternatives to air travel? [] Yes, [] No
The answers were predictable; to the first question, I had 5 "yes" responses and 6 "no" responses, while to the second I had 9 "yes" answers and 0 "no" responses! Very gratifying to manipulate my students so. :)
(It would appear that my students would support Amtrak if asked an unbiased question, which makes sense, since they're probably relatively liberal on average and because most of them live in Chicago and have to commute to see their friends and family.)
I had another question, similarly biased, about U of I busses. But this year, I decided to try something new, something which surely would not work. Half of my students were asked:
Which do you prefer: [] Coke, [] Pepsi.
The other half were asked:
Which do you prefer: [] Pepsi, [] Coke.
Surely, I thought... surely this could not have a big effect on responses. But I put it in anyway, because it amused me to see the results.
To the first question: 7 Coke, 3 Pepsi.
To the second question: 2 Coke, 6 Pepsi.
!!!
They are putty in my hands! To be manipulated as I see fit! Bwahahahahaha...(*)
(In full disclosure, there were two people who gave no answer, one who checked both, and one who made a new box for himself and indicated "Dr. Pepper.")
Facetiously, I asked my students, "So, which order were presidential ballots in?" This is actually a serious concern, although it won't bias results to the same degree (I hope!). Indeed, these results were so easily skewed because students didn't have strong opinions; most presidential voters do. But in a close election, it might matter! (In fact, I'm reasonably sure there are studies on this, though I haven't searched for them myself; can anyone point me in the right direction?)
That said, there are states that vary the order of the candidates randomly from ballot-to-ballot, to try to account for this issue. (Just as good pollsters do the same in polls.)
(*) To any students reading this: just kidding. I love you guys. :)
Do you think that the government should spend tax dollars to support Amtrak even though the company is not commercially viable and is far inferior to other countries' rail systems? [] Yes, [] No
While the other half received this question:
Do you think that the government should continue to support Amtrak to help link communities together and provide cheaper, more environmentally sound alternatives to air travel? [] Yes, [] No
The answers were predictable; to the first question, I had 5 "yes" responses and 6 "no" responses, while to the second I had 9 "yes" answers and 0 "no" responses! Very gratifying to manipulate my students so. :)
(It would appear that my students would support Amtrak if asked an unbiased question, which makes sense, since they're probably relatively liberal on average and because most of them live in Chicago and have to commute to see their friends and family.)
I had another question, similarly biased, about U of I busses. But this year, I decided to try something new, something which surely would not work. Half of my students were asked:
Which do you prefer: [] Coke, [] Pepsi.
The other half were asked:
Which do you prefer: [] Pepsi, [] Coke.
Surely, I thought... surely this could not have a big effect on responses. But I put it in anyway, because it amused me to see the results.
To the first question: 7 Coke, 3 Pepsi.
To the second question: 2 Coke, 6 Pepsi.
!!!
They are putty in my hands! To be manipulated as I see fit! Bwahahahahaha...(*)
(In full disclosure, there were two people who gave no answer, one who checked both, and one who made a new box for himself and indicated "Dr. Pepper.")
Facetiously, I asked my students, "So, which order were presidential ballots in?" This is actually a serious concern, although it won't bias results to the same degree (I hope!). Indeed, these results were so easily skewed because students didn't have strong opinions; most presidential voters do. But in a close election, it might matter! (In fact, I'm reasonably sure there are studies on this, though I haven't searched for them myself; can anyone point me in the right direction?)
That said, there are states that vary the order of the candidates randomly from ballot-to-ballot, to try to account for this issue. (Just as good pollsters do the same in polls.)
(*) To any students reading this: just kidding. I love you guys. :)