Good Questions / Bad Questions
What kinds of questions will be asked by AmericaThinks? How do we ask good non-biased non-partisan polling questions? It can be harder than it appears.
Good Questions
Good polling questions are part science and part art. George Gallup writing in “The Pulse of Democracy” makes these suggestions for good polling questions.
Questions should be brief and to the point as possible.
Words and phrases should be simple and in common use among all groups in the community.
Questions should not include words with a strong emotional content.
Questions must avoid all possible bias or suggestions in favor of or against a particular point of view.
Questions should include all the important alternatives which may emerge on a given issue.
Questions with a choice of alternatives should list options as early in the question as possible.
Questions should be written down, especially complex questions.
AmericaThinks will strive to ask unbiased questions that are fair and balanced. Practically, there is no such thing as a perfect poll question and people will disagree on the amount of bias contained in a question. However, questions will be reviewed by our Advisory Council before being published in an attempt to hear objections from diverse viewpoints and make appropriate changes to questions before the question is published. If we published a question that needs critical changes based on feedback from a broader audience, we will retract the results and re-ask the question in a more neutral way.
AmericaThinks will be conducting election and issues polling. Scholarship has suggested much about asking good questions and we will continue to follow best practices in creating polling questions. Types of questions will include questions about values, priorities, participation, and approval. One of the question strategies may include presenting a public quote in context and asking for participant opinions.
AmericaThinks will avoid election polling topics such as pre-election candidate preference, straw polls, “horse race” polls, and polls that try to predict the outcome of an election. We do intend to ask about election topics such as voting participation, campaign issues, party platform, job approval, and generic ballot questions.
AmericaThinks polling will also include issues polling on cultural and governance topics. Some topics will be perennial topics that arise on a monthly or yearly basis. Many topics will be driven by the issues of the day. Legislative topics will receive a particular focus so that constituencies may offer their opinions and evaluate job performance of their elected representative.
Bad Questions
Biased polling questions are those that lead the respondents towards a specific response. In the courtroom, it would be called leading the witness. Many times, biased polling questions are unintentionally created. Gallup relates that despite their best intentions, sometimes they catch biased questions even after the poll is in progress. AmericaThinks will work to avoid these common errors in polling questions.
Leading Questions: directly and obviously leading respondents towards a certain answer.
Loaded Questions: a stronger form of leading question that practically forces respondents to answer in a certain way.
Assumptive Questions: assuming respondent agreement to statement; assuming respondent understanding of a complex or jargon filled statement.
Compound Questions: asking about two or more topics in the same question.
Absolute Questions: restricting the respondent to two extreme responses, effectively cutting off other reasonable responses and forcing respondents to choose an unwanted answer.
Acquiescence Bias Questions: ignoring other possible responses, effectively cutting off other reasonable responses and forcing respondents to choose an unwanted answer.
Vague, Unclear, or Ambiguous Questions: avoiding clarity by using questions with a broad possible interpretation or using terms that are not commonly understood by the respondents.
Multiple Answer Questions: asking the respondent to select more than one answer.
Limited Responses Offered: giving the respondent an artificially limited set of allowable answers, in some cases not allowing a “neutral” or no opinion answer.
Timing Bias: conducting a poll in close proximity to an event in which the poll compares the event to other events in the past or future.
Order Bias: asking questions in an order such that the respondent is lead toward a certain answer.
Word Bias: using emotionally laden words or words that have special meaning to a subset of the polled population.
Ugly and Awful Questions
AmericaThinks will not engage in “ugly and awful” polling questions. These types of questions strongly push the respondent to answer in a certain way and are known as “push polls.” Push poll questions typically present some type of negative and then ask a question that demands a certain answer. These types of push polls are especially common in political communications.
For example, “Does the fact that he has had problems with the IRS make you more or less likely to support him?” Or a non-political question, “Is deep-dish pizza better than that thin-crust junk they eat in New York?”
Even worse are questions that imply or state a pejorative, an ad hominem attack, assume the worst, or express a negative hypothetical and then ask a question. For example, “If someone were to abandon their children, would you be more or less likely to vote for them?”
More to Come
Creating good polling questions is an ongoing topic for discussion. For now, please know that AmericaThinks will be guided by our Advisory Council, participant suggestions, and polling norms as we create fair and balanced polling questions.