Polling 101: How It Works

Mar 15

My goal for this 101 series is to present easily understood overviews of public opinion polling in order to show how AmericaThinks is re- imagining polling by combining the best of the past and the best of the future.

This series discusses:

Polling 101: How it works
Polling 101: Polling in Crisis
Polling 101: Literary Digest Revisited
AmericaThinks Polling 101: A Big Step Forward
AmericaThinks Polling 101: Asking about Issues, not predicting Elections
AmericaThinks Polling 101: Majority Opinions and Minority Rights

Polling 101: How it works.

Volumes have been written about the details of public opinion polling and how it is accomplished. Trying to explain public opinion polling is a bit tricky because public opinion polling is part science and part art.

This overview will focus on the big picture of traditional public opinion polling. If you would like to explore public opinion polling in more depth, I will provide some helpful resources at the end of this post.

Types of Public Opinion Polls

Public opinion polls can be divided into three broad categories: election polls, issue polls, and marketing polls. Election polls, also known as “horse race” or “pre-election” polls, attempt to predict the outcome of an election. Issue polls ask about the opinions and experiences of people to discover what people know about an issue, how it affects their lives, the importance of the issue, and what they think should be done. Marketing polls attempt to understand consumer sentiment, preferences, and motivations. This overview is not concerned with marketing polls, but instead focuses only on election and issue polls.

Polling Methods

Traditional polling methodologies assume that it is not feasible to ask the opinion of everyone in a given population. Therefore, the entire art and science of traditional opinion polling is focused on how to obtain accurate public opinion by polling only a tiny fraction of a population. Traditional polls are divided into two categories based on how to select the people to be polled. A poll that uses probability sampling is commonly known as a “scientific poll” or “Gallup poll.” A poll that uses non-probability sampling is commonly known as a “straw poll” or a “convenience poll.”

Scientific Polls

There is a wide variety of opinions and experiences about how to accomplish scientific polling. This disparity of opinions continues to generate vigorous and adversarial debate, occasionally even devolving into ad hominem attacks. These debates expose the obvious truth that public opinion polling is more art than science. Scientific polling methodologies do use quantifiable probabilistic sampling techniques, but additionally rely on subjective judgement and/or external data comparison at almost every step.

The practice of scientific polling follows a basic outline.

Create the questions to be asked.
Select a representative sample of people.
Ask people to respond to the questions.
Tally the responses.
Apply demographic corrections to the responses.
Summarize or interpret the results.

First, create the questions to be asked. Questions are simple, right? Not so much. Misunderstandings are very common in our communication with each other. So, it is important that poll questions are as simple as possible, clear, and understandable by people from a wide variety of backgrounds. A previous blog discusses how we can ask good questions and stay away from asking bad questions.

Second, select a representative sample of people. The most difficult aspect of polling has been and remains the question of how to select a representative group of people from a population. One of the early breakthrough discoveries by Gallup and others was that a carefully selected sample of about 1000 people could provide representative opinions for the entire country. In the early days of polling, convenience and quota sampling was used to match the selected sample to the demographics of the population. However, it was discovered that in many cases both conscious and unconscious bias could negatively affect the accuracy of the results. Eventually, the polling community settled on the idea of “randomly” selecting participants from the whole population. This concept is also known as probability sampling. Every member of the population has an equal chance of being selected to participate in the poll. As a practical matter, random sampling is difficult because it assumes that a master list exists that includes all the possible participants. In the past, telephone numbers, addresses, and other methods have been used as a master list. Today, for various reasons, these types of master lists are not nearly as effective. A growing trend is to recruit people to join a polling panel in which participants agree to respond to multiple polls. In some cases, these panels of people are randomly selected and invited to participate. In other cases, anyone can volunteer to participate in the panel, but people are randomly selected from the panel to answer certain polls.

Third, ask the selected sample of people to respond to the questions. In the early days of polling, interviewers would knock on doors and ask questions of the people who happened to be at home. This worked well because most people were delighted to give their opinions. Another method of asking questions has been the mailed survey. The mailed survey can be completed in relative anonymity at a convenient time at much less cost. In the 1980s, the use of telephone interviews become widespread. Telephone interviews provide all the benefits of a live interview without the cost of face-to-face interactions. Today, it has become very difficult to find people who are willing to complete polling interviews. Recently, text messages are used more and more to get limited opinions from a selected sample of people. People are sometimes contacted in person, by phone, text, email, or web and asked to fill out an online survey. All of these methods have their advantages and disadvantages.

Fourth, tally the responses. Tallying simply translates a response into a measured outcome. A well-designed poll will easily correlate responses with measured outcomes.

Fifth, apply demographic corrections to the responses, commonly known as “weighting.” Weighting is used to adjust the results of a poll so that particular demographic groups are represented in the poll at the same ratio as they are found in the general population. If the poll is an election poll, the results are further adjusted to reflect likely voters.

Sixth, summarize or interpret the results. Review the poll results in light of the bigger context of the population, expected results, and previous poll results. This tends to be a judgement-based evaluation to determine the overall veracity of the final results.

Convenience Polls

Straw polling, also known as convenience polling, is a simplistic method of polling in which the participants have unknown or non-representative demographics. Therefore, the poll is assumed to use an unrepresentative sample of participants. Examples of convenience polls include call-in polls, man-on-the-street polls, website polls, interest group polls, event polls, and door-to-door polls. Even scientific polls with response rates below 10% may be classified as a non-representative convenience poll. In each of these examples, the sampling is considered to be non-probabilistic. Ironically, convenience polls are much less subjective than scientific polls but are thought to be correspondingly less accurate because of the unknown characteristics of the participants and representativeness of the sample. In general, convenience polls are not considered to be reliable methods of polling.

The practice of straw polling follows a basic outline.

1. Create the questions to be asked.

2. Allow any participant to respond to the questions.

3. Tally the responses.

First, create the questions to be asked. As noted in the comments on scientific polling, good questions are not easy to ask.

Second, allow any participant to respond to the questions. This non-probability selection of participants with unknown or non-representative demographics is the key characteristic of convenience polls. At the heart of this type of poll is the inability to adjust the poll results based on participant characteristics. Without such adjustments, these types of polls are considered to be of unknown quality and are therefore unreliable.

Third, tally the responses. Tallying simply translates a response into a measured outcome. Since the participants have unknown or non-representative demographic characteristics, it is not possible to weight the results in an attempt to create a representative sample. Therefore, tallying the responses produces the final results.

Ugly Polls

This brief survey of polling would not be complete without discussing “ugly” polls. Ugly polls are polls that make no pretense of being a fair-minded survey of elections or issues. While these polls may be disguised as scientific or convenience polls, in truth these polls are not really polls at all.

Ugly polls are designed to influence public opinion or the actions of the participant.

Ugly polls break faith with the participant by attempting to manipulate the participant into a certain response. These polls are also known as “push” polls or “suppression” polls. Their purpose is to attempt to gain legitimacy for a preconceived outcome. Sometimes these polls are used to influence public opinion. Sometimes these polls are designed to manipulate the participant in various ways such as changing how they might vote, gaining a product endorsement, or convincing them to give money to a cause.

More Information

If you are part of the extra-curious and want more in-depth information on polling, following are some of the resources that I have found to be helpful. Some of the best introductions to public opinion polling are given by industry leaders.

· Pew Research describes public opinion polling basics,

· Gallup explains how polls are conducted and discusses polling methodology,

· Roper Center gives illustrations of polling fundamentals, and

· Roper Center provides a Glossary of Terms.

Other introductory resources that may be helpful are:

· The Pioneers of Polling,

· Standards and Ethics of Public Opinion Research,

· How One Man Used Polling to Change American Politics,

· Pew Research American Trends Panel,

· Pew Research Frequently Asked Questions,

· American Association for Public Opinion Research: Sampling Methods for Political Polling,

· Pew Research After 2016 and 2020 Does Polling Need to be Fixed Again,