David Flaherty on Polling 101: Part 1

Republican Pollster and President of Magellan Strategies presents great insight into how to effectively use polling, how polls are conducted, and the future of polling

This week, we present Part 1 of our Polling 101 series with Republican Pollster, David Flaherty, President of the Polling and Survey Research firm, Magellan Strategies.

Parts 1 and Part 2 provide candidates, campaign managers, and political consultants great insight into how to effectively use polling and get better results.

In Part 1, David Flaherty lays out everything you could possibly want to know about how polls are conducted, how to effectively use them, and what can be done to meet the challenges facing pollsters over the next decade. We talk about why so many polls recently have been wrong, including the recent British elections, the Greek Referendum, and former Majority Leader Eric Cantor’s surprise defeat. 

In Part 2, we continue to look under the hood to discover the nuts and bolts of effective survey research, including how to put together your polling plan, what goes into an effective Benchmark poll, and we use a case study to demonstrate how to use polling to hone your message.  Then, we discuss how regular people can better evaluate the public polling the media reports on in the upcoming 2016 presidential race.

We asked David Flaherty to be on the program because he has nearly 25 years of experience working with data and voter files. He is known as an expert on mapping technology and the redistricting process. As a pollster, this deep knowledge of data, demographics, and voter files is a real asset when it comes to understanding a population that you are trying to survey. 

David spent 14 years in Washington, DC working for organizations like the Republican National Committee, the US House of Representatives, and Americans for Prosperity. In 2006, David moved to Colorado where he later started Magellan Strategies. David and his team provide survey research, predictive data modeling, and voter data analysis services to political and public affairs clients.

Show Notes:

Check out David Flaherty’s Full Bio [Here].

Check out the Magellan Strategies [Here].

Check out some articles by David Flaherty:

• Using Data to Make Better Campaign Decisions
• Magellan Strategies 2012 Polling Review

Download the accompanying Polling 101 Presentation [Here]. We roughly follow this through our conversation.

PART 1

In this episode, we discuss:

How David Flaherty got started in politics:

• Dave’s first job in the Voter Programs Department of the Republican National Committee in 1992. 
• The department’s primary role was to build a national voter file each election cycle

The importance of a data background in polling:

• The transition from solely focusing on the crosstab analysis to also focusing on your sample, demographics and the math behind your poll at a more granular level
• Understanding Data, Demographics, and Voter Files gives a pollster a better handle on what the electorate will look like on election day, improving results over time
• A strong data background is what sets Magellan apart from some pollsters

David’s background and experience with Redistricting and the Census:

• The full redistricting cycle occurs every 10 years
• Census taking place on the 10’s: 2000, 2010, 2020, etc.
• So the Census data is available and Redistricting takes place the following year
• David used collected voter file data and GIS mapping software to assist the Republican side during the Redistricting process
• Developed a skillset and reputation for understanding maps and Redistricting
• Recruited to work for the U.S. House of Representatives Sub-committee on the Census in the run up to the 2000 Census
• Returned to the RNC as the head of Redistricting Technology for the 2001 Redistricting process
• Dave describes in detail how the Census is conducted

Polling 101:

The Sample is Key: Pollster needs to answer “What is the population we are attempting to measure?”

• Polling is like taking a sip of soup; you don’t need to eat the entire bowl to know how it tastes
• Different approaches to accurately survey the right population: “Where are we going to get our sample?”
• List Based Sampling: 
• Starts with a list of all of the registered voters in your target community (district, state, nation) and then at random pull a sample from these registered voters 
• Magellan uses this because of the heavy data and voter file background and their preference of only contacting households that are able to vote
• List Based Sampling is particularly good for surveys of districts; for instance a Congressional District
• Random Digit Dialing (RDD)
• A computer program creates phone numbers at random using the telephone prefixes that are know to be in your target population (district, state, nation)
• RDD requires you to ask more screening questions upfront to make sure they are a registered voter and that they live in the district
• Even when you ask someone if they live in a particular district, many voters really don’t know which districts they live in.
• RDD does better in Statewide and National surveys than in surveys of districts
• Both methods can be accurate or inaccurate
• The less solid your sample is the less likely you are allowing likely voters at random to participate in your survey
• Coverage Problems
• The Ongoing Challenge: Decreasing number of households with landlines and the increase of mobile-only households
• Surveys for certain groups require a different mix of sampling tools because otherwise they won’t be representative of the population
• A Democrat Primary survey will be off if it only includes landlines
• A Republican Primary survey currently can be pretty accurate if it only includes landlines
• The Center for Disease Control tracks the number of adults who are cell phone only. The younger the population and the more minority the population the less likely they are to have a landline phone.
• 2012 Election: Many pollsters over sampled older voters and under sampled younger voters, which then under measured Barack Obama’s strength
• How are Pollsters going to deal with coverage problems that come from declining landline households over the next 10 years?
• Dave predicts you will not be able to field a phone only presidential election survey by 2024
• Simply significantly increasing cell phones in your sample will not give you the coverage you need to accurately represent the population
• It’s cost prohibitive
• Cell phone interview respondents are changing behaviorally
• You know landline respondents are in their home, but with cell phone respondents they may not be
• They are in a quiet, private place where they can focus on the questions
• Cell phone interviews require a lot of callbacks to get someone when they aren’t driving or when it is a better time to talk
• The Incidence Rate (the response rate) is much lower for a cell phone because they may not be comfortable answering questions about their political views in public
• The people you get on their cell phones will likely not be representative enough of the projected population you are trying to understand
• You can only weight results so much to make up for missing data
• Online Panel Surveys may be able to help supplement
• There has been some success here
• There have been some big failures also
• Europe uses more online surveys
• British Parliamentary Elections: The polls underrepresented conservatives and failed to predict a Conservative victory
• Why did they get it wrong in Britain?
• Underpredicted the Scottish break from Labour
• Herding (i.e. Pollsters were afraid their numbers were wrong)
• How has Dave handled this situation of having a poll that’s different from other pollsters?
• Trust your numbers
• Trust your methodology
• Be thick-skinned
• Example: Magellan predicted the Republican would win the Special Election for disgraced Democrat Congressman Anthony Weiner’s New York City district when no other pollsters were predicting that. The Republican won.
• What is a good mix of demographics and methodology?
• Depends on the state or district
• Midwest is whiter and older
• West, California and Florida are cross-cultural
• Older and whiter demos can use more landline interviews
• Other regions need multi-mode interviews and can be complicated
• Where are online panels now?
• Not mature enough yet in most places
• Panels tend to be too female, too educated, too wealthy
• They also include many unemployed
• Promising supplement
• People are also paid or incentivized to participate in panels
• How do online panels need to change?
• More participants
• Behavioral changes
• People not researching their answers during the survey

How a Poll is Conducted:

• Prepare script with client
• Reserve call center time for the length of the survey
• Randomly draw people to call from the voter file
• Check change of address list
• Do landline and cell phone match
• Make sure the draw matches demographics of the district
• Upload script into CATI machine (Computer-Assisted Telephone Interviewing)
• Check upload
• Check pronunciations
• Listen in on the calls at the beginning of the first night
• Receive data in the morning
• Take unweighted data and review it
• Primarily uses UNCLE software
• Many use SPSS
• Weight responses up or down
• Look at each segment
• Randomly select interviews to be included twice or excluded in order to weight a demographic up or down
• How many are too few for a segment or for a survey overall
• 50-75 in a segment
• 300 minimum interviews for the entire survey
• 500 is ideal for the entire survey
• Do more surveys help?
• Methodology matters the most – a bad sample creates bad results regardless
• More surveys help to a point – 1000 is about right for a larger survey
• There is a law of diminishing returns. The margin of error starts to only marginally decrease as the number of interviews increases
• How many interviews are needed for a microtargeting project?
• This is modeling and a different kind of project
• Absolutely need more interviews to be able to understand voters on an individual level
• 5,000 is the preferred threshold on these projects
• How many days do you need to be in the field 
• Depends on the survey
• Benchmark- 2 to 3 nights
• Tracking- Longer
• Data collected from the survey
• Marginals are created after each day
• They don’t give you the full picture
• Can be misleading if they don’t match your overall sample
• What do you do after all of the survey data has been collected
• Create a topline document/results
• Setup Crosstabs
• Create Presentation
• Write Executive Summary
• Present Phone Presentation to Campaign

Be sure to tune in for Part 2, when we cover effective polling analysis, use a case study to explain effective message testing, talk about what it takes to conduct an effective Benchmark poll, and discuss how to evaluate all of the public polls in 2016 Presidential race. 

We want to learn from you! Please post whatever comments or questions you have from this episode below.

IF YOU LIKE WHAT YOU HEAR, PLEASE:

• Subscribe to the show on iTunes [Click Here]
• Leave a quick review [Click Here]
• Let us know what other topics we should cover!