In a New Capitol Weekly Article, Pollster Ben Tulchin Breaks Down What You Need to Know as a Consumer of Polling Data
To paraphrase a line from the movie “Hustle & Flow” and its soundtrack, it is hard out here for a pollster these days.
Between declining landline participation rates, the steady rise of cell phone use, the prevalence of cheap “robo polls”, the emergence of on-line polling, the “top-two” primary system in California that has resulted in candidates from the same party facing off in November, and a lot of noise due to the ever-increasing amount of polling data available (some good, some not so good), the pressure on a pollster to figure out what is really going on has never been greater at any time since polling established itself as a required and necessary component of modern-day campaigns. So what are pollsters and the campaigns and political junkies that rely on them to do this election?
Now let me address some of the main challenges facing pollsters and consumers of polling data.
Landline vs. cell phones – This is a legitimate challenge as many as 1/3 of households have cell phones and no landline and this number is rising every year. Furthermore, those who have cell phones are disproportionately younger and urban (and include many voters of color) and tend to vote more Democratic. It’s more difficult and expensive to call cell phones as pollsters have to manually dial these numbers unlike with landlines where we can auto-dial them. Based on my own polling as well as studying polls from around the country, especially on the presidential race, undersampling cell phones can undercount support for Democratic candidates, especially in districts with large urban and minority populations.
The good news is that pollsters can call cell phones. In fact, cell phone numbers are on the voter file, so when doing voter surveys drawn from lists of voters, it is feasible to call both landlines and cell phones with surveys done using live callers. Most legitimate pollsters call cell phones, though some may not or may call fewer than they should to save costs and this could have an impact on the results.
California’s new “Top 2” system – If anyone tells you they know how these top 2 races will play out when two candidates from the same party face each other in November, do not believe them. We are in total unchartered waters on this. In races where two Republicans are facing off against one another, it is impossible to predict how Democrats or no party preference voters are going to behave. The same goes for two Democrats running against each other and how Republican and no party preference voters will vote. These voters appear to be up for grabs, but many may not cast a ballot in the race. So the key to look for on election night is just how many votes are cast in these “top 2” races to see if voters from the odd-party-out choose a candidate or take a pass on the race.
Robo-polls (also known as “IVR” (Instant Voice Recognition) – Some are good; some are bad. The two biggest problems with this methodology is robo-polls can’t call cell phones (see above) and, since they are automated, there is no way to confirm the person on the other end is who they say they are (e.g. male or female). To be fair, some robo-polls have been fairly accurate over the years – their track record is better closer to the election when voters have heard more about the candidates or ballot measure. But there are some really bad ones. Rasmussen Reports, for example is evil, pure evil. They are not real pollsters. They do not try to make their surveys representative. Instead, they are pushing an agenda by putting out polls that tilt heavily conservative and Republican in an effort to shape the broader political narrative. If there is one thing you take away from this column, it is to PLEASE disavow polls done by Rasmussen Reports and to tell everyone you know that they are bogus.
Online surveys – Surveys conducted online are an emerging trend in the polling business. While some traditional pollsters look at online polls skeptically, I believe there is a place for them in our profession and they could very well be the dominant methodology in the future. At present, they are still a developing field and their results now sometimes diverge from telephone surveys. Therefore, this election will be an important test for them to see how online results compare to traditional telephone surveys in predicting the outcome of elections. Most interesting to watch will be how they perform in assessing ballot measures, since they offer a potentially more realistic way of having voters “read” a ballot measure and then “voting” on it online as opposed to having to listen to what is often complicated language over the phone and trying to decide on the spot whether to support or oppose a measure. So stay tuned for the results on this.
Presidential polling – The Super Bowl of campaigns, and much of the political world and polling are focused on the presidential race. This year, there is a preponderance of polls and it’s a close election, making polling all that more relevant and interesting. At least that’s what all the polls tell us! So what’s to make of all the polls on the presidential race (national polls, swing state polls, etc.)? Well, since all polls have a margin of error, it is impossible to say which poll is right. Nevertheless, I recommend using the lessons above as your guide – rely on quality pollsters that call both landline and cell phones and which use live callers; reference multiple polls to compare results; and don’t watch them every day but rather every few days, since polls can bounce around day to day due to statistical variance.
The bottom line is polling still works when done right. But remember, polls are a snap shot in time and they have a margin of error, so factor that in when interpreting the results. Above all, like with most things, polling is perfectly healthy as long as it is consumed in moderation.
The original article can be read on the Capitol Weekly Website.