Polling Simulator

Many news organizations and academic institutions conduct political polls. Have you ever wondered why some polls are better at predicting the outcomes of elections than others?

Relying on a dataset of American voters, this polling simulator allows you to test a variety of polling methodologies and experiment with how different methodologies perform under different circumstances — for example, what happens if Republicans are more likely to pick up the phone than Democrats when pollsters call? What if wealthy voters are more likely to lie to a pollster than middle class voters? Use this simulator to test your theories about what makes a poll accurate.

Feel free to jump right in! If you want, you can use the default settings as a starting point (by pressing the ‘click here to use default behaviors and methodologies’ button) and work from there. To learn more about the data, the simulator’s features, and how to navigate the more technical stuff, check out the fine print below.

The data for this project is derived from the Subnational Ideology and Presidential Vote Estimates dataset (Warshaw and Tausanovitch 2022). Joined with survey results from the Congressional Cooperative Election Survey (CCES), this dataset provides various demographic indicators for about 500,000 survey takers between the years of 2006 and 2021, along with numerical estimates of partisan ideology for each respondent. These ideological estimates — or ‘ideal points’ — lay on an approximate range of -2 to 2, with negative values representing liberals and positive values representing conservatives. Ideal point estimates are generated from a two-parameter item response theory model, based on answers to policy questions in the CCES (Lewis and Tausanovitch 2018).

Each time the simulator runs, it randomly samples a large number of individuals from this dataset (with replacement) according to weights which capture the relative proportion of U.S. residents who are demographically similar to each individual. This ensures that the final dataset is representative of the U.S. voter population.

population behaviors

The simulator has five tabs which allow you to set attributes of individuals in the population.

The first tab is latent party preference, which describes whether individuals are ‘truly’ democrats, republicans, or other. This ternary variable can mirror individuals’ responses to party self-identification in the original survey (the ‘id’ model type), or it can be a function of the ideal point estimates discussed above, assigned in either a deterministic (‘strict’ model type) or probabilistic (‘distributed_weights’ model type) way. Under the strict model type, the partisan breakdown of the population can be changed by shifting the cutoffs left and right — consider experimenting with how a more Democratic or Republican population affects polling outcomes.

The second tab is latent voting intention, which describes whether an individual will or will not vote. This binary variable can be assigned using historical data (‘historical’ model type), either through self reporting or validated matches to voter files, both of which are included in the original dataset. Voting intention can also be determined by a weighted random choice according to user-specified weights. These weights can be static (they apply equally to everyone; the ‘static_weights’ model type), but you can also add ‘components’ which increase or decrease these weights for certain subgroups of the population (the ‘custom_weights’ model type). Consider experimenting with a custom component that emulates a scenario in which one party has a difficult time driving out turnout.

Third is latent response probability, which determines the probability that an individual will ‘pick up the phone’ when contacted by a pollster. Here, you specify weights which are converted to probabilities through simple Manhattan normalization (dividing each weight by the sum of all weights). Again, these weights can be static (the ‘static_weights’ model type), or custom components can be specified (the ‘custom_weights’ model type). Consider experimenting with custom weights that make members of one particular party more likely to answer the phone.

Fourth is stated party preference, which describes how individuals would answer the question, ‘which party do you plan on voting for in this election?’ Like latent party preference, this is a ternary variable. Stated party preference can mirror latent party preference (the ‘honest’ model type), or can be weight based like other rules (the ‘static_weights’ or ‘custom_weights’ model types). Again, consider experimenting with varying weights for particular subgroups — for example, increase the likelihood that Republican voters don’t share their true preference to replicate the alleged ‘shy Trump voter’ phenomenon.

Fifth is stated voting intention, which describes how individuals would answer the question, ‘do you plan on voting in this election?’ Again, this variable can mirror its latent counterpart (the ‘honest’ model type), or can be assigned based on static or customized weights (the ‘static_weights’ or ‘custom_weights’ model types). Consider experimenting with varying levels of honesty and dishonesty among subgroups — what happens if one party is generally less likely to share whether or not they will vote?

polling methodologies

Once these behaviors are specified, you must specify polling methodologies. You can specify up to 5 methodologies with which to analyze polls. For each methodology, you must specify a number of parameters.

First is desired sample size. For a methodology with a desired sample size of n, n/(the average response probability in the full population) individuals will be ‘contacted’ by the pollster, and approximately n will respond.

Next is the screen. If the screen is all_respondents, all survey respondents will be considered during analysis of the poll results. If the screen is stated_voters, only individuals whose stated voting intention is ‘voting’ will be considered during analysis. This is analogous to a likely voter screen.

Next is weights. Users can specify whether or not to weight a sample on a variety of demographic categories, including party identification, race, age, sex, income, and education. Survey samples are weighted using iterative proportional fitting, more commonly known as raking, implemented using the python package ipfn.

If you specify weights, then must also specify weight targets and the standard deviation of the weight target error distribution (if no weights are specified, these variables are not needed). If the weight target is the full_population, then the sample will be weighted according to the distribution of various demographic groups in the full population; if the weight target is voting_population, the distribution of various demographic groups in the population of individuals who have a latent voting intention of ‘voting’ is used instead. The weight target error distribution is used to allow for uncertainty around the aggregates of the voting population. One value is sampled from this distribution per demographic group per category, and is converted to a scale factor (for example, 5.5 and -7.8 sampled from N(0, 10) would translate to 1.055 and 0.922, respectively). These scale factors are then multiplied by their respective marginals.

polling error distributions

Finally, an election is simulated among the population. The results are recorded and compared to the poll results from the user-specified methodologies. From this comparison comes polling error distributions, which are displayed using d3.js. Users can view these distributions for the democratic versus republican 2-way vote share, or for any particular party on its own. Hovering over elements on the methodologies legend will lower the opacity of all other distributions for ease of viewing.