What Can We Learn From Asking People to Talk About Candidates?
By Jon Green
In principle, a survey is a conversation. Like all of those Rust Belt safaris that feature political reporters asking a wildly unrepresentative segment of the voting public what they think about politics, but no longer wildly unrepresentative. However, political surveys are typically limited as they heavily constrain what respondents can say as part of this conversation. Respondents can say yes or no, agree or disagree, and so on, and what researchers lose by restricting everyone to a common vocabulary they gain in being able to systematically analyze the results. However, it’s becoming easier to both collect and analyze the less constrained vocabularies that citizens have when offered the chance to talk about politics, as we have found in the most recent wave of our national Democratic primary panel survey.
In this survey, we tried to get back in touch with as many of the 2,953 respondents who participated in the first wave of the survey, which we fielded from June 24th through July 2nd of last year. We were able to recontact 1,619 likely Democratic primary voters between January 18th and 27th. In addition to asking them the usual things you see on presidential polls, such as who they were planning on voting for and which candidates they were considering, we asked them to write down the first thing that came to mind when they thought of each of the presidential candidates who appeared in the last debate: Joe Biden, Bernie Sanders, Elizabeth Warren, Pete Buttigieg, Amy Klobuchar, and Tom Steyer. Here, I focus on responses for the first three of those candidates -- Biden, Sanders, and Warren -- both for the sake of space, and because respondents didn’t have as much to say about some of the others.
In order to systematically analyze the text included in respondents’ word associations, I used pivoted text scaling (also referred to as pivot scaling), a method recently developed by Cornell professor William Hobbs for short text segments. Pivot scaling identifies common pivot words, and words that tend to appear alongside them, which allows for the creation of interpretable latent dimensions in short text documents such as open-ended survey responses. This is especially useful when each individual document is short -- as is the case with open-ended survey responses -- since it is less sensitive to noise introduced by uncommon words. Essentially, applied to our data, the method produces a generalizable, numeric estimate of what people say when asked to summarize their thoughts about each of these candidates.
Before continuing, it’s important to make a quick note on text analysis methodology: Since my computer is not fluent in English, I took some common preprocessing steps before running these word associations through the pivot scaling algorithm to help it make sense of the text. This included sending all words to lower case and, most importantly, stemming them. This involves reducing words with substantively similar meanings but distinct spellings to their common root -- “knowledge” and “knowledgeable” both become “knowledg,” for instance, so that the algorithm treats them as the same rather than completely independent. Long story short, if the words in the tables below look cut off to you, it’s because they are, and that was intentional.
For each candidate, I extracted the first pivot scale dimension in respondents’ word associations for that candidate -- respondents who did not write anything about a given candidate are assigned values of zero on that respective dimension. The initial results from this procedure are shown below, with tables showing the commonly-used pivot words associated with each pole of the first dimension for each candidate.
A first glance at the results shows that respondents have meaningful, divergent associations for each candidate such that each candidate-specific dimension that emerges can be reasonably interpreted as running from negative to positive sentiment. Respondents who associate things like “Republican,” “status quo,” or “corporate” with Joe Biden are on the opposite end of this latent dimension than respondents who associate the former vice president with words like “compassion,” “honest,” and “experience.” For Bernie Sanders, respondents who mention positive words such as “leader,” “consistent,” and “strong” emerge as distinct from respondents who express concerns about his health (“heart”) or describe him as angry. Finally, respondents who mention Bernie Sanders when asked to think about Elizabeth Warren, or those who express concerns regarding her electability or sincerity (the word “win” tends to appear in the context of respondents expressing concern that she can’t win, or that she will “say anything” to win), are on the negative pole of her first latent word association dimension -- the positive pole includes words such as “fighter,” “knowledge,” and “honest.”
First Dimension (Stemmed) Keywords: Biden:
| Negative Pivots | Positive Pivots |
|---|---|
| republican | compassion |
| chang | honest |
| status | knowledg |
| quo | experienc |
| trust | strong |
| corpor | govern |
First Dimension (Stemmed) Keywords: Sanders
| Negative Pivots | Positive Pivots |
|---|---|
| may | compassion |
| heart | fighter |
| vote | leader |
| seem | strong |
| angri | consist |
| plan | help |
First Dimension (Stemmed) Keywords: Warren
| Negative Pivots | Positive Pivots |
|---|---|
| berni | compassion |
| vote | experienc |
| support | fighter |
| say | honest |
| just | knowledg |
| win | american |
While negative pivot words tend to be specific to the candidate, there are marked similarities in the positive pivot words across each. Biden is criticized for being too centrist, Sanders is criticized for being too angry, and Warren is criticized for being insincere, but they are all praised (by different respondents) for being compassionate. Biden and Sanders are both praised for being strong; Biden and Warren are both praised for being knowledgeable; Sanders and Warren are both praised for being fighters. One of the more revealing aspects of these positive pivots, then, is which candidate is missing when a word appears as a positive association for the other two. As in, it says a lot about how the likely Democratic electorate is thinking about their options that “fighter” does not tend to appear in positive associations for Joe Biden, that “knowledgeable” does not tend to appear in positive associations for Bernie Sanders, and that “strong” (which, it is important to note, is a heavily gendered concept in American politics) does not tend to appear in positive associations for Elizabeth Warren.
Recall that this is a panel survey, so we know which candidates respondents were considering voting for last summer. For each respondent/candidate combination, I created a variable that compares whether they the respondent was considering voting for the candidate in the first wave of the survey to whether they are considering voting for the candidate now. So a respondent could be considering a candidate in both waves, not considering them in either wave, or moving in or out of a candidate’s pool of potential voters. In the charts below, we see that different candidates have different distributions of word association responses by respondent type.
For Joe Biden, for instance, respondents who didn’t have anything nice to say tended to not say much at all -- with a few extreme exceptions. You can see this represented in the fact that the distributions for respondents who are not currently considering voting for Biden -- regardless of whether they were considering him in the first wave of the survey -- are both centered around zero, but there is a long tail to the left showing that a small number of respondents who were not considering supporting Biden in either survey wave (shaded in green in the chart) had extreme negative sentiment. Respondents who are newly considering Biden (orange) -- and especially respondents who were considering Biden in both waves of the survey (grey) -- tend to have more positive values on this latent dimension.
Repeating this process for Bernie Sanders, we see that respondents who have consistently included him in the list of candidates they are considering (shaded in grey in the chart) have much more positive word associations for him than other respondents -- even those who are considering him now but were not over the summer (orange). This pattern is not repeated in the distributions for Elizabeth Warren, for whom those who are new to her pool of potential support have a similar distribution of word associations compared to those who have been considering her all along, and those who were considering supporting her in the first wave of the survey but are not anymore (blue) report only slightly less negative word associations regarding her than those who have consistently omitted her from their list of candidates they’d consider supporting (green).
All this is interesting, but the question remains: does this help us understand who respondents are planning on supporting? To answer this question, I predict each candidate’s support using four sets of variables. The first set, which we’ll call the baseline model, uses demographic variables such as age, race, gender, education, and income; as well as political identity variables (self-reported partisanship and ideology) and attitudinal variables (how respondents answered the racism, sexism, left populism, and social dominance batteries in Wave 1 of the panel -- you can find question wordings for these items here). The second set includes all of the variables in the first, but adds the information derived from respondents’ word associations for the candidate in question. The third builds on the baseline model by including an indicator for whether the candidate in question was the respondent’s first choice in Wave 1. The final model includes all of these variables, combining our baseline with the latent text dimension and Wave 1 support.
In order to test the robustness of these relationships, I split the data into 1000 different training and test sets, with each training set containing a randomly sampled 70% of observations and each test set containing the corresponding 30%. For each split, I estimate each model on the training set, and used the results from that model to predict new outcomes in the test set. This allows me to test how robust the relationships identified in the models are when subjected to different subsets of the data that they did not see.
In cases such as these, it is easy for a model to be accurate without being informative. If ten percent of the electorate supports a candidate, simply predicting that no one supports them will be right 90% of the time. To avoid this issue in model interpretation, I evaluate model fit using a measure called the area under the precision-recall curve (AUC-PR). Precision is defined as the percent of cases the model predicts to be positive that are correct; recall is defined as the percent of true cases the model correctly predicts; AUC-PR balances between them. The AUC-PR is derived by taking the predicted probabilities of support, plotting precision against recall against each other at different thresholds of predicting support (as in, predicting support when the probability of support is above .1, .3, .5, .7. .9, etc.), and taking the area under that curve. An AUC-PR of 1 means that not only are all predictions correct, they are correct at every possible cutoff for determining what predictions to make.
I find that it’s helpful to think of precision and recall in terms of a game of Battleship: precision is what percentage of your guesses hit; recall is the percent of possible hits you’ve successfully made. If you guess the entire board, your recall will be perfect, but you will have been terribly imprecise.
The results from this prediction routine are shown below. Generally speaking, the models are better at recovering Joe Biden’s support compared to Bernie Sanders’s, and are better at recovering Sanders’s support compared to Warren. And as one might expect, including Wave 1 support dramatically improves fit over that which can be recovered using the variables you would normally use to model candidate support in a cross-sectional survey. However, we also see that each candidate’s word-association text dimension improves fit over these models in both cases -- especially for Bernie Sanders, where the improvement in fit relative to the baseline from including text features approaches the improvement in fit from including Wave 1 support. And despite starting from a lower baseline, the full model for Sanders support approaches the fit achieved in the full model for Biden support. Again, it’s important to keep in mind that these are predictions out of sample, which allows us to be confident that we are not simply picking up noise and overfitting.
To be clear, it is not surprising that saying positive or negative things about a candidate is associated with supporting them in an election. And may only be somewhat surprising that saying positive or negative things about a candidate improves our ability to predict whether someone supports them relative to the information we typically have at our disposal. For me, one of the biggest takeaways here is that it is easy and efficient for those conducting surveys in the Democratic primary -- especially online -- to let their respondents speak for themselves. There is information in what likely Democratic primary voters have to say that tells us more about who they’re supporting now, and why, than simply who they are or even who they were supporting in the past.
Moreover, letting respondents literally tell us what they think about each candidate reveals interesting things about the nature of their appeals. For instance, Joe Biden is described positively as knowledgeable and strong, but not as a fighter -- and he is negatively associated with Republicans and the status quo. The extent to which a given voter wants someone with a more antagonistic posture toward Republicans and the political system more broadly says a lot about whether they intend to vote for him. Bernie Sanders is described positively as strong and as a fighter, but those who have positive things to say about him don’t tend to mention his knowledge. Voters who care about having a candidate who touts their policy expertise are likely looking elsewhere. Finally, Elizabeth Warren is described positively as knowledgeable and as a fighter, but not as strong -- and is criticized for perceived inauthenticity. This is particularly interesting given the gendered nature of the concept of “strength” in American politics, and suggests the need for a closer look at the relationship between gender attitudes and support for Elizabeth Warren in the Democratic primary. But that will have to be the subject of another post.
Methodology
This survey was conducted in two waves. The first included 2,953 interviews conducted from June 24th to July 2nd, 2019 by YouGov on the internet of registered voters likely to vote in the Democratic presidential primary in 2020. A sample of 6,116 interviews of self-identified registered voters was selected to be representative of registered voters and weighted according to gender, age, race, education, region, and past presidential vote based on registered voters in the November 2016 Current Population Survey, conducted by the U.S. Bureau of the Census. The sample was then subsetted to only look at respondents who reported they were likely to vote in their state’s Democratic primary or caucus. The second wave included 1,619 interviews based on recontacting respondents participating in the first wave (a 55% recontact rate). Respondents participated from January 18th to January 27th, 2020.
Jon Green is a cofounder of Data for Progress