By Dr. George Barna, Director of Research | Cultural Research Center at Arizona Christian University
Glendale, AZ—You cannot spend more than a few minutes engaged with news and information media without being exposed to the results of some survey conducted by one of the nation’s many media-based polling units. Yet, in this era when many people are afraid to openly discuss their political beliefs and preferences there is reason to wonder about the reliability of public opinion surveys.
Are the results delivered by national public opinion surveys these days truly reliable indicators of current public opinion? Not always.
A stunning example of why caution, if not skepticism, must be deployed when reading election-related polling results is a recent survey conducted by CBS News with its polling partner, YouGov. Both organizations are widely known and have a long history of producing surveys. Longevity does not produce reliability, however, as was obvious upon a deeper examination of the data from their August 23 survey. It is not possible to discern whether they intentionally skewed the results, but they shamelessly promoted outcomes that are clearly unreliable.
The most celebrated finding of their survey was that Joe Biden was leading Donald Trump in nationwide presidential preference by a 52% to 42% margin. The sizable gap between the two major party candidates makes the election appear to be a likely runaway victory for Mr. Biden.
However, a simple review of the sample used—that is, the group of people who were interviewed—shows that the sample does not reflect the national population and therefore the margin of preference reported by CBS News is inaccurate and biased in Mr. Biden’s favor.
To be fair, one of the confounding factors faced by all survey researchers engaging in election studies is that no one accurately knows what proportion of Americans are Democrats or Republicans. There is no central database that contains current, accurate, and reliable statistics as to party identification.
In fact, surveys measure party affiliation in different ways. Some ask about the party with which a voter is registered. Others inquire as to how the voter thinks about himself/herself at that moment, regardless of their party registration. Some surveys ask which party the voter feels most aligned with, or best represents their political views. Methodological studies have long demonstrated that differently worded questions will produce different results, despite inquiring about the same condition.
You can be sure, however, that the CBS poll was not an adequate representation of the national population, regardless of the measurement approach taken in that survey. CBS released data based on a sample showing that 41% of voters are Democrats, 30% are Republicans, and 27% are independents. That’s an 11-point gap between those associated with the Democrat and Republican parties.
However, two of the most respected polling organizations—Gallup and Pew—have recently shown that the difference in party affiliation is more likely to be four or five percentage points—not 11. My own national surveys through the Cultural Research Center at Arizona Christian University indicate that the gap is five to six percentage points. The average of the three organizations suggests a party split that associates 35% of registered voters with the Democrats, 30% with the Republicans, and 32% to be independent of party. (FYI, those three numbers do not add to 100% because a small proportion of voters align with other parties, such as Libertarian or Socialist.)
Does that difference between the CBS sample profile and that of other leading researchers really matter? Yes, in a significant way. Party affiliation bears a very strong connection to candidate preference. CBS showed that 95% of Democrats would vote for Mr. Biden with just 3% choosing Mr. Trump. Likewise, their survey found that 93% of Republican voters would side with Mr. Trump versus 5% opting for Mr. Biden.
Bear with me for a moment while I run the numbers to show what happens if the CBS sample were to be more consistent with common norms. If we recalculate the CBS/YouGov statistics to assume that 35% of registered voters are Democrats, 30% are Republicans, and 32% are independent, the race shifts from a 10-point Biden advantage (52% versus 42%) to just a 4-point lead (48% versus 44%).
Another consideration is the voting group evaluated in the survey. Pollsters might gather voting preferences from all adults, from registered voters, or likely voters. The likely voter category is another hidden quagmire to beware of since it is based on a mixture of questions used to estimate how likely someone is to actually vote in an upcoming election. Survey organizations use different combinations of questions that they believe will produce a reliable guess as to who will actually cast a ballot on Election Day. Many pollsters have realized in the wake of an election that their likely voter screen was misguided and led to inaccurate predictions.
The CBS/YouGov survey did not identify the questions involved in their likely voter screening, but that is another potential source of error in their research.
There are many other intricacies that can distort the outcomes generated through surveys: question wording, question sequencing, data weighting, response rate, and other sampling issues (such as ethnic and racial balance) are among the most common.
As if all of those potential landmines were not enough of a hazard, the 2020 election has introduced yet another important consideration: whether respondents feel safe revealing their real opinions to the research firm. With widespread data hacks, street violence, “push polls” (marketing efforts that collect personal data, disguised as opinion surveys), and the loss of friends or even jobs due to political preferences, there is some evidence that survey respondents this year are more likely to avoid answering controversial questions, or to give a “socially desirable” answer in order to protect their well-being. With the well-known bias of CBS toward liberal and progressive candidates, one must wonder what effect that bias had on the survey answers they received.
The bottom line is that CBS’s research reported misleading information—fake news, you might say—by reporting that Mr. Biden has a 10-point lead over Mr. Trump. The axiom is true: the only thing worse than no research is bad research because it provides misleading information.
Dr. George Barna is a professor and is the Director of Research at the Cultural Research Center at Arizona Christian University. He has written more than 50 books.
The Cultural Research Center (CRC) at Arizona Christian University is located on the school’s campus in Glendale, Arizona. CRC conducts nationwide research studies to understand the intersection of faith and culture and shares that information with organizations dedicated to transform American culture with biblical truth. Like ACU, CRC embraces the Christian faith, as described in the Bible, but remains inter-denominational and non-partisan
Phone:
Email: