To get good answers, you have to ask good questions.

“Generally, do you think science and religion are often in conflict?” is not a very helpful question. It’s vague from start to finish. Religion could refer to the beliefs and practices of evangelical Protestants, Jesuits, secular Jews, or Theravada Buddhists — or even some imagined mashup of all global faiths. Science is fuzzy, too. It refers to a wide range of disciplines (e.g. astronomy, botany, psychiatry) and occurs in a variety of settings (laboratories, forests, high school auditoriums). In the popular imagination, it’s practiced by everyone from white-coated nerds to egomaniacal monsters.

We founded the Cubit, Religion Dispatches‘ religion and science portal, with the goal of moving beyond simplistic formulations of religion and science. We believe that as soon as you ask a question like the one above, you’re already dealing in abstractions that are of very little use.

So it’s a shame that this vague question is the centerpiece of a major survey from the Pew Research Center. It suffers from serious methodological flaws, and seems formulated to fuel internet hot-takes rather than productive public discourse.

This is just one study, of course, but it matters for two reasons. First, Pew’s study manufactures and reinforces a narrow view of religion and science, while claiming to objectively measure public opinion. And second, the study’s extensive and largely uncritical media coverage is emblematic of the dysfunctional relationship between pollsters and reporters.

The Basics
Pew surveyed 2,000 Americans about their perceptions of the relationship between religion and science. According to the survey, 59 percent of Americans believe that the two are often in conflict, up from 55 percent in 2009.

Pew also reported two surprising aspects of these general views.
First, although most Americans apparently believe religion and science to be in conflict, far fewer think that their own religious beliefs conflict with science.

Second, the least religiously observant are most likely to say that science and religion are in conflict.

Secular Americans, it seems, widely perceive religious believers to be anti-science. Meanwhile, those who attend religious services weekly are more likely to believe in a middle ground.

Below these headline-friendly findings, the report offers some useful data about public perceptions of religion and science, including fine-grained details about American attitudes towards GMOs, nuclear power, and other science-and-society issues.

But while they may offer rough estimates of public opinion, these results are far from objective or clean. Beneath the colorful surface of each graph lie a series of filters, amplifiers, and cultural biases that mediate any representation of public opinion.

Digging In
We recently discussed, at length, the inherent limitations of polls and surveys when it comes to religious belief. Some of the nuts-and-bolts methodological limitations are relevant here — in particular, the poll obscures the views of religious minorities, and it doesn’t report its response rate, which means we don’t know how many people just ignored the pollsters outright (judging by comparable Pew polls, the non-response rate was probably around 90 percent).

But the main issue with this survey has to do with the way the central questions were formulated.

Most popular polls, like Pew studies, are conducted over the phone using a list of simple multiple choice questions. Many forms of public opinion are accurately predicted this way, including voting behavior, because they are easily distilled into multiple choice questions. “Are you voting for Hillary/Sanders or Rubio/Christie next week?” is a straightforward question.

Religious beliefs, on the other hand, are fuzzy. They’re dynamic and complicated. It isn’t unusual for Americans to fall one way on an ethical issue when discussing it at church and then, in another context later that week, shift a few degrees over. Polls can’t capture this flexibility.

Pew’s questions also assume that people have pre-formed opinions on issues that they may have never considered. Take the following question from deep in this report:

“Thinking about the use of biological engineering to create artificial organs for humans needing a transplant operation, would you say this is making appropriate use of medical advances OR is it taking medical advances too far?”

We write about science, religion, and ethics for a living, and neither of us has a concrete answer to this question. Assuming that most participants haven’t already thought this through, the answer they choose after mere seconds of consideration isn’t likely to reflect any longstanding, steadfast belief.

“Polls are hastily done by telephone rather than in person,” Princeton sociologist Robert Wuthnow said in a recent interview with us:

“If you’re a respondent in a poll, you’re being asked anywhere from four to five complex questions every minute. They’re throwing questions at you so rapidly that you often have no idea whether you really gave the right answer that you really believe or not.”

Since survey participants are answering rapid-fire, multiple-choice questions about fuzzy beliefs, they’re likely to be influenced by the context of the survey. In the case of Pew’s science and religion survey, the most obvious influence is in the way that they formed the question.

When asked, “Generally, do you think science and religion are often in conflict?” the participant is locked into a specific formulation of concepts. “Religion” and “science,” as we mentioned earlier, can point to a very wide range of traditions, activities, and groups of people. “Conflict” here could refer to a fundamental epistemological incompatibility between religion and science, or to secondary social tensions between dogmatists on one or both sides. Or it could refer to some combination of both. Asked whether science and religion are, “often in conflict” or, “mostly compatible,” the poll participant has to collapse any such nuances and pigeonhole their beliefs into one side of a binary.

It’s important to remember that Pew is measuring quick and narrow responses, not steadfast beliefs. The takeaway of their survey should not be that 59 percent of Americans believe that religion and science often conflict, which implies that more than half of the American public are walking around, right now, independently convinced of this proposition. It would be more accurate to say that, when asked to choose whether religion and science are “often in conflict” or “mostly compatible,” 59 percent of Americans chose the former as more representative of their beliefs.
In simpler terms, Pew has given us a narrow, skewed insight into how a select subset of Americans respond to leading questions, posed by strangers, over the phone.

But It’s Science!
Public opinion polling ends up existing in a strange gray space between science and journalism. A Pew report is definitively not a scientific paper: its goal is to serve media outlets, not to advance a body of academic inquiry. It’s not peer-reviewed. Its methods lack the scope and rigor of a formal social science study.

That doesn’t make these reports useless. Pew is performing a kind of large-scale journalism. But, because it involves numbers and stats, journalists generally treat polls as if they’re hard science, establishing basic facts about the American public, rather than with the kind of cautious skepticism that, in theory, journalists should bring to any story.

Anyway, it’s boring; who wants to wade through all those stats? And accounting for fuzziness and sampling problems makes for even duller headlines. Meanwhile, polls provide ready-made hooks for journalists to use for quick, easy copy. Dozens of media outlets reported on Pew’s results, proclaiming that “Most in U.S. Say Religion and Science Conflict,” or “Conflicts between Faith, Science May be Overhyped.” These headlines are misleading, but they’re much sexier than our own best attempt: Someone Prompted a Small Sample of Americans to Choose One of Two Options Signifying the Relationship Between Science and Religion, and You Won’t Believe What Happened Next!

Look, we’re trying.

Reporting the Report
While the study provided a bevy of individual findings to choose from, with one major exception, journalists generally selected one element of the survey data and rebroadcast it, accepting the report’s claims at face value. We dragged a wide net through the media ocean, and found that responses to the survey tended to fall into one of three categories.

The Straight and Narrow: A significant number of articles made an easy hook out of the topic, emphasizing the hot button issue of science and religion in their headlines. The headlines emphasized either conflict (“Most Americans Think Science, Religion Conflict”), or reconciliation (“Reconciling faith and science: Pew study”; “Conflicts between faith, science may be overhyped, survey results show”). Whatever the hook, these articles then offer straightforward summaries of Pew’s general findings.

The Counterintuitive: A more focused group of articles highlighted the negative correlation between religiosity and the perception that science and religion are in conflict. This makes for a good hook because it’s initially counterintuitive: Slate’s Rachel E. Gross introduces the finding as a “head-scratcher,” while Grist’s Kate Yoder remarks, “Turns out that if you’re snoozing on your mattress on Sunday morning, you’re actually more likely to see science and religion in conflict than the pious folks sitting in the pews.”

Reuters focused in on this result too, but no one asked the critical question: what if the more religious you are, the more likely you are to assume that true science agrees with your faith, and that all other science (evolutionary biology, climatology) is just fake science? This is an increasingly common attitude within certain pockets of conservative Christianity, yet in this survey it would be lumped in with liberal ecumenism towards science. Once again, we butt up against a basic problem: it’s not clear that everyone is talking about the same science. (It’s also not clear what makes someone religious; Pew just asked subjects to self-report their attendance at religious services. The problem: people consistently, egregiously lie to pollsters about this).

The Hypocrisy: A third set of articles highlighted the finding that Americans are likelier to say that religion and science are in conflict than they are to say that their personal religious beliefs conflict with science. Time, CNET, and LiveScience all took up this angle, and many chalked this up to hypocrisy. LiveScience’s Tia Ghose, for example, interprets Pew’s report to indicate that Americans perceive anti-science to be “a problem with those other people.” CNET’s Chris Matyszczyk attributes this to a holier-than-thou attitude that’s fundamental to human nature.

If we remind ourselves that Pew is recording quick reactions to narrow questions, this finding is much less scandalous. The question of whether religion and science are “often in conflict,” for example, could have been interpreted by respondents to be about religion-and-science interactions in the broader culture. Since the media often focus on stories about religion and science in conflict, amplifying the voices of militant atheists and creationists, many Americans probably perceive there to be more conflict “out there” than in their own personal beliefs. There is no data here to support the idea that Americans are hypocrites.

The Map Is Not the Territory
“A map” writes the anthropologist James Scott, “is an instrument designed for a purpose.” No map can encompass every single detail of the territory it intends to cover. Instead, all maps must simplify — they must emphasize or obscure various details of the landscape in order to be useful. With this, writes Scott, comes “the apparent power of maps to transform as well as summarize the facts that they portray.”

Polls and journalism often work together to map society. In the questions they choose to ask, and the categories that they choose to highlight, they elevate and coronate a very particular slice of the messy social terrain.
When it comes to science and religion, the mapmakers, again and again, focus on a familiar narrative of conflict. And, because they look for it, they find it. That’s not to say that this conflict doesn’t exist — people are obviously bickering about this stuff all the time — but that, by mapping a certain conflict, in certain terms, you miss all the other ways that religion and science, interact, mingle, and, yes, clash.

An even bigger problem is that journalists often treat the pollster’s map as territory. Rather than scrutinize the pollster’s methods, they report their findings as object facts and add their own spin. Readers are left with a map of a map — a caricature, drawn in thick lines on thin paper.

This article is reprinted with permission from Religion Dispatches. Follow RD on Facebook or Twitter for daily updates.