The Poll That Didn't Want My Story: Qualitative vs. Quantitative Research and How AI Is Changing Both

In this post:

• Why a Gallup pollster couldn’t capture the real story behind a graduate student’s drinking habits and what that reveals about a century-old trade-off in behavioral science
• The fundamental tension between quantitative research (broad and scalable) and qualitative research (deep but slow)
• How AI is dissolving that trade-off by enabling thousands of in-depth, adaptive conversations at once
• How conversational AI surfaces reasons for behavior that no checkbox survey would ever think to include

When I started graduate school in the fall of 2013, fate called my number—or at least Gallup did. As I left an orientation event, my phone rang. After answering, I heard:

“Hello, my name is Matt, and I’m calling on behalf of Gallup, a research organization. We’re conducting a brief national survey about people’s health and behavior. Do you have a few minutes to participate in our survey?”

I couldn’t believe it. Standing in the lab where I intended to spend the next five years studying human behavior, I had one of the most venerated polling organizations in the world in my hand. Of course I had time.

What followed was an education in the gap between how researchers ask questions in surveys and how people think about their lives.

Matt began by asking about my drinking habits. He wanted to know if I had consumed alcohol in the past week. “Yes,” I said. Then he asked, “In the past month, would you say your drinking has gone up, gone down, or stayed about the same?”

“It’s gone up,” I said, “but that’s because I just started graduate school and—”

“Thank you,” Matt said. Then he continued, “In the past month, how many days per week would you say you typically consumed alcohol?”

“That’s hard to answer,” I said. “I just started graduate school, and there have been a lot of orientation events with alcohol. I just came from one. Plus, students have been organizing happy hours to get to know each other. And, we’re in New Orleans, so you know…alcohol is everywhere. I’m not a heavy drinker, but lately, I’ve had more than normal.”

“So would you say one to two days, three to four days, or five or more days per week?” Matt asked.

Things went on like this.

Matt was polite, and I politely insisted I wasn’t a lush. My recent drinking, I informed Matt, was strictly academic, a product of my situation.

But as we talked, it became clear Matt’s questions didn’t have room to account for my answers. He didn’t have a way to represent the truth: I was a recently transplanted graduate student adjusting to a new social environment in a city famous for its drinking culture. He had checkboxes. My job was to pick one.

By the end of the call, Gallup had its data point—whatever it was worth. What they didn’t have was an understanding of why. As I hung up, I remember feeling that my reasons for drinking were worthy of being measured. But in the Gallup survey, they were not.

The Quantitative vs. Qualitative Trade-Off That Defined a Century of Behavioral Science

My call with Gallup illustrates a tension that has existed within the social and behavioral sciences for generations.

On one side there is quantitative research. Surveys with fixed response options, checkboxes and Likert scales. These methods are efficient and scalable. Researchers can reach thousands of people and analyze their responses statistically. The downside is that quantitative methods force people’s experiences into predetermined categories. If your reason for drinking isn’t on the list, it doesn’t exist in the data.

On the other side is qualitative research. Here you find in-depth interviews and open-ended questions. Researchers have conversations that follow wherever the participant leads. These methods capture how people actually think and feel, and they reveal the why behind behavior.

But, qualitative research comes at the cost of time. A skilled interviewer might complete three or four detailed interviews in a day. Then comes the analysis. It may take weeks, months, or even years to read through transcripts, identify themes, and code what people have said.

For as long as social and behavioral scientists have been conducting research, they have had to choose between the breadth of quantitative measures or the depth of qualitative ones, between closed-ended or open-ended research questions. But AI is starting to change this calculation.

Qualitative vs. Quantitative Research: A Primer

Quantitative research uses structured methods such as surveys with fixed response options, Likert scales, and multiple choice questions to collect data that can be analyzed. Quantitative research is efficient and scalable, but forces human experience into predetermined categories.

Qualitative research uses open-ended methods such as interviews, focus groups, and free-response questions to capture how people actually think and feel. It reveals the why behind behavior, but is difficult to scale.

Mixed methods research combines both approaches, using qualitative insights to inform quantitative measures or using quantitative patterns to guide qualitative exploration.

For decades, researchers had to choose between breadth and depth. AI is changing that equation.

What Changes When AI Enters the Conversation

Let’s imagine a different version of my Gallup call.

Instead of Matt reading from a script, imagine an AI interviewer asks the opening question: “Have you consumed alcohol in the past week?”

“Yes,” I say.

“Has your drinking changed recently?”

“Yes, it’s gone up, but there’s context—” I say.

The AI responds: “I’d love to hear about that. What’s been happening?”

From here, I go on to explain my boozy introduction to New Orleans and what a contrast my recent behavior is with my past. The AI listens, asks follow-up questions, probes parts of my answers that seem interesting, and transcribes our conversation.

Another version of this conversation might incorporate mixed-methods. The AI may first ask me to answer a close-ended question about the frequency of drinking and then follow up with an open-ended question that asks me to explain the answer I chose. Either way, the truly remarkable thing is that while the AI has a detailed conversation with me, it can do the same thing with hundreds or even thousands of other people at the same time.

Then, after the data points are gathered, AI can analyze them. This accelerates how quickly researchers can identify patterns across participants, code responses into categories, and discover themes that emerge from what people said rather than what researchers thought to ask about.

Because AI makes it possible to have a thousand personal conversations at once, the line between quantitative and qualitative research is dissolving. In the future, most studies may contain mixed-methods.

Discovering What You Didn’t Know to Ask

It’s clear that AI can increase the efficiency of qualitative and mixed methods research, but that may not be its greatest strength. Its greatest strength may be discovery.

When researchers used AI to conduct in-depth interviews about why people drink alcohol, the familiar reasons showed up: to relax (40%), to socialize (51%), to cope with stress (43%). These are the options that would appear on any traditional survey.

But the conversations revealed something else: 4% of people said they drink because they feel obligated to, despite having no desire to do so.

This wasn’t an option anyone had thought to include on a checkbox survey. It emerged only because people had space to explain themselves. When someone mentioned feeling pressured, the AI followed up: What kind of pressure? From whom? How does that make you feel?

Other unexpected findings surfaced. Eleven percent of people mentioned drinking to enhance the flavor of food. Thirteen percent drink because it makes movies and music more enjoyable. Twenty-one percent specifically mentioned that alcohol eases their social anxiety—not just that they drink “to socialize,” but that they drink because social situations feel unbearable without it.

These distinctions matter. If you’re designing a public health campaign, “drinks to socialize” and “drinks because social situations feel unbearable” suggest very different interventions. The checkbox would have grouped them together. The conversation pulled them apart.

This is what I wanted to tell Matt from Gallup: my drinking had gone up, but the why mattered. I wasn’t developing a problem. I was adjusting to a new social environment. Understanding the difference requires listening to the story.

What You’ll Learn in Chapter 8

Chapter 8 of Research in the Cloud introduces you to the transformation taking place in research methodology, and it does so by focusing on a survey tool called Engage.

You’ll start by learning about the distinction between quantitative and qualitative approaches. You’ll see concrete examples of how fixed-response questions can miss important nuances, and how open-ended conversations can reveal what you didn’t know to ask about.

Then you’ll explore how AI is changing the equation. You’ll learn how AI-powered conversational surveys work: how researchers design interview guides, how AI conducts adaptive conversations, and how the resulting data can be analyzed. You’ll see an example of a study that asks about drinking in detail, and illustrates how a simple question about alcohol consumption led to the discovery of eleven distinct reasons for drinking.

The chapter’s main project walks you through a mixed-methods study of moral reasoning using the Heinz dilemma. You’ll see how AI interviews can replicate the kind of in-depth qualitative research that Lawrence Kohlberg conducted decades ago, coding responses according to his six stages of moral development and identifying patterns across a full sample. You’ll learn about thematic analysis and how AI tools can assist with thematic analysis while keeping the researcher in control of interpretation.

By the end, you’ll understand not just how these tools work, but when to use them. Quantitative surveys aren’t going away—they remain the right choice for many research questions. But for understanding why people do what they do, conversational methods now offer possibilities that didn’t exist before.

This post is part of a series exploring the chapters of Research in the Cloud: A Hands-On Guide to Behavioral Research in the Digital Age by Aaron Moss, Jonathan Robinson, and Leib Litman. Explore Chapter 8 here.