From the Lab to the Cloud: How Online Platforms Transformed Behavioral Research

In this post:

• Why slow and grueling in-person data collection gave way to a revolution that now powers roughly 5 billion online surveys a year
• How the online participant ecosystem is structured: panels, aggregators, and researcher-centric platforms
• The trade-offs that matter most when choosing a source: data quality vs. scale, cost vs. control, and breadth vs. depth
• Why the deceptively simple tools of online research require real expertise to use well

Dissertations are a slog. You develop an idea, read papers, form a committee, write a proposal, defend the proposal, gather the data, analyze it, write about it, and compile everything into a final report that you then defend, again. No wonder people lose interest or decide they hate the project before it’s all over. My dissertation was no different, although I never grew to hate it.

I’m trained as a social psychologist, and even though much of my field was moving data collection online by the time of my dissertation, I wanted to conduct studies that drew upon the classic paradigm of the field: high-impact, laboratory experiments that use deception and research confederates (people secretly working with the research team) to manipulate the situation participants experience and measure its effect on behavior. So that is what I did.

The goal of my dissertation was simple. I wanted to test how much of people’s reaction to finding out they have some prejudice is a result of social norms. Specifically, I gave people a well-known test of implicit prejudice—the Implicit Association Test—and then provided them with false feedback indicating they held a moderate preference for White people over Black people. The question was: what happens next? Do people get defensive? Do they acknowledge the result? And can you change how they respond by shifting the social norms around admitting bias?

A Year in the Lab

To answer these questions, I ran two experiments. The first used a video of prominent researchers (some of whom created the test) discussing their own implicit biases to establish a norm of openness. That study was relatively straightforward to run; the second study was not.

In my second experiment, each participant arrived at the lab and was paired with another person—someone they believed was a fellow participant but who was actually a trained confederate working with my research team. Both the participant and the confederate took the implicit bias test, and both received the same false feedback. Then came the manipulation.

The experimenter asked the confederate to describe their results first, and depending on the condition, the confederate either openly acknowledged their bias, expressed skepticism and denial, or said nothing about the results (instead discussing whether the instructions were clear). The participant then responded, and we recorded everything.

It was the kind of study social psychologists are famous for: a controlled, deceptive, face-to-face manipulation of social influence in real time. It was also a logistical headache.

The confederates had to deliver their lines naturally every time. The experimenter had to stay blind to the condition until the last possible moment. We audio-recorded participants’ responses and later coded them for themes like genuine acknowledgment, skepticism, and self-directed disappointment. Sometimes the recordings failed. Sometimes the confederate made a mistake. Sometimes our undergraduate participants didn’t show up, or when they did, they were drunk or high (thank you, New Orleans and Mardi Gras!).

I collected data for a little more than a year. In the end, I had just over 400 usable participants across the two studies. While that number might sound respectable, consider the pace: I often ran multiple sessions per day, each requiring a trained confederate, an experimenter (me or a research assistant), space in the lab, and about 45 minutes of time. Some weeks we gathered 10 or 15 participants. In other weeks, the number was smaller. I had to (politely) fight for space in the lab and run sessions late at night and on weekends. I was willing to do anything to get the data.

I’m proud of my dissertation. The research showed that watching someone else acknowledge their bias made people significantly less defensive about the possibility they might have some bias too. But the work was slow, and it required a lot of resources. It also occurred when I was looking to finish my graduate education and move on with life. What came next gave me an even deeper appreciation for the high cost of laboratory data.

The Shift to Online Research Participant Recruitment

Before finishing my dissertation, I joined CloudResearch, a technology company that builds tools for online behavioral research. Even though the team I joined was small, it was poised for growth. The researchers I started working with routinely collected data from hundreds of people in an afternoon by launching their studies online. Projects that would have taken months in the lab were completed within hours. Science was sure to follow.

And that is what has happened over the last ten to fifteen years. Since 2010, social and behavioral scientists have gone from conducting very few studies online to having participants complete something like 5 billion surveys in 2025. Most of the researchers who use these tools take the change for granted, but it has been nothing short of a revolution in how scientists learn about people.

Thanks to CloudResearch, I’ve been at the forefront of these changes. My team not only studies issues like data quality and sampling, but we publish books, give talks, create tutorials, write blogs, host webinars, and teach people how to successfully use online tools. Why is so much education necessary, you may wonder. How hard can it be to set up an online study? The answer is: harder than it looks.

While the tools that facilitate online research are easy to use, the knowledge required to successfully gather online data is deceptively complex, often obscured by the simplicity of the tools themselves. Over the past 15 years, a network of online platforms has emerged to give researchers access to millions of potential participants across the globe. These platforms vary in how they operate, who they provide access to, and what kinds of research they support. Understanding this ecosystem—and knowing how to navigate it—is essential for anyone conducting behavioral research. If you don’t believe me, check out some of the horror stories, such as when researchers at the CDC falsely reported that 4% of Americans drank bleach during the COVID-19 pandemic, when researchers looking to recruit trans adults online wound up with few real participants but thousands of fraudulent ones, and when another group of researchers excluded 97% of the data they gathered.

My team at CloudResearch teaches people both the basics and the complicated stuff about online research.

The Online Participant Ecosystem: Panels, Aggregators, and Platforms

Online participant recruitment is larger and more complex than most researchers realize. Roughly 5 billion surveys are completed online every year, and most are facilitated by market research panels—companies that maintain large groups of people who have signed up to take surveys in exchange for compensation. These panels range from small operations with a few thousand members to massive companies with millions of participants worldwide.

At the broadest level, the ecosystem has three tiers. At the base are panels: companies like Prodege, TapResearch, and Toluna that recruit and maintain groups of survey-takers. In the middle are panel aggregators like Cint (formerly Lucid) and CloudResearch’s Prime Panels, which route studies to hundreds of individual panels simultaneously, giving researchers access to enormous and diverse populations. At the top are research services that bridge the gap between researchers and the panel infrastructure, handling the coordination and integration needed to make studies work.

The market research ecosystem is powerful. It offers extraordinary reach, the ability to target people of different demographics down to the zip code level, and rapid data collection at scale. You can sample internationally (beyond WEIRD people, as social scientists like to say) or hyper-locally. If you need 1,000 responses from women over 50 living in the southeastern United States, market research panels can deliver that.

But the system has limitations. The most serious one concerns data quality. Studies show that 40% or more of responses may be low quality or fraudulent. Participants are typically compensated at low rates, which means they don’t do well on long or complex tasks. Studies generally need to be under 20 minutes and longitudinal studies aren’t really an option beyond a few weeks to one month. Researchers cannot communicate directly with participants or implement studies that require much beyond a standard survey, experiment, or video interview.

To conduct more high-engagement studies, researchers turn to “researcher-centric platforms.”

Platforms Built for Researchers

As market research panels grew for market research, a different kind of platform emerged to meet the needs of behavioral and social scientists working in academia and other places. These platforms go by different names—crowdsourcing sites, microtasks, marketplaces—but what defines them is they put the researcher in control. Thus, they are also called researcher-centric sites.

Amazon Mechanical Turk (MTurk) was the first of these platforms and it was adopted almost by accident. MTurk’s original purpose was to connect people with an on-demand, scalable, human workforce to complete tasks that required human intelligence. In 2010 and 2011, enterprising behavioral scientists realized MTurk had everything needed to conduct research: direct access to a large pool of people willing to complete tasks online, a system for sending and receiving payments, and mechanisms for controlling data quality. Things took off from there.

What Is Amazon Mechanical Turk (MTurk)?

Amazon Mechanical Turk is an online marketplace where researchers can post tasks (“HITs”) for workers to complete. Originally designed for tasks requiring human intelligence (like image labeling), MTurk was adopted by behavioral scientists around 2010-2011 as a fast, affordable way to collect research data. It offers high researcher control but variable data quality, especially since 2018 when fraud became a significant concern.

What Is Prolific?

Prolific is a participant recruitment platform built specifically for academic research. Launched in 2014, it emphasizes data quality through participant vetting, fair compensation, and tools designed for behavioral scientists.

What Is CloudResearch?

CloudResearch is a research technology company offering tools for online behavioral research. Its products include Connect (a researcher-centric participant platform), Prime Panels (a market research panel), Engage (an AI-powered survey platform), and Sentry (a data quality tool). Founded in 2013, CloudResearch focuses on data quality and helping researchers run quality online studies.

Meanwhile, alternatives like CloudResearch and Prolific were launched in 2013 and 2014, respectively. These platforms were built by and for academic researchers, and as a result they have always emphasized data quality. They also have lots of tools that make conducting research easier than it is on MTurk.

Regardless of which researcher-centric platform you examine, all of them differ from market research panels in important ways. They typically vet participants more rigorously before allowing them to join, which can reduce fraudulent or low-quality responses to less than 5% compared to 30–40% on unvetted panels (MTurk is an exception to this). They give researchers more control over compensation, screening of participant demographics, and study design. And they support a wider range of research activities: not just surveys, but video interviews, real-time interactions, multi-session longitudinal studies, daily diary studies, dyadic data collection, content evaluation tasks, and even mock jury trials.

The trade-off is scale. Researcher-centric platforms generally have smaller participant pools than the aggregated market research ecosystem (a few hundred thousand people per year vs. tens of millions). If you need a very niche population—say, left-handed vegetarians over 60—you may need the reach of a panel aggregator. But for most behavioral research, the improved data quality and flexibility of researcher-centric platforms make them a strong choice.

How to Choose the Right Participant Platform

With so many options, choosing the right participant source is itself a research skill. The decision depends on several factors: the complexity of your study, the population you need to reach, your budget, your timeline, and how important data quality is relative to sample size.

Market research panels excel at rapid, large-scale data collection and the ability to target specific demographic groups. They’re well-suited for descriptive studies, polling, and market research where breadth matters more than the depth of individual responses. Researcher-centric platforms are better for experiments, multi-method studies, and projects where the quality of each response is critical. Some studies call for a combination of both, using a panel source for initial screening and a researcher-centric platform for the core study.

The key lies in understanding that not all participants are the same, and not all sources serve the same purpose. Understanding where your participants come from—how they were recruited, how they’re compensated, what motivates them to participate—matters for the quality of your data and the conclusions you can draw.

What You’ll Learn in Chapter 9

Chapter 9 of Research in the Cloud walks you through the online participant ecosystem in detail. You’ll learn the history of how participant recruitment moved online and why that shift matters. You’ll explore the structure of the market research industry—panels, aggregators, and research services—and understand how these pieces fit together.

You’ll also compare the strengths and weaknesses of market research panels and researcher-centric platforms across dimensions that matter for study design: data quality, participant engagement, demographic reach, longitudinal capability, and cost. By the end of the chapter, you’ll be equipped to make informed decisions about where to find participants for your own research and to evaluate new platforms and services as they emerge.

Whether you’re a student designing your first study, a researcher expanding into online methods, or a professional who needs to understand where behavioral data comes from, Chapter 9 gives you the map. The ecosystem is large, but it doesn’t have to be confusing.

This post is part of a series exploring the chapters of Research in the Cloud: An Introduction to Modern Methods in Behavioral Science by Aaron Moss, Jonathan Robinson, and Leib Litman. Want to learn how to find the right participants for your research? Read Chapter 9 for free here.