Blog
The Worst-Kept Secret in Market Research
Everyone in the room has known about the data quality problem for years. The only question is why it took this long to stop laughing about it.
A few years ago I was at SampleCon in California. Beautiful outdoor venue, the usual industry crowd milling around. I found myself in conversation with a sales executive from one of the larger panel companies — a smart, affable guy who clearly knew his business well.
We were talking about respondent quality. Specifically, whether people who claim professional credentials — doctors, lawyers, engineers — are actually who they say they are. I asked him directly what he thought.
He was mid-bite on a piece of sushi when he answered. He replied, more or less: “Look, none of these guys who say they’re doctors are actually doctors. They’re kids playing slither.io who want a new skin. They take the survey, get their points, and move on.”
He wasn’t whispering, or embarrassed. He was amused. A few others in the circle laughed along. And then the conversation moved on.
That moment stuck with me. Not because it was shocking — it wasn’t — but because of what it revealed about where this industry had landed. The problem wasn’t hidden. It was right there, acknowledged openly over sushi, in front of a small crowd of industry professionals. Nobody pushed back. Nobody looked uncomfortable. It was, apparently, just how things were.
Why nothing changed
The executive at that party wasn’t a bad guy. He was operating rationally inside a system designed to make honesty expensive.
Panel companies get paid on volume. If you implement serious quality controls and knock out 40% of your respondents for low effort, inattention, or false demographics, you’ve just cut your revenue by 40%. Your client still needs 10,000 responses. They’re not paying a premium for 6,000 cleaner ones. So the business calculation is straightforward: don’t look too hard.
The buyers have their own version of this. They need a deliverable. Their boss needs a report. Whether the underlying data holds up over the next year or two is a problem for later — and many of them won’t even be in the same role by then. The system selects for people who produce convincing outputs on schedule, not for people who ask uncomfortable questions about what’s underneath those outputs.
The result is an industry that has, for years, been collectively looking the other way. Openly. That conference conversation wasn’t unusual. I’ve had versions of it at nearly every industry event I’ve attended. It’s considered honest talk among people who know how things actually work. What’s remarkable isn’t that anyone says it — it’s that nobody seems particularly bothered by it.
AI didn’t create this. It just made it harder to ignore.
When the bot panic arrived — and it did arrive, loudly — something interesting happened. The industry suddenly had a vocabulary for alarm. Bots. AI agents. Synthetic respondents. The word “existential” started appearing in trade publications. Conferences held emergency panels.
The underlying concern is legitimate. A paper published in the Proceedings of the National Academy of Sciences demonstrated that AI agents can pose as human survey respondents and go completely undetected. That’s real and it matters.
But the data quality problem is not new. The fake doctor profiles are not new. The 40% inattention rate is not new. What’s new is that the problem got technical enough, and visible enough, that the usual response — a laugh and a shrug — stopped working socially.
If you weren’t worried when it was humans gaming the system, the honest question is what exactly changed — other than the problem now having a name.
The people most alarmed about bots right now are, in many cases, the same people who built platforms designed not to notice the humans who were already failing. If you weren’t worried when it was humans gaming the system, the honest question is what exactly changed — other than the problem now having a name.
What accountability looks like
Talking about data quality is easy. This industry has been producing whitepapers about it for years. What’s hard is building something that puts the question to a public test — with real stakes, adversarial scrutiny, and results that can’t be quietly filed away.
Introducing: the Bot Olympics. An open competition, in collaboration with MIT as independent judge, where attackers try to defeat leading detection systems and defenders try to stop them — with $50,000 on the line for whoever can prove our detection can be beaten at scale. And beyond the bots: an independent, public assessment of which platforms are actually supplying real, attentive, honest human respondents. That question has never been answered publicly, until now.
We’re not publishing a whitepaper about our confidence. We’re inviting anyone to prove us wrong, with the promise of real money if they succeed.
Why this matters beyond research
During the COVID pandemic, survey data circulated suggesting a disturbing pattern: a significant number of people reported drinking bleach as a health remedy. It was alarming. It drove attention, concern, and coverage.
It was also wrong. Our research showed it was an artifact of bad data — inattentive respondents clicking through without reading, low-quality panels producing noise that looked like signal. The slither.io kid the sushi executive laughed about, grinding through a survey to unlock a game skin, answering whatever gets him to the end fastest: that’s not a hypothetical. That’s a description of what actually happened. There was no bleach crisis. The data had manufactured one.
This was subsequently published in PLOS ONE and covered by the Harvard Business Review. I don’t raise it for the citations. I raise it because it’s a clean example of what’s actually at stake. Bad data in a research platform isn’t just a business problem — it’s a decision-quality problem. The people using that data to make health recommendations, policy decisions, and product launches don’t know it’s contaminated. They trust it because they paid for it. They publish on it. They act on it.
We’ve been treating data quality as an industry inconvenience for long enough. The Bot Olympics is our attempt to stop accepting the status quo and start building the accountability structure this industry should have built years ago.
Read more about the competition at cloudresearch.com/botolympics.
✦
Special thanks to Leib, Aaron, Theresa, Michael, Reuben, Yosef, and Meirah for their work building this initiative.