If you’re a researcher you’ve been there. You and your team have worked hard on a project for weeks—designing the study, gathering the data, analyzing the responses, and polishing your insights. Then, the moment comes when you need to hand your work to other stakeholders or share it with the world. You take a deep breath and click submit.
Most of the time, things work out fine. After all, you’ve worked hard to dot your i’s and cross your t’s. But, as a professional, you always dread WHAT IF? What if things don’t work out fine? What if there is a massive flaw with your work?
Unfortunately, people’s work sometimes has flaws and it sometimes contains critical mistakes—we are all human after all. But considering the time and money research requires, a mistake you should never make is gathering bad data because you or your sample supplier failed to vet respondents properly. Data quality is a known issue within online panels, and the stakes of not properly vetting respondents are just too high.
You probably don’t need to be reminded, but things were scary early in the COVID-19 pandemic. The virus was new, scientific evidence was scant, public policy was inconsistent, and there were no effective treatments or vaccines. Amid rising infections and deaths, people didn’t know how to keep themselves safe besides avoiding others.
Worried about what actions people may take to ward off COVID-19, the CDC released a report in June 2020 titled “Knowledge and Practices Regarding Safe Household Cleaning and Disinfection for COVID-19 Prevention.” The results were alarming. According to the study, “39% of Americans engaged in at least one high-risk behavior during the previous month.” These behaviors included using bleach to wash food items (19%), using household cleaning products on the skin (18%), and drinking or gargling diluted bleach, soap, or other disinfectants (4%).
As you might expect, these findings garnered a lot of media attention. Within days, hundreds of news outlets worldwide reported that Americans were doing things like drinking and gargling bleach to avoid COVID-19. Although the news traveled fast, the findings were false.
When the CDC gathered their data, they contracted with a market research vendor. The vendor sourced respondents from a common sample supplier but failed to vet respondents to ensure they answered honestly and that problematic respondents did not muck up the data. We know this because CloudResearch replicated the CDC study. We obtained the same results using the same questions and sample suppliers when we gathered the data without vetting respondents. When we vetted respondents with our patented Sentry® system, however, the data told a different story.
Respondents vetted with Sentry reported far fewer dangerous cleaning practices than those who were not vetted. In most cases, the gap between vetted and non-vetted respondents was close to 20%. Further, for the most high-risk behaviors of drinking or gargling dangerous cleaning solutions, the percentage of people saying “yes” almost disappeared among vetted respondents.
On its face, the idea that almost 40% of Americans are dangerously using cleaning products is a bit hard to believe. The data CloudResearch gathered show why: the numbers were inflated by inattentive respondents. Inattentive respondents answer randomly and say “yes” to statements like “I eat concrete for breakfast due to its iron content.” Not coincidentally, these are the same respondents who reported gargling bleach in our studies.
There are multiple, negative societal consequences of the CDC’s messy data. At a minimum, such false reports risk damaging the CDC’s reputation and introducing professional embarrassment among the researchers who gathered the data but failed to vet it properly. More importantly, the CDC recommended costly public health interventions to combat a problem that did not exist. The societal implications of this false CDC report were discussed in a recent article in Harvard Business Review.
As unpleasant as it is to think about the professional embarrassment that can come from suspect data, the good news is there are steps you can take to prevent these sorts of mistakes. Three steps you can take are to: add data quality measures to your survey, ask your sample provider for evidence of what they’re doing to deal with these issues, and consider a tool like Sentry.
The research you do is too important to get wrong. When respondents provide data that is incorrect, unreliable, or inaccurate and these issues go undetected it can create a precedent that you and your organization are not a credible source. Taking these three simple steps can help you avoid the consequences of bad data quality.