By Cheskie Rosenzweig, MS, Josef Edelman, MS & Aaron Moss, PhD
Attention check questions are frequently used by researchers to measure data quality. The goal of such checks is to differentiate between people who provide high-quality responses and those who provide low quality or unreliable data. Although attention checks are an important component of measuring data quality, they are only one of several data points researchers should use to isolate and quarantine bad data. Others include measuring inconsistency in responses, nonsense open-ended data, and extreme speeding.
A good attention check question has a few core components:
In some studies researchers ask participants to think back to an earlier part of the survey and recall specific information previously presented to them. This kind of question certainly has something to do with attention, but it also is measuring memory capacity, and the difficulty level of this recall is unclear.
Please think back to the story you read earlier in this survey. How many oranges did Steven have before trading some with Josephine?
a. 1 b. 2 c. 3 d. 4
Sometimes an attention check isn’t overly difficult because of the content it is asking participants to provide or recall, but because it is designed to only be answerable by participants who are highly attentive to every minute detail and are conscientiously reading and following every single word in a survey. Questions like these end up excluding participants who are otherwise high-quality respondents, but are not extraordinarily attentive and conscientious:
Most modern theories of decision making recognize the fact that decisions do not take place in a vacuum. Individual preferences and knowledge, along with situational variables can greatly influence the decision making process. In order to facilitate our research on decision making, we’re interested in knowing certain factors about you, the decision maker. Specifically, we are interested in whether you actually take time to read the directions; if not, then some of our manipulations that reply on change in the instructions will be ineffective. So in order to demonstrate that you have read the instructions, please select “Reading Instructions” from the list below, instead of any other hobbies that you may have.
Which is your favorite hobby?
Attention check questions should not violate expectations of natural human behavior. Checks shouldn’t require participants to ignore direct questions that are being posed to them that appear straightforward. By making questions that participants need to go against their instincts to answer correctly, you’re left with attention checks that are overly difficult. With such questions, researchers may end up categorizing good respondents who are paying attention and acting as normal humans as low-quality, inattentive respondents who are providing terrible data. This can bias the sample of qualified respondents and can impact the conclusions you reach in your research.
Here is another example of a bad attention check. Researchers may intuitively believe that a question does a good job discriminating between attentive and inattentive respondents, when in reality, the question does not have a clear enough answer.
I am interested in pursuing a degree in parabanjology
a. Strongly Disagree b. Disagree c. Strongly Agree d. Agree
While as you may have guessed the field of “parabanjology” does not exist, this does not mean that anyone who says they are interested in “parabanjology” is not paying attention to your survey. In fact, research revealed that up to 41% of people who are being attentive and thoughtful can still fail this question (Curran & Hauser, 2019).
The above examples are reasons why we recommend that researchers use attention check questions that have been previously tested and validated as good checks of attention. When researchers create attention checks based on personal intuition, the checks they use often lack validity (see Berinsky, Margolis, & Sances, 2014).
Rooted in these best practices in developing attention checks and other measurements of data quality, CloudResearch.com has undertaken several large scale initiatives aimed at improving data quality in online surveys. We have cleaned up data on one platform often used for online research: Mechanical Turk. We also have data quality solutions that can be applied to any and all sample sources: our SENTRYTM system. SENTRY is an all encompassing data quality solution that ensures top notch respondents through multiple behavioral and technological analyses, including attention checks. Sign up for a webinar or learn more about SENTRY today!