Ensuring Data Quality in Online Experiments: The Role of Attention Checks

Key Takeaways

• Online experiments face a data quality challenge: Unlike physical labs, researchers cannot control whether participants are distracted by other activities during the study.
• Inattentive participants create noise: Their responses create misleading results that can mislead researchers.
• Attention checks are necessary but should be minimal: Research shows that even a single, well-designed attention check can effectively separate attentive from inattentive participants.
• Task-specific attention checks work best: Questions tailored to the specific experiment (like asking about observed payoff ranges) are more effective than generic attention tests.
• Platform choice matters for data quality: While MTurk showed 30-50% inattentive participants, the CloudResearch Connect platform achieved 85-90% attention check pass rates with higher overall data quality.

Online cloud-sourcing platforms provide tremendous advantages in experimental data collection – among other things, collecting data is fast, cheap, and far more diverse than was previously possible with physical lab “convenient samples”. Yet, online data collection faces its own challenges – perhaps the most pressing one is ensuring data quality. While in physical labs the experimenter maintains almost full control over subjects’ attention—at least in the sense of whether the subjects are physically engaged with the experimental task—there is usually no such control in online labs. 

The Challenge of Attention in Online Experiments

For example, while in a physical lab it is quite easy to ensure that participants are not scrolling on their phones or engaging in other activities instead of actively participating in the task, it is very challenging with online experiments (usually, we cannot say for sure whether the participants were not simultaneously doing other tasks – such as talking with someone, listening to music, or engaging in other unrelated activities). Inattentive participants pose a serious challenge because they tend to behave more randomly, thereby significantly increasing the amount of noise in the experiment, which can lead to erroneous conclusions.

A Thought Experiment: The Cost of Noise

To illustrate this point, consider the following thought experiment: participants are required to pick a number between 1 and 6 – this number represents the amount of dollars they will receive as their payment. While it is safe to assume that most of us prefer a higher salary to a lower one and would choose “6”, if inattentive participants choose completely at random the mean of their choices will be 3.5. This can lead us to very strange conclusions – perhaps that participants are altruistic, have strong other-regarding preferences, or possess motives other than money when participating in experiments – for example, participating out of curiosity or a drive to advancing our knowledge. We could go on writing a lengthy paper that is most probably based on… noise! 

Attention Checks: A Common Yet Imperfect Solution

One common practice to ensure data quality is to include attention checks (hereafter ACs) during the experiment and include in the data analysis only those participants who have passed the ACs. However, incorporating ACs has its own drawbacks as well. First, including numerous ACs can irritate genuinely attentive participants; they are also costly in terms of time. Another important concern is the validity of excluding participants based on attention: one might argue that laboratory experiments are meant to represent reality rather than a sterile situation. In the real world, especially nowadays when a tremendous number of different stimulus compete for our attention, we are rarely fully attentive—so why should we be in experimental tasks? 

Investigating Attention Checks in the Decision-from-Experience Paradigm

Together with Ofir Yakobi, we set out to investigate the necessity of attention checks in experiments based on the decision-from-experience paradigm. In a typical decision-from-experience study, subjects observe a number of boxes on the screen and repeatedly choose between them. Each box (or button) is associated with a certain payoff distribution, usually unknown to the participants. After choosing (by clicking) one of the boxes, the participants observe both their obtained payoff and the payoffs of the other, unchosen alternatives. Our studies focused on a “checking paradigm” – a variant of the decision-from-experience task. Very briefly, one of the buttons, which can be thought of as a search agent, revealed the highest payoff among all the alternatives in each trial but at a cost (a search agent fee). In some conditions, when the search agent cost was low, using the search agent was beneficial, while in others (when the search agent cost was high) it was detrimental. We were mostly interested in the following questions:

1. Whether inattentive participants would behave differently from attentive participants in terms of their checking rates
2. How their behavior would compare to that of physical lab participants. 

Key Findings from MTurk-Based Study

In a recently published paper, based on an MTurk sample, we found that inattentive participants behaved more randomly than attentive participants. Furthermore, while inattentive participants significantly differed from physical lab participants, attentive participants were indistinguishable from the physical lab participants in terms of their checking rates. Importantly, even one AC was enough to separate the attentive from the inattentive participants. It seems that the most efficient attention test was one specifically tailored to the investigated task (e.g., asking about the range of the observed payoffs) at the end of the experiment. Still, one major drawback was the high percentage of inattentive participants (between 30% to 50%, depending on the task and quality restrictions on MTurk). 

Replication Using CloudResearch’s Connect Platform

Thanks to generous support from CloudResearch, we were able to replicate our findings using the highly convenient Connect platform. The results are encouraging – first, we were able to replicate all the main findings mentioned above. Importantly, using the Connect platform seems far more efficient, as between 85% to 90% of the participants passed the attention check (and, as a side benefit, it also provides a convenient interface to communicate with participants). 

To conclude, online platforms provide a convenient, cost- and time-effective way to run experiments. To ensure data quality, it is necessary to include attention checks; however, at least in decisions from experience, it seems that even one task-tailored attention test is enough to obtain responses that are indistinguishable from those of physical lab participants. Finally, the Connect platform offers a user-friendly interface with a large, yet still high-quality subject pool, which reduces the number of inattentive participants and ensures high data quality.