Is Online Data Collection Affected by Time of Day Bias?
By Tommy Roggina & Aaron Moss, PhD
Behavioral researchers from both corporations and universities have turned to the Internet as a fast and efficient way to find research participants. The Internet allows researchers to administer surveys, polls, and experiments exponentially faster than traditional methods such as mail surveys, telephone interviews, or bringing participants in person. Although the speed of online data collection has primarily been celebrated as a strength, a new study from CloudResearch suggests that too much speed may sometimes lead to time of day bias.
Time of day bias occurs when the people who are available to take an online study in the morning are substantially different than the people available to take a study in the evening. Our research found that people who were active in an online participant platform in the morning had a greater circadian preference for mornings than evenings.
Further, people active in the morning were more conscientious and less anxious, depressed, neurotic, prone to Internet compulsion, and disruptive sleep patterns than people active in the evening. All these findings are in line with years of research in the biological and psychological sciences and demonstrate a potential way that psychological and personality differences between online respondents may skew behavioral survey results – unless a researcher controls for time of day bias.
How Can You Control for Time of Day Bias?
Findings from our recent studies suggest there are instances in which researchers may want to slow down data collection in order to control the time of day, day of the week, or the local time zone in which participants complete a study. Changing sampling practices around the time when a study is launched may help researchers avoid biasing their study by participant age, conscientiousness, anxiety, level of depression and other characteristics known to vary with a preference for mornings and evenings.
At CloudResearch we have a variety of tools that make controlling the timing of your data collection possible.
How to Control for Time of Day Biases with CloudResearch:
Strategy 1 – Slow down the rate of data collection
The easiest way to control for time of day bias is to slow down data collection. When running MTurk studies with the CloudResearch MTurk Toolkit, researchers can use the MicroBatch feature to spread out data collection. MicroBatch sets a delay between when successive batches of your study are launched, slowing down data collection and mitigating time of day bias. To use the MicroBatch feature, simply select ‘Microbatch’ during study setup.
The speed of your data collection can be customized by selecting how often new participants are allowed to enter your study. You can specify how long to delay between batches of your study using the “Auto Restart Time in Minutes” field.
While this is the most convenient and automated way to mitigate time of day bias, it provides only limited control over the speed of data collection. To better account for the differences between online participants at different points in the day, researchers can actually sample workers at different times of day.
Strategy 2 – Collect data at different times throughout the day
Another way to mitigate time of day bias is to create multiple HITs that are saved and set to launch at different times throughout the day. To set a HIT launch time, researchers can use the scheduled launch time feature on Tab 4 of the study setup process.
When scheduling a time for multiple studies to launch, you can also set a time for HITs to automatically close in order to control when participants can take your study. In the same Setup HIT and Payment tab, scroll to ‘HIT expires in’ and set the amount of time the HIT should be open for participants.
Lastly, to ensure that there are no duplicate participants in your study, add your HITs to a Survey Group. This feature prevents anyone who participates in one HIT in a Survey Group from participating in other HITs within that Survey Group. To add your HITs to a Survey Group, go to Tab 8 ‘Pro Features’ and scroll down to ‘Survey Group.’
Strategy 3 – Collecting data at different times throughout the day, while controlling for time zone
The best way to ensure time of day bias does not affect your study is to target workers in specific time zones as we did in the second study of our recent research. To create a study that controls for time of day bias across time zones, you can: (1) create a HIT that is set to launch and close according to each time zone, (2) add each HIT to a survey group, and (3) set each HIT to target participants from specific time zones.
To implement steps (1) and (2), follow the instructions provided for the study design above. To implement step (3), “set each HIT to target participants from specific time zones,” navigate to Tab 8, ‘Pro Features.’ Then, scroll down to ‘US Regions’ and select the area of the U.S. you would like to target according to each time zone (see below for a Time Zone Map).
Alternatively, CloudResearch’s MTurk Toolkit also allows you to target participants based on state-of-residence. Location targeting can be found in Tab 6, ‘Worker Requirements’.
CloudResearch’s tools allow you to incorporate many complex and interesting study designs. To learn more about our features and start planning your next study, see our Knowledge Base section on designing and launching projects. Further, if you are interested in running a complex study design of this nature, but would like CloudResearch to set up the project for you, see our Managed Research services.
