CloudResearch is Retiring the “Block Low-Quality Participants” Option

Starting November 1, 2021, CloudResearch will be retiring the Block Low-Quality Participants feature we created in 2020 and continuing with our CloudResearch-Approved Participants option. In this blog, we explain why we’re making this change and what it means for researchers.

Why Are We Removing the Blocked List?

Since Spring 2020, CloudResearch has been engaged in continuous vetting of people on Mechanical Turk. As participants pass through our vetting measures we pair their data with information provided by researchers and make a decision about how likely people are to provide quality data in future studies. People who we think will provide quality data are placed on our Approved List and those who are likely to provide poor data are blocked. 

The main reason we are removing the Block Low-Quality Participants feature is because our Approved List is now large enough to handle all studies. Originally, we were concerned that large studies, studies targeting specific groups of participants, or studies with other sampling limitations would be slow to gather data. This is no longer the case. The Approved List now has over 75,000 people and we add new participants each week. As a result, there are enough participants of various demographic groups and levels of experience that the Approved List can support all studies.

Another reason we are removing the Block List option is to better control quality. When researchers run a study using the Block Low-Quality Participants option, everyone CloudResearch has vetted and found to provide suspect data is ineligible for the study (currently over 40,000 accounts!). However, when a study is run using the Blocked List, CloudResearch doesn’t control what proportion of the sample is made up of people on our Approved List or people we’ve never vetted. Because people we’ve never vetted may provide poor data, the Blocked List gives us less control over quality than the Approved List which only includes people we have vetted.  

What Effect Will This Change Have on Research?

We do not expect the removal of our Blocked List to affect the viability or speed of research studies. There is also no change in cost to researchers.

As noted above, the Approved List now has over 75,000 participants, and over the last year and a half more than 33,000 studies have been conducted using the Approved List. Supplemental data in our latest preprint show that the demographics of people on the Approved List match those of the U.S. MTurk population well. Thus, we think of our Approved List as an option researchers can use to sample high-quality participants within the US and as a way to restore MTurk to what it was before 2018 when concerns about foreign workers, “bots,” and low-quality data emerged.