Problem:
How can you increase Amazon Mechanical Turk HIT Worker participation rates and speed completion of a HIT? This is particularly an issue with HITs that have a large number of required participants or have Qualifications that limit the number of qualified Workers
Solution:
By monitoring the participation rates of hundreds of HITs we have observed the following patterns that increase participation significantly:
- High Turkopticon Ratings. Turkpticon is a Worker forum that rates Requesters on metrics like pay rate, communication, speed and fairness and allows comments. These reviews are instantly visible to Workers on the Amazon MTurk web page via a browser plug-in. Workers shy away from low rated Requesters and gravitate towards high rated Requesters. We have observed HITs run by low rated Requesters that increased their completion rate by an order of magnitude just by increasing their Turkopticon scores.
- Limiting Qualifications. A highly restrictive qualification requirement will limit the pool of workers significantly. Recently a Requester ran a HIT open only to workers from Indonesia. That HIT received barely any responses. It has been well documented that MTurk workers are not available in large numbers outside the United States and India.In contrast, requiring workers to have a very high approval rating (>95%) and many completed HITs (>10,000) does not seem to significantly impact participation. We recently ran a survey with even more restrictive qualification requirements that received 500 completes in a matter of days. Of course, with more lenient qualification requirements the HIT would have reached 500 completes in a matter of hours.
- Worker Pay. The amount paid to workers has been documented to encourage workers to participate greatly, US workers in particular will avoid low paying HITs and those HITs will take much longer to complete, if at all. We recommend paying at least $0.10 / minute especially if your HIT has stringent qualification requirements. See this discussion how you can predict time needed. If you need to increase the worker pay of a slow moving HIT you can do that within TurkPrime (see a discussion here) while your HIT is running (which is not possible via the Mechanical Turk Requester web site)
- Age of HIT. Newly launched HITs fill faster than older HITs since they have greater visibility on the Mechanical Turk Worker web site. An old HIT with diminished viability will complete much slower. A HIT can be “Restarted” and gain the viability of a new HIT by using the “Restart” feature in TurkPrime, discussed here. We recently ran a number of HITs whose participation slowed to a trickle and would be unable to reach the required participation goal. However, after the HITs restarted they handily reached their goal shortly thereafter.
- Social Networking. Requesters can notify Workers of their HIT’s availability via social networking sites like reddit and turkernation, among others
- Notify Workers. Workers can be notified of HITs they qualify for it they have taken a HIT with you in the past. You can notify (via email sent by Amazon) by using the TurkPrime Notify Workers feature.
