Book:
Litman, L., Robinson, J. (2020). Conducting Online Research on Amazon Mechanical Turk and Beyond. April, 2020. SAGE Publications. https//us.sagepub.com/en-us/nam/conducting-online-research-on-amazon-mechanical-turk-and-beyond/book257367
A guide to the world of online research and how to optimally carry out research projects with online samples.
Peer-Reviewed Publications:
Litman, L., Robinson, J., & Abberbock, T. (2017). TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences. Behavior Research Methods, 49(2), 433-442. https://link.springer.com/article/10.3758/s13428-016-0727-z
A description of the purposes and features of the CloudResearch MTurk Toolkit.
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019). Online panels in social science research: Expanding sampling methods beyond Mechanical Turk. Behavior Research Methods, 51(5), 2022-2038. https://link.springer.com/article/10.3758/s13428-019-01273-7
Study examining data quality and participants representativeness of Prime Panels as a participant recruitment platform.
Robinson, J., Rosenzweig, C., Moss, A.J., Litman, L. (2019) Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool. PLoS ONE 14(12): e0226394. https://doi.org/10.1371/journal.pone.0226394
Analysis of the size of the MTurk participant tool, and suggestions for how to apply better sampling strategies to reach less experienced high-quality participants.
Moss, A. J., Rosenzweig, C., Robinson, J., & Litman, L. (2020). Is it Ethical to Use Mechanical Turk for Behavioral Research? Relevant Data from a Representative Survey of MTurk Participants and Wages. Manuscript submitted for publication. https://psyarxiv.com/jbc9d/
Exploration of MTurk workers’ views of MTurk, satisfaction with requesters, and hourly wages.
Moss, A. J., Rosenzweig, C., Robinson, J., & Litman, L. (2020). Demographic Stability on Mechanical Turk Despite COVID-19. Trends in Cognitive Sciences. https://www.cell.com/trends/cognitive-sciences/pdf/S1364-6613(20)30138-8.pdf
Commentary on the stability of participant demographics on MTurk from Jan. 2019 through May 2020, a time frame that includes the first three months of the COVID-19 pandemic.
Litman, L., Robinson, J., & Rosenzweig, C. (2015). The relationship between motivation, monetary compensation, and data quality among US-and India-based workers on Mechanical Turk. Behavior Research Methods, 47(2), 519-528. https://link.springer.com/article/10.3758/s13428-014-0483-x
Examination of the impacts of compensation on data quality on MTurk.
Fordsham, N., Moss, A. J., Krumholtz, S., Roggina, T., Robinson, J., & Litman, L. (2019).Variation among Mechanical Turk Workers Across Time of Day Presents an Opportunity and a Challenge for Research. Manuscript submitted for publication. https://psyarxiv.com/p8bns/
How participant demographics and clinical symptomatology differ across time of day, and best practices for recruitment.
Litman, L., Robinson, J., Rosen, Z., Rosenzweig, C., Waxman, J., & Bates, L. M. (2020). The persistence of pay inequality: The gender wage gap in an anonymous online labor market. PloS ONE 15(2): e0229383. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0229383
Description of the gender pay gap on MTurk and suggestions for how to pay equally.
Litman, L., Robinson, J., Weinberger-Litman, S. L., & Finkelstein, R. (2019). Both intrinsic and extrinsic religious orientation are positively associated with attitudes toward cleanliness: Exploring multiple routes from godliness to cleanliness. Journal of Religion and Health, 58(1), 41-52. https://link.springer.com/article/10.1007/s10943-017-0460-7
Best practices for sampling specific groups of individuals on MTurk.
Litman, L., Williams, M. T., Rosen, Z., Weinberger-Litman, S. L., & Robinson, J. (2018). Racial disparities in cleanliness attitudes mediate purchasing attitudes toward cleaning products: A serial mediation model. Journal of Racial and Ethnic Health Disparities, 5(4), 838-846. https://link.springer.com/article/10.1007/s40615-017-0429-y
Best practices for accurately sampling minorities.
Schnur, J. B., Chaplin, W. F., Khurshid, K., Mogavero, J. N., Goldsmith, R. E., Lee, Y. S., … & Montgomery, G. H. (2017). Development of the Healthcare Triggering Questionnaire in adult sexual abuse survivors. Psychological Trauma: Theory, Research, Practice, and Policy, 9(6), 714. https://europepmc.org/article/PMC/5659978
Collecting open-ended data for scale development using online participants.
Litman, L., Rosen, Z., Spierer, D., Weinberger-Litman, S., Goldschein, A., & Robinson, J. (2015). Mobile exercise apps and increased leisure time exercise activity: A moderated mediation analysis of the role of self-efficacy and barriers. Journal of medical Internet research, 17(8), e195. https://www.jmir.org/2015/8/e195/
Explored how health behaviors of MTurkers conform to expected results and explored new behavioral health models relating to the relationship between the use of exercise apps and health.
CloudResearch Blogs and Other Resources:
Clean Data: the Key to Accurate Conclusions
New Solutions Dramatically Improve Research Data Quality on MTurk
Best Practices That Can Affect Data Quality on MTurk
Should I Reject This MTurk Worker? How to Fairly Identify Low-Quality Research Participants
Are Polls in Pennsylvania Still Missing “Shy Voters”?
Study: Are Election 2020 Poll Respondents Honest About Their Vote?
Conducting Online Research During COVID
What Is It Like To Participate In Online Research On Amazon Mechanical Turk?
The Superworker Sampling Bias on Mechanical Turk
Data quality on MTurk compared to other online panels
The size of the MTurk population
Running Longitudinal Studies on CloudResearch
The Gender Pay gap on MTurk and how to address it
Anonymize Mechanical Turk Worker IDs
A Guide to Statistical Significance
A Guide to Market Segmentation