What is MTurk
Amazon Mechanical Turk (or MTurk) is a popular tool to use for recruiting human subjects for social and behavioral research. MTurk is known as a crowd-sourcing marketplace that provides businesses and researchers (the “requesters”) with the opportunity to collect information from a large number of people (the “workers”) in an efficient and cost-effective manner. Data is collected using a selection of thousands of online human intelligence tasks (HITs) using simple surveys that last a few minutes to more complex experiments lasting over an hour.
How to use MTurk
Discover what MTurk can do for you by visiting the below sites and reading about MTurk features:
MTurk in Social/Behavioral Experiments
MTurk allows researchers to have access to a large population of willing participants for research studies. Researchers can generate a HITs that gives MTurk users a title and description of the online task. It also states the amount of compensation users will receive and the amount of time it takes to complete. These tasks can also serve as a recruitment site providing links for online survey’s that are housed at different online survey software sites (i.e. – Qualtrics, Survey Monkey, Google Forms, etc.). MTurk allows by having a third party (Amazon) to oversee payments (compensation) to participants.
When creating the Kuali IRB application describing the use of MTurk in a study, consider the following:
- include the title and all relevant HIT description in the Kuali application. This allows the reviewer to understand the purpose and procedures of the research. Be sure the researchers name and/or school affiliation is listed either as the Requester or in the description of the HIT
- be clear about compensation and bonuses. Describe how long it will take for the researchers to approve their HIT (i.e. – how long it takes for MTurk workers “turkers” to receive payment)
- clearly and accurately state the time required to complete the task. For instance, if the task involves completing a survey, writing, or watching videos, this should be stated in the description. Include a link to the online survey in the HIT description (and attach a copy of the survey to the Kuali Protocols application). Also be aware that certain types of tasks, such as writing tasks, elicit higher compensation
- inform participants should be told if there are screening questions required in order to qualify and if participants are being paid for the time it takes to complete the screener
- list qualifications for participation in the HIT description
- understand the implications of rejecting HITS. Rejections remain with MTurk workers forever, negatively impacting their ability to get future work on MTurk. Avoid arbitrary rejections or rejecting work in error. If rejections are made in error, work with MTurk to promptly correct these mistakes.
- if any extraneous software is required to complete the task, this should be stated in the description (tasks requiring javascript or inquisit)
- INCLUDE the consent information. The first page of the online survey should be the consent document. The online consent will have all of the elements of a regular consent, but it will not require a signature. Participants will either click an “I Agree” or an “I do not Agree” box. The “I Agree” box will take them into the survey. The “I do not agree” box will thank them for their time and take them away from the survey. The template for the online survey is found here. Exemption determination applications do not require a need for a request for a Waiver of Written Consent documentation but applications involving additional risk do. Pay attention to the application instructions for additional detail.
- if the researchers are using deception or incomplete disclosure (i.e. – are not stating exactly what the study is about so as not to bias participants responses), then it is important to include a debriefing form at the end of the survey. This debriefing form could be embedded into the last page of the survey and would require participants to answer a final question allowing researchers to use their data (or not use their data) now that they know the true purpose of the study. The IRBs guidance on debriefing in found here.
- Confidentiality. Participation of turkers is anonymous to academic researchers, but in reality, anonymity cannot be guaranteed in any online environment where data is being collected. Minimize the collection of Personally Identifiable Information (PII) and do not collect IP addresses or worker IDs. MTurk worker IDs can easily be linked to individuals Amazon profiles including individuals wish lists and previous product reviews. This means that researchers must be careful in deciding what information to collect from participants. The default should be that participants MTurk worker IDs not be collected. If it is necessary to collect worker IDs, then the researchers should ensure that worker IDs are kept confidential and secure, are not linked back to survey data, and are deleted after use. Pay attention to the MTurk Terms of Service as it relates to the collection of participants online behavior and history (through the use of cookies or other tracking systems) and the selling of participants data to third parties.