DECEMBER 28, 2017
Brett Wallace[1]
Introduction
Amazon Mechanical Turk is an online crowdsourcing marketplace founded by Amazon in 2005. The service, described as “artificial artificial intelligence”, provides micro-work in the form of atomized tasks that humans can perform better than computers. The name was inspired by “The Turk”, a mechanical chess playing automaton, created by Wolfgang von Kempelen in the 18th century. Kempelen toured Europe with his machine, telling audiences it was a form of artificial intelligence before it beat opponents in a game of chess. Before each game started, Kempelen would open the cabinet doors to reveal to the audience the intricate machinery and gears, which also cleverly hid a human chess master inside. This chess master controlled the life-sized wooden mannequin above.
Amazon came up with the idea for Amazon Mechanical Turk to solve its own internal data- processing problems, such as categorizing objects that required human intelligence. The company then decided to release Mechanical Turk as an external web service so that Requesters outside of Amazon could submit tasks. Mechanical Turk operates as a startup within the Amazon enterprise today. Mechanical Turk is different from how science fiction portrays scenes of humans asking questions to a machine to reveal hard-to-find answers. Mechanical Turk does just the opposite; the invisible crowd of workers completes tasks which machines are unable to accomplish yet.
Requesters visit Mechanical Turk to post Human Intelligence Tasks (HITs) to the diverse, always- on workforce. Workers (also commonly referred to as Turkers) select HITs to complete whenever they choose and collect a reward set by the requester for completion. The HITs are activities that humans can perform better than computers. Amazon states the use cases include “image recognition, audio transcription, machine learning algorithm training, sentiment analysis, data normalization, and surveys.”[2] At the time of posting this article, there were 842,000 HITs on the website.[3] There are eight examples hits posted in the FAQ on Mechanical Turk.[4]
For workers, Mechanical Turk provides the opportunity to earn money in their spare time, wherever and whenever they want to work. Workers register on the website as self-employed contractors, not Amazon employees, by signing a participation agreement. They do not benefit from employment laws regarding minimum wage, over-time and workers compensation. In this agreement, workers agree to interact with Requesters in a professional way and use their best judgement to perform HITs. They accept the HIT, complete the work and submit it. If the work is approved by a Requester, money is deposited into the Workers’ mTurk account, which they can then either disburse to their Amazon Payments account or their Amazon.com gift balance. If their work is rejected and therefore unpaid, Workers can send an email to the Requester to ask why, but the Requester does not have to answer, and Amazon will not mediate any such dispute. All Workers in the US and some in India can disburse their Amazon Payments funds to their bank account, while all other international workers can only transfer earnings to an Amazon.com gift balance.
Mechanical Turk Demographics
To extend the many existing studies on the demographics on Mechanical Turk, I conducted my own survey of 600 Mechanical Turk workers. Workers were paid $1 per survey, which was projected to take four minutes to complete. This equated to $15 per hour, a premium wage on the platform. The survey instrument included 21 questions exploring demographics. The survey data will be made publicly available in CSV form shortly via the web. The survey explored –
· Who are Mechanical Turk workers?
· Where do they live?
· How much do they make?
· What are the most and least gratifying parts of work on Mechanical Turk?
The survey data showed that the majority of workers (72%) are between 21-39 years old.
55% are male and 45% are female.
More than half of the workers (52%) hold a bachelor’s degree or higher degree.
Most of the workers are employed.
Many more workers are Turking part time (60%), while 31% were Turking as a full-time job. The full-time Turkers can be considered professionals, also known as Super Turkers[5]. These professional Turkers engage in online community sites, such as Turkopticon and Turker Nation, to share best practices and custom-built tools which boost their productivity. These websites help crowd workers watch each other’s backs, preventing new workers from being hurt by shady employers. There are a large number of professional Turkers in my survey because these workers gravitate towards high paying HITs; and they also are able to grab HITs quickly, meaning more novice workers never even seen them.[6]
In terms of income, 87% of Turkers are making less than $75,000 USD per year.
Most Turkers (55%) are making $500-5,000 per year completing HITs. Workers self-report earning on average about $4-5 per hour[7]. According to Pew Research Center, most of the HITs (61%) offer very little pay, under 10 cents per HIT.[8]
Most of the workers (76%) are new to the platform and have been Turking since 2014.
Most workers (55%) are earning between $10-40 per day. The majority of Turkers report that the one thing they would fix on Mechanical Turk is the pay grade. A new task-level analysis revealed that workers earned a median hourly wage of only ~$2 per hour and only 4% earned more than $7.25 per hour.[9]
58% of workers are working 4 hours or less per day, supporting the idea that this is mostly part- time work. The other 42% of workers are making it a more significant part of their work life.
The vast majority are Turking in the morning.
Most Turkers (67%) complete less than 50 HITs per day.
According to surveys conducted on the mturk-tracker tool, in the last 2 months 0f 2017, 75% of the workers who responded live in the U.S. Another 18% live in India, while 7% live in other countries.[10]
In 2012, due to complaints about poor work quality from International IP addresses, Amazon blocked workers outside the US for a period of time.[11]
I asked workers what they found was most gratifying about this work. The responses can be bucketed into three areas:[12]
1. working from home on their own schedule
2. making extra money to pay bills
3. learning new things
I also asked workers what they found was least gratifying about this work. The responses can be bucketed into three areas:
1. low-paying tasks
2. no way to file an appeal against Requesters who reject work or do not pay
3. trying to find tasks that are worth doing – which equates to unpaid work
Conclusion
My analysis shows that Amazon Mechanical Turk is a work haven for early to mid-career professionals looking for extra income to make ends meet. 80% of workers are coming to the platform from the United States and 96% of those workers are making below the United States Federal minimum wage.[13] This is because there are no enforceable rules around wage rates and most of the HITs are small, short-burst tasks, that pay less than 10 cents. There are opportunities for Amazon to help Requesters understand how to set wages in order to treat workers more ethically, perhaps in the form of a wage calculator. There are also opportunities for Workers to be able to identify themselves as professionals in order to build relationships with Requesters and therefore, through increased communication, receive more clarity about the work to be completed and have the ability to contest rejected HITs.
[1] Brett Wallace is an American conceptual artist exploring the future of work.
[2] From a software engineer job description for Mechanical Turk.
[3] There were 842,000 hits on the Mechanical Turk website on Dec 11, 2017.
[4] Example HITs from the Mechanical Turk website.
- Select the correct spelling for these search terms
- Is this website suitable for a general audience?
- Find the item number for the product in this image
- Rate the search results for these keywords
- Are these two products the same?
- Translate a paragraph from English to French
- Choose the appropriate category for products
- Categorize the tone of this article
[5] Fort, K., Adda, G., & Cohen, K. B. (2011). Amazon Mechanical Turk: Gold Mine or Coal Mine? Computational Linguistics, 37(2), 413–420. https://doi.org/10.1162/COLI_a_00057; Bohannon, J. (2011). Social Science for Pennies. Science, 334(6054), 307–307.
[6] Fort, Adda, & Cohen, 2011
[7] This data point is supported by over 50 testimonials and 100 videos sent into me by Mechanical Turk workers. Data also shows that a task worth $1 takes an average completion time of 12.5 minutes, resulting in $4.8 per hour wage. Analyzing the Amazon Mechanical Turk Marketplace, P. Iperiotis, ACM XRDS, Vol 17, Issue 2, Winter 2010, 99 16-21.
[8] Research in the Crowdsourcing Age” Pew Research Center.
[9] A Data-Driven Analysis of Workers’ Earnings on Amazon Mechanical Turk. Kotaro Hara, Abi Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, Jeffrey Bigham.
[10] Analyzing the Amazon Mechnical Turk Marketplace, P.Ipeitoris, ACX XRDS, Vol 17, Issue 2, Winter 2010, pp 16-21
[12] These answers were supplemented with 100 video testimonials I also gathered in Fall 2017.
[13] A Data-Driven Analysis of Workers’ Earnings on Amazon Mechanical Turk. Kotaro Hara, Abi Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, Jeffrey Bigham.