Amazon’s Mechanical Turk crowdsourcing task marketplace has changed social science research over the past 10 years. Originally designed as a distributed human workforce for small tasks that cannot be automated, such as complex categorization or labeling, the platform is now creatively used for a variety of tasks, including survey research.
Some sources give Mechanical Turk credit for creating a golden age in survey research. Others cite major drawbacks and ways to mitigate some of those problems.
Due to financial constraints, academics have traditionally relied on convenience samples, e.g. college students surveyed via clipboard on campus. Mechanical Turk represents a major step up over this methodology - reaching farther into the general population for $0.10 per survey complete.
“We show that respondents recruited in this manner are often more representative of the U.S. population than in-person convenience samples—the modal sample in published experimental political science—but less representative than subjects in Internet-based panels or national probability samples”
|Mechanical Turk||Internet Panels||Phone Polling||TapResearch|
|Demographic profiling & targeting|
|Entry census balancing|
Feasibility on small targets
(e.g. minority teens)
|Throttling / pacing|
|Fresh respondents every day|