AI Product Research Series: Using Mechanical Turks for Quick Market Research
How many times have you had that sinking feeling that you can’t make an informed or defensible product decision without more information? Questions arise constantly throughout the entire product development process and so often there isn’t time or budget to conduct formal research. There are options if you need information quickly, including talking to a couple people within your target market, but surveys “are the norm” for us.
Surveys are quick and easy to generate (see basic tips in this article https://www.qualtrics.com/blog/10-tips-for-building-effective-surveys/) but finding a specific type of recipients quickly and without a material effort can be difficult unless you’re lucky enough to have a CRM full of existing clients of similar products that you can access and contact.
Amazon Mechanical Turk (MTurk, https://www.mturk.com/) is a great service to use when you need information quickly with minimal budget. MTurk is a crowdsourcing marketplace that connects people around the world willing to provide services with those in need of a particular service. You can post almost any type of job or activity, whether it’s completing a survey or translating text. The service makes it easy to identify the exact characteristics of respondents that you need, and you can set the compensation for completion at whatever you’d like.
Generally, the harder to find or higher level personas (VP’s and above) will require higher payment. If you need the general population to complete your survey, often ten cents or a quarter will be enough, but if you’re looking for VP’s of Marketing, as we were during our AI Marketing concept research, paying closer to $1 or more will help. The higher amount also tends to increase the rate at which the job is completed, so if you need a high number of responses quickly, increase the payment.
The service is not perfect and has its potential pitfalls. There is no way to ensure that the people responding to your surveys actually have the characteristics/background that you are looking for, so we recommend adding a question or two that helps you decipher this and also make sure the title of the job calls out the persona you're looking for.
There have been documented reports of bot responses and Turkers going through numerous surveys very quickly and responding incoherently to the questions. There are a couple of ways to reduce the risk and help ensure that your responses are legitimate:
Require answers to open ended questions. If you get incoherent responses, you know something is wrong and you you can either remove those that don’t look legitimate or cancel the survey.
Start with a small workforce or size, learn, then increase it. It's an iterative process and can take 2 - 3 adjustments before it's worth requesting a large number of respondents.
We received some valuable insight from open ended questions to our Marketing AI surveys and were reasonably sure we had VP’s of Marketing responding to our surveys. Below is an example of the type of insight we gained from open ended questions from our Marketing AI survey.
MTurks is best used for directional results. You can probably get a statistically significant number of results if you have the time and budget, but with the higher uncertainty given the potential for known issues, its best to end the survey once you’ve gathered the qualitative insight you needed.
Given the lost cost, low effort, and quick response (if the budget is set properly), MTurks is worth a try. If you need quantitative results or if you’re unsure about MTurks, try an established market research firm, Survey Monkey Audience (https://www.surveymonkey.com/collect/audience/preview/), or Centiment (www.centiment.co).
Have you used Mechanical Turks before? What was your experience like? Let us know in the comments below.