How to use mTurk for recruiting

Online experiments
7

MTurk seems to come in 2 different flavours: The MTurk UI, and the MTurk API - the latter usually made available through 3rd-party services such as TurkPrime or Prolific Academic, though if you have the technical skills, then you could develop an application yourself.

Either way, the process is similar:

  1. Create a project on MTurk and include the URL to your Pavlovia experiment. On the MTurk UI, this is done in Create > New Project > Survey Link > Create Project > Design Layout.
  2. In PsychoPy, go to Experiment Settings > Basic > Experiment info, and add a field called “workerId” to record the participant’s MTurk Worker ID. All other fields can be removed if not used.
  3. In PsychoPy, add a routine at the end of the experiment to provide participants with a survey code.
  4. Set your Pavlovia experiment to Running, assign sufficient credits to it, then save and publish the MTurk project when ready.

Collecting MTurk Worker IDs automatically in MTurk UI:

The MTurk API automatically provides the MTurk Worker ID in a URL parameter called workerId. It also provides hitId (the HIT) and assignmentId (the worker+HIT assignment). PsychoJS automatically captures all URL parameters and includes them in output data files.

The MTurk UI on the other hand, does not provide the Worker ID by default. One way to deal with this is to ask participants to enter it manually using a field in the experiment’s info dialog called “Please copy+paste your MTurk Worker ID here:”. However, it is also possible (if you wish) to hack the MTurk UI to provide the same information as the API automatically:

  1. On MTurk, go to Create > Edit the project > Design Layout > Source.
  2. Paste 1 line of Javascript (jQuery) code - the one with ‘dont-break-out’ - at the end of the source, so that it looks like this:
    // end expand/collapse
    $('.dont-break-out').attr('href',function(i,href){ return href + location.search; });
  });
</script><!-- Close internal javascript -->

This code appends all URL parameters to the experiment URL much like the API so that participants do not need to enter it manually.
Note: If your experiment URL already contains parameters, then you will need to modify the above code accordingly.

Note: You may choose not to collect Worker IDs in order to preserve participant anonymity, and rely on survey codes for assignment validation instead. This is the default in the MTurk UI, but is not the default for the MTurk API, and typically needs to be turned off if you want to go this route. Also note that this only hides the Worker IDs from the Pavlovia data output files - Worker IDs are still listed in MTurk when approving assignments. Since Worker IDs themselves are already fairly anonymous, this route has limited value, and anonymity can also be addressed by removing the Worker IDs from data output files after assignments are validated.

Providing the survey code:

The MTurk UI “Survey Link” project template requires workers to submit a survey code after they complete the task.

For this, just add a routine at the end of the experiment, with a text component instructing participants to copy an arbitrary code into the MTurk HIT: “Please use the following survey code when submitting the HIT: 23764” (use whatever number you like). Note that workers must enter this survey code manually - I have not found a way to automatically send this to MTurk as Jon suggests above.

When approving assignments, ensure that Worker IDs appear in Pavlovia’s data output files. If Pavlovia is set to Save incomplete results, then you may also want to check if each worker actually completed the experiment. Unfortunately, this process is tedious when there are many participants. It can presumably be automated with the help of the MTurk API.

Note: If you do not collect MTurk Worker IDs, then survey codes should be generated randomly and recorded in data output files instead. If you do collect Worker IDs as above, then the submitted survey codes can actually be disregarded - use the Worker IDs to validate assignments. It is also possible (if you wish) to hack the MTurk UI to remove the survey code requirement, using the following changes to the the same source as above:

    // end expand/collapse
    $('.dont-break-out').attr('href',function(i,href){ return href + location.search; });
    $('.form-group label').html('Click the Submit button when done.  Task completion is recorded during the experiment.');
    $('.form-group input').attr('type','hidden').val('NA');
  });
</script><!-- Close internal javascript -->
3 Likes