How accurate are online experiments vs on-site experiments with regards to reaction time?

Online experiments
1

URL of experiment:

Description of the problem:
Dear Psychopy hivemind:

I have been developing my experiments for on-site data collection. After the pandemic started I have been considering doing my experiments on-line. I have 4 tasks:

  1. Word recognition task - written words on the screen.
  2. Semantic discrimination task - written words on the screen.
  3. English to American Sign Language translation task - Audio stimulus to recorded sign translation equivalent.
  4. ASL Sign recognition task - Video stimulus (sign or sign parameters) on screen.

I was doing it online to avoid variables and confounding variables associated with differences in equipment, internet speed, and room and distractions. I have several concerns with online experiments for these tasks:

  1. How did you deal with internet and equipment issues?
  2. Will the experiment stop working on it’s own when something is not working or will I receive the results anyway? How did you figure out when a participant had issues on their end?
  3. Is the data of online experiments saved in a cloud on a folder I create and I download them or are they automatically sent to me or saved on my computer?
  4. What are the minimum equipment specifications for running the online experiments? is there a recommended browser that is preferred over others or any browser incompatible with Psychopy?

I know these are many questions. I am a new Psychopy user and this are the first experiments and tasks I develop in it. Any support will be greatly appreciated.

Sincerely yours,
Ricardo

2

Hi there,

First - welcome to psychopy!

Second, please can I recommend the recent timing study by the team which details the most recent timing measurements for online and offline studies https://peerj.com/articles/9414/

With regards to your questions:

  1. This is really a constraint that will always be there with online studies and there are many issues. Resources are loaded at the start of the study so internet connection should really only effect the loading of resources.
  2. I think this will become clearer when you pilot your experiment. Usually an error message will appear when something is wrong, and usually you will spot this during piloting. But I would recommend you try your experiments on several devices and browsers to check it is running before rolling it out for data collection. with regards to data - you can select if you want partial data or only complete datasets to be saved and therefore use a credit.
  3. the data are saved to gitlab, you can click “download results” in your experiment within dashboard and it will download this as a single folder for you.
  4. there are some issues with safari - that you may see when reading through issues on the discourse. But this will depend on what type of experiment you are running. I would recommend testing your experiment on several browsers to be sure.

Thanks and good luck,
Becca

1 Like
3

Thank you so much Becca! I apologize for the late response but wanted to let you know that your response was really helpful. I have three of my tasks already online and piloted them and are working. Two with written strings of letters and one with audio stimuli. The only one that seems to be giving me trouble is the fourth task which has video. This is the message I got: “rortizterp/american-sign-language-lexical-decision-task is currently in PILOTING mode but the pilot token is missing from the URL.” I will post it as a separate post to see if you or anyone available could shed some light. I honestly don’t know what the pilot token is yet. Hense why I haven’t rolled them out.

Best,
Ricardo