Recreate csv data files from log.gz?

Online experiments
1

Background:
I was having trouble syncing updates for a known-working Pavlovia experiment across multiple computers (laptop + desktop, synced via Google Drive; syncing with Pavlovia servers repeatedly generated gitlab 404 errors), so recently tried installing the latest version of Psychopy (2023.1.0, updating from 2022.2.3) in the hopes that it would fix the problem. In the process, the “Is trials?” checkbox for the main loop of my experiment somehow became unchecked; I suspect that might result from a compatibility problem with the new version of Psychopy, though I can’t completely rule out user error.

Description of the problem:
The situation now is that 8 participants have completed the experiment using the new version of this script. The csv file only saved data for the last of its 500 trials (because “Is trials?” somehow became disabled for the main loop), but the log.gz file that was saved along with it shows data from all of the trials. Is there a script available that could use the log.gz files to recreate the csv files for these participants?

Thanks!

2

I ended up writing some code myself to parse the log files and merge them with the corresponding conditions files and truncated csv files. It would have been useful to if Psychopav had some script that automatically parsed the log files, though.

3

Hi. Would you be happy to upload and share your solution, or is it too specific to your experiment?

4

I think it’s too specific to my application to be of much use to anyone who couldn’t program their own script from scratch. But basically what I did (in R) was:

  1. Create a list of csv files in my data directory and read each file, one at a time.
  2. If the current csv file has too few trials, salvage the info that is there and then read in the *.log.gz file with the same name.
  3. With the log file read in and parsed as a tsv, it becomes a dataframe with 3 columns and n rows.
  4. Skip the first n rows until you get to the first row that looks like it represents trial level data; I read through a log file to identify some identifiers for my expt, but it would be different for other expts.
  5. Initialize a dataframe with n rows, trialData, and initialize a currentrow counter at zero.
  6. Figure out how to identify the start/end of each trial.
  7. At the start of each trial, increment currentrow, and create a vector, identifying, parsing, and saving relevant events and info in that trial to the vector.
  8. At the end of the trial, save the vector as currentrow in the trialData df.
  9. After parsing all trials, trialData should have one row with entries for each trial in your experiment, and if you did it right it should have a fixed number of entries in each row.
  10. Assuming that you used a csv to specify trial conditions, you can read that in and join or cbind it to trialData. If, like me, your csv specified a fixed sequence and you didn’t use psychopy to do any further randomization, then you can basically just tape it to your recovered trial data; if you did further randomisation in psychopy, then you’d need to figure out a way of unambiguously identifying which line from your conditions file goes with which recovered trial, and depending on your experiment and script that might not be possible.
    Figure out how to verify that nothing went awry, and fix any problems.
  11. Combine the trial data with whatever you were able to salvage from the csv file.
1 Like