No csv file saved - scrape log file

A few people have found csv files not saving, combined with corrupt psydat files; a nasty combination that was occurring for a while (mostly this was seen in version 1.84).

For those that need a workaround to extract data you might want to think about trying to extract the data from the log file, assuming it existed there. Doing this manually would mean scrolling through many lines and manually extracting relevant information from all the other stuff but you could also write a Python file to do it for you.

The script below is a demo of that. Notes on how it works (and what you need to change) below the script.

import re
import codecs
import glob

files = glob.glob("data/*.log")  # the folder with the logs (relative to this file)
# we'll use a regex for this even though they're ugly and painful:
scanQuote = re.compile(r"'(.*?)'")  # finds any number of words within a quote

for filename in files:
    # open log file
    print "analysing", filename, "...",
    thisFile = codecs.open(filename, 'r', 'utf-8')
    #open a data file to send csv data
    outFilename = filename[:-3]+".csv"
    outFile = codecs.open(outFilename, 'w', 'utf-8')
    outFile.write("image,rating,RT\n")
    #go through each line in the file
    for line in thisFile.readlines():
        if "New trial (" in line:  # new trial reset all entries
            image = rating = RT = ''
        elif "NameOfMyImageCompon: image = u" in line:  # setting an image so get it's name (in quote)
            image = scanQuote.search(line).groups()[0]
        elif "rating=" in line:
            rating = int(line.split("rating=")[1])
        elif "RT=" in line:
            RT = float(line.split("RT=")[1])
            # RT is the last record in each trial so can write line now
            outFile.write("{},{},{}\n".format(image, rating, RT))
    outFile.close()
    print "OK"

It loops over every line in a log file (actually over every line in all the log files in a folder) and checks whether the line contains relevant info. In this particular script the analysis looks for any line that indicates a new trial and sets everything to blank, then if it looks for an image component called NameOfMyImageCompon being updated to a new image and also any updates to a rating scale or assignments of reaction time. Any lines with those things get grabbed and checked and the values get assigned to a variable which is then written to a new csv file.

To tweak this for your own use you need to alter the lines about outFile.write() to make sure they have useful field names for you to analyse at the top, and then format the final “write” line so it has as many {} entries as variables that you include.

The line about scanQuote is to extract the contents of any quote within a line. When the image file is set the name of the image is a string so we can find it like this. For the Rating it isn’t a string so we can just find the numeric value.

2 Likes

Hello @Jon
I came across your reply and I’m experiencing a similar issue. I need to extract information from a .log file and convert it into a CSV file—or at least retrieve all the relevant data in a CSV-like format, since the ESC key was pressed at the end of the experiment, resulting in a log (.log) file instead of a CSV file.
so I wanted to ask:

  1. Does the script you shared still work with the latest version of PsychoPy (2024.1.5)?
  2. Is there a newer method or an updated solution for extracting data from log files?
    Since I don’t have a CSV file, I’m not sure how to properly extract the necessary data from the log file. Any guidance or recommended tools would be greatly appreciated!
    Thanks a lot for your help!