Recover from log file

Builder
1

Using PsychoPy version 1.90.1 on a MacBook Pro running OSX 10.13.6, running a script that I have run many many times, on one subject there was no .cvs file, but there was a log file (I have .cvs file saved, not excel files). Is there a program or script to recreate a .cvs file from a .log file?

My RA was running and he said he had several crashed prior to that, but there was always a .cvs file. So for the subject with several .cvs files I can piece them together. But for the subject where there was no .cvs file, only a log file, can the data be recovered?

Why this happened I do not know. I wasn’t there when this happened, and I have never failed to have a .cvs file. I will work on that later. The program is mostly in Builder, but in code segments, I save more information with, e.g., thisExp.addData(‘Hue1’,Hue1) . The cause of the problem is probably one of those.

Thanks.

2

Was there a .psydat file? There is a utility to extract a csv file from that.

To use a log file would require some intensive one-off effort I think…

3

Hello Bill!

Not sure if this would help, but one part of my code converts PsychoPy log file to csv. I attached the code snippet below. It will create a csv file with the exact same name of log file with .csv extension at the end:

import csv

filenameA = 'string of log name' 
txt_file = str(filenameA) #check extension
csv_file = str(filenameA) + ".csv"

in_txt = csv.reader(open(txt_file, "rb"), delimiter = '\t')
out_csv = csv.writer(open(csv_file, 'wb'))
out_csv.writerows(in_txt)

and since PsychoPy logging includes bunch of EXP level logging, I use below code to extract just the items with logging level ‘DATA’ or other logging tags. Basically, you can save the below method as a file, and within python, call in filter_pylog(‘string of log name’, ‘string of logging level’) to filter out less useful stuff.

from __future__ import division
import csv
import time

def filter_pylog(filename,columnName):
	"""
	Syntax: filter_pylog('string of csv file name w/o extension','data')
	Example: filter_pylog('test2','data')

	This function is specific for PsychoPy log csv. It calls in the converted csv fileself.
	First cleans data with random '' and data types.
	Then filters out specific column data types.
	"""
	filtered_pylog=[]
	columnName=columnName.lower()
	csv_name=str(filename) + ".csv"
	with open(csv_name) as csvDataFile:
		pydata=list(csv.reader(csvDataFile))
	#clean out empty ' ' in data
	for i in range(0,len(pydata)):
		try:
			pydata[i][1]=pydata[i][1][:-1].lower()
		except IndexError:
			pass
	#convert from 'string of seconds' to 'float of milliseconds'
	for i in range(0,len(pydata)):
		try:
			pydata[i][0]=round(float(pydata[i][0])*1000,2)
		except ValueError:
			pass
	#filter out data by columnName
	for i in range(0,len(pydata)):
		try:
			if pydata[i][1]==str(columnName):
				filtered_pylog.append(pydata[i])
		except IndexError:
			pass
	return filtered_pylog

Hope any of this could help!

Best,
Jin Jeon

2 Likes
4

I’m not sure, but lets assume there is. Could you send me the code or a link to the code. The computer is not connected to the internet and I will not be able to drive to the lab until tomorrow (a min of 45 min each way). Also, I am not sure if the .cvs file just got put somewhere else and the RA didn’t do a thorough search. I have never had a cvs file lost because it saves after each trial.
I checked and the log file does not have all the information I need anyway.

Thank you. I hope I never have to do this again. I am perplexed because PsychoPy saves the data after each trial.

5

Switch to the coder view and from the demos menu, select “csvFromPsydat.py”.

To be honest, the most common cause of this is the experimenter accidentally stopping the experiment by pushing the big red emergency “Stop” button in the toolbar, which forcibly and immediately terminates the programme, not allowing the usual clean-up procedures (like saving data) to run. People will swear black and blue that they didn’t do it, but for an experiment that otherwise runs reliably, it is the most likely cause of this problem. Maybe the log file can give some clues as to whether that happened, but I’m not familiar enough with it to say.

Actually I just noticed this, which adds a new dimension to things. A forcible quit would only explain failure to save data at the end of the experiment, not loss of data that is saved progressively.

6

That is so cool that you have thought of almost everything! I mean it. Earlier I was trying to change cieLch to RGB. I struggled and struggled. Then I found you had a ‘cielab2rgb’ function which was 95% of problem. I added 2 lines of code and done!

I don’t know what my RA did. I have Full Screen On so he can not touch the big red button. I forgot to “prevent changes” to program and its possible the file got corrupted some how.

I am enjoying the book!

Switch to the coder view and from the demos menu, select “csvFromPsydat.py”.

csvFromPsydat.py”

Bill Prinzmetal