Best way to export a massive list of data


Part of my experiment involves collecting continous keypresses over a 3min period for each condition, and recording the data in a list of lists, formatted like this:[ [‘key’, time],[‘key’, time]…] . So when I add this data to the psychopy .csv output I end up with an extremely long list of about 23724 values.

Screen Shot 2023-03-17 at 3.44.19 PM

This is my issue: When I try and import the data from .csv into Excel it only imports the data from the first 3 seconds. I’ve tried copying and pasting the entire list, but that also imports just the first 3 seconds. Given that I’m going to eventually have 16 of these very long lists, I’m wondering if there’s a better way to handle the data.

One way I can think of would would involve writing data using code in the each frame tab to a csv file for each of the the 16 conditions, with the key in one column and the time in another. Does this sound like something that could be accomplished with CSVwriter? (I have no experience with it but am willing to try!) Or is there a better way?

You could put thisExp.nextEntry() in your Each Frame code to move your data to the next line after every key press (or every second or however you want to divide it).

Thank you very much! I tried thisExp.nextEntry() in a few places but realised that it wouldn’t work if I was just appending to one list created at the beginning then adding that list to the data the end. But I ended up changing things so that it would create a new list each with routine (my keypresses span multiple routines) and add it to the data (in the same column each time) at the end of each routine. However, these lists are still too long to export to excel properly, so my next goal is to figure out if I can collect keypresses every 0.25 seconds or so to make the lists much smaller.

Are you adding data every frame? I would recommend saving keypress data when a keypress is actually made rather than having lots of empty results. Then you could have a row per keypress and not have to deal with lists at all.

Participants will always be pressing 1 of 4 keys (the end goal is to generate a time-course of data with time on the x axis and the rating on the y), but I was able to log the data every 0.5 seconds (or every 30 frames) so I think I should be good

What I would do is only record the times that the key pressed is different from the key pressed on the previous frame, rather than also including all of the times that it is the same.