when psychopy exports data in cvs format, it incorrectly quotes the quotes…
if the field contains double quotes (ASCII code 34) the resulting csv file should have it
as “double”. That is " should become “”
however, only a single " is produced which causes “downstream” problems while reading the data
LF
Dear Lazar,
This is nothing to do with psychopy afaik, and just an excel problem. ASCII 34 is straight quotes; programs (like word) will fix those to make them “curly” quotes with context, whereas excel will just take them exactly as such. In ASCII, it would be ASCII147/148, and in unicode would be U+201C/U+201D. Maybe there is an excel setting for it?
For data cleaning ctr+r, search for " and replace with “ and then " and replace with ” (_= space). That only works if your data cell has a space preceding the quoted word. If not, you will have to make a python script or something to run through and replace them.
Otherwise, you need to fix your downstream data cleaning. Do you need the quotes at all?
Issac
Dear Isaac
the csv I am talking about is created by psychopy.
so it is not an excel problem. it is a psychopy one.
however, looking at the code
I do not see an attempt to comply with csv data format in
ExperimentHandler.saveAsWideText()
or
TrialHandler.saveAsText()
either correct the problem or stop calling them csv files because
no attempt is made to comply with the csv format.
these are just text files…
Lazar
Hello @Lazar
I don’t think the conclusion is obvious. There is a code line.
csvData = self.data.to_csv(sep=delim,
encoding=encoding,
columns=self.columns, # sets the order
header=(not matrixOnly),
index=False)
in psychopy.data.trial. But there are three trial handlers and only one use this code. There is this cue that TrialHandler2 is used in an experiment
if isinstance(self.getCurrentLoop(isTrials=isTrials), TrialHandler2):
in the ExperimentHandler class.
If you think it is bug, report it on GitHub - psychopy/psychopy: For running psychology and neuroscience experiments · GitHub
What program do you use to inspect your data?
Best wishes Jens
Thank you for the bug-report link. this is helpful.
I use TrialHandler (without 2)
and in general.
due to perpetual problems with psychopy, I advise people to avoid the software whenever possible.
when it is not possible to avoid psychopy
the questions such as this arise..
I am reporting an observation: psychopy exports csv files incorrectly.
I have looked at the code of the task which function is called and then I looked at the code
of the functions (the links I provided) and did not see any effort to comply with the csv format.
I am not sure what exactly the problem is. Maybe share your experiment? I did some tests to see how different things are exported to csv. Curly quotes are exported and saved as would be expected
| “hello” |
|---|
However, when using non-curly quotes, the quotes are removed in the export.
| “hello” |
|---|
becomes
| hello |
|---|
But this is what CSV already does if you try to copy and paste “hello” into the csv, it automatically removes the normal quotes.
Is your problem that you want it to be " “hello” ", but you are only getting “hello”. If so, that is due to excel, not psychopy.
Furthermore, if you open the data file in notepad instead of CSV, the quotations marks are displayed correctly. Which only points further to the issue being excel and not psychopy. Can you share your experiment or data to see what specifically is happening in terms of “psychopy exporting csv files incorrectly”.
Issac
The problem is that when the text field contains regular double-quotes, they have to be escaped according to csv rules.
that is
a csv field is always encapsulated in double quotes like this
If I want to store
Hello, what do you think of “what”?
in a csv field, it must look like this in the file:
… , “Hello, what do you think of ““what””?”, …
notice comma-double quote in the beginning and double quote-comma at the end.
everything which is between ," and ", is the content of the field and
if the content has a double quote, it has to be escaped by another double qoute to avoid confusion with field separation.
this is the specification of the format…
however, psychopy does not do that…
Honestly, I am unable to replicate your issue.
This is the requested field contents:
This is what psychopy saves the field content as:
As you can see I don’t need a double quote in a field containing a comma.
If you could provide a snippet of your experiment and data, it would go a long in terms of figuring out your problem.
Issac
I struggled with this issue as well. Note: I am using a way outdated version of Psychopy (2025.1.1), so maybe this is not an issue anymore. But by using a TextBox component for inputting text, I can reproduce the problem with no effort.
Setup:
Running the experiment and entering the following text:
This "test" will not work, though. Lets, add, some, more, commas
maybe even
some
line breaks?
Will generate this faulty csv:
Plain text contents of csv:
thisRow.t,notes,r_read_word.started,textbox.started,key_resp_2.started,r_read_word.stopped,textbox.text,key_resp_2.keys,key_resp_2.rt,key_resp_2.duration,participant,session,date,expName,expVersion,psychopyVersion,frameRate,expStart,
,,0.004067899993970059,0.008697099998244084,0.008697099998244084,36.312751699995715,"This "test" will not work, though. Lets, add, some, more, commas
maybe even
some
line breaks?",0,36.28649900000892,None,470535,001,2026-07-28_10h50.57.603,untitled,,2025.1.1,59.97487052815985,2026-07-28 10h51.01.039077 +0200,
As you can see, it is not properly escaped.
For some reason ticking the “save as Excel option” wasn’t working for me in the experiment settings. Otherwise that would be my first choice.
My fix was to remove
thisExp.addData('textbox.text', textbox.text)
and replace it with something like this:
clean_response = textbox.text.replace('"', "'").replace('\n', ' | ')
thisExp.addData('textbox.text', clean_response)
Issac