Trial Handler saved CSV split data into several cells

Coding
1

Hello,
I’m using the trial and experiment handlers to save my data (probably not in the optimal way as I find their use a bit confusing).

I start with trials that receive a list of sentences as trialList.
Some of the sentences have characters such as commas and Apostrophes.

thisExp = data.ExperimentHandler(name='mystudy',  dataFileName=filename)    
trials = data.TrialHandler(trialList=Qnames, nReps=1)
thisExp.addLoop(trials)

Then I loop over my trials, and in each trial add data (e.g. RT) using addData. For example.

trials.addData('RT', RT)
thisExp.nextEntry()

The result is a CSV that has all of the data saved. All is great except that in some cases the field with the sentence (i.e. my trial) is split into more than one cell.
My guess is that it has to do with the above mentioned characters, since one split occurred right after the following expression: ",
Is my guess correct? if so, where can I define a different encoding when saving the file? If it could be due to some other issue, please advise.

Thank you!

2

Hi Shai,

Your reasoning is correct. The ", breaks the csv file structure. With the minimal test case we can repeat the problem you mentioned:

thisExp.addData('col8', """a",b""")  # illegal

If your string is complex, I recommend encoding the string to base64. This is always a simple solution without having to change the encoding of csv (specifying the details of csv is always exhausting).

following is an example:

import base64

def encode_to_base64(s: str):
    return base64.b64encode(s.encode()).decode()
def decode_from_base64(s: str):
    return base64.b64decode(s.encode()).decode()
text = """a",b"""
encoded_text = encode_to_base64(text)
thisExp.addData('col10', encode_to_base64(encoded_text))  # legal
print(decode_from_base64(encoded_text))