Saving data from all subjects in a single file

Coding
1

Hi,

Right now my code creates a separate output csv file for each of my subjects with the data for all items (e.g., practice routines included). What I’d like to try to do is save the data of all subjects for the actual routines (i.e., excluding the practice trials) are appended in an additional csv file.

I figured that one way to do this is to use ExperimentHandler similarly to what I have done to create the individual csv files - i.e., by using addData() for all variables I want to save in the cumulative csv file. I was just wondering if there is any smarter way to do this.

Thank you in advance!

2

Honestly I think the smarter way is to not change the default functionality of saving in separate files, and use another tool or script to concatenate the files and put them in a format that you like better.

The reason for this is, if you have edited your experiment to use a single file, and something goes wrong, it is very very easy to corrupt the entire file, and as a result lose ALL of the data from your previous successful runs.

Search online for some examples with something like the pandas library. It’s got some great tools for this type of thing.

2 Likes
3

Here’s a simple script to combine all csv files in a folder into a single file. concat_data_files.py (2.6 KB)

I’m going to leave pretty explicit instructions for future readers.

Create a folder to work in

To follow this example here, create a folder on your Desktop called concat_files. Inside that, create a folder called ‘input_files’ and another called ‘output_files’. Copy the python script included here into the ‘concat_files’ folder. Copy all of the files you want to combine into the ‘input_files’ folder.

Install pandas

You need to have pandas installed. To check for pandas, open a terminal, enter this:

python
>>> import pandas

If you don’t get an error, you have pandas installed.

If you don’t have pandas installed, I highly recommend you use a virtual environment. Virtualenv is the most common, but since we’re all in the sciences here I’d recommend using Anaconda. If you don’t have Anaconda installed, you can start with Miniconda, it’s a smaller download.

Create a virtual environment with conda (optional)

If you’re using conda, create a new folder somewhere, copy this file into it. Open a terminal and cd into it. For example, if you created a folder ‘concat_files’ on your Desktop, type this in Mac / Linux:

cd ~/Desktop/concat_files

Windows:

cd \Users\<your-username>\Desktop\concat_files

Create a conda virtual environment.

conda create -n "concat_files" python=3.6

Activate the conda environment

conda activate concat_files

Install pandas:
Conda only:

conda install pandas

Or using pip:

pip install pandas

Run the script

If you’re following along, just type this:

python concat_files.py

You can always edit the script to your liking.

2 Likes
4

Hi Daniel,

thank you for your posts! I’m sure this will be helpful for others too.