Memory storage on Gitlab repository - Pavlovia

Online experiments
1

Dear all,

I was just wandering whether there is a max limit in storage size for the dataset recorded through online experiments. I am saving results in .csv format. According to the dataset size that I will record, I expect to have ~2’300 MB of data by the end of the experiment.

Is there enough space for this dataset size? or there is the risk that the data won’t be saved anymore on the server due to limits in the max storage size ?

Sorry for the naive question, and many thanks for your help!
P.

2

Dear All,

Does anyone has an idea about that? Many thanks for your help :slight_smile: @dvbridges @jon

Pool

3

We haven’t manually inserted any limits ourselves at this point. Do you mean 2.3GB (sorry I’m not sure about the ’ notation you’ve used)? I’m wondering what would cause such large files. What are you saving into these? Or are the files relatively small but you’re expecting thousands of participants?

4

Many thanks for your reply!

Yes, I mean 2.3 GB, sorry about that. You are right: We have a longitudinal study, each participant (30 in total) is expected to complete 30 experimental sessions, for a total of 900 sessions. Each session produces ~ 2.7 MB of data (log + csv). That’s why the final dataset will be quite large.

I guess there won’t be any issues with our study since there are not data storage limits?

Many thanks again for your reply!

Paolo

5

It’s something we’ll have to monitor for excessive usage, but for now there is no limit to prevent this

6

2.7Mb is a huge data file for one session though, since these are raw text. What are you saving in the file?

7

Many thanks for your reply.

Each session produces a CSV that is actually quite small (457 KB), and a .log file that is actually 1.9 MB. The problem I would say is the log file

Is there a way to avoid saving the .log file?

Cheers,
Paolo

8

I can provide both if you wish to have a look at it, no problem at all.

Paolo

9

Oh. Interesting. That might be something we can make smaller at our end if you could send one (or just a link) that might be handy

10

Sure, with pleasure ;).

The log file can be downloaded from this link:

Cheers

11

Speaking of the size of log files …
I’m currently trying to implement a task in pavlovia that used to run as a PsychoPy Coder experiment in the lab.
The stimuli are pretty complex. There are 409 elements (predominantly of the type polygon) on average on the screen in each trial. Since every single element is logged, the log file accumulates to a size of up to 300 MB (with approximately 2 million lines).

Is there an option to switch off the log file or change the logging level in an online experiment?
Might the accumulation of such a huge amount of plain text cause a visible increase of RAM usage over time as well?

Any help is appreciated and thanks in advance!
Kim

1 Like
12

I would also like to know if there are ways to avoid saving the log files on Pavlovia! Thank you!

13

I managed to “turn off” the log files by commenting out certain lines of code in the core lib-file.
I’m not sure whether that alteration of the code leads to any side effects. In my task, I didn’t notice any.

When I ran the online experiment with the altered file, no log file was generated/saved and according to the DOM-inspector of my browser there seemed to be less storage spent on strings (I’m no expert but to me it seemed reasonable to be related to the alterations).

It did not resolve all of my issues, but depending on what you’re trying to achieve it might work for you.

I used an altered version of the script provided by @benwang (see topic below) to update the relevant files after every sync with pavlovia.

In the altered version of the core-file (core-2020.2.js) I commented out line 527, 531, 539 - 544 (class Logger code chunk)

    exp(msg, time, obj)
	{
		//this.log(msg, Logger.ServerLevel.EXP, time, obj);
	}
	data(msg, time, obj)
	{
		//this.log(msg, Logger.ServerLevel.DATA, time, obj);
	}
	log(msg, level, time, obj)
	{
		if (typeof time === 'undefined')
		{
			time = MonotonicClock.getReferenceTime();
		}
		/*this._serverLogs.push({
			msg,
			level,
			time,
			obj: toString(obj)
		}); */
	 }

and line 806 (class Window extends PsychObject code chunk).

logOnFlip({
							msg,
							level = Logger.ServerLevel.EXP,
							obj
						} = {})
	{
		//this._msgToBeLogged.push({msg, level, obj});
	}

You can find the original lib file version corresponding to your prefered PsychoPy release by changing this link: https://lib.pavlovia.org/core-2020.2.js

2 Likes