In our first run of the experiment a lot of students seem to have ended it before it had finished by closing the browser or by simply shuting down the computer. As far as I understand, data are logged if the experiment is ended by hitting Escape? In our first run more than 50 kids started the experiment, however there haven’t been created any new csv. files (saving format is set to csv. files), no credits have been consumed, and 18 credits have been reserved.
Clarifications about when exactly data is stored and when a credit is consumed/reserved would be very helpful. Thanx
Credits are reserved when somebody starts a study and consumed when data collection is confirmed or released when we’re confident that the participant is not finishing the study (this is currently an extremely conservative 24hrs later but that will change).
At present, no, data are only sent back at the end of the study.
Closing the browser/window: this certainly aborts the run. I think we have to assume that when a participant quits the browser they do mean to be excluded from the study and we should not be keeping their data.
Hitting escape: I actually thought that hitting escape does save the data but I might be wrong - there was a discussion about whether escape indicates that the participant wants to exit and expects you not to keep their data, or whether we think keeping the partial data is reasonable in this case.
To work out what’s going on in your particular case I think you’d need to observe someone running the study and see what happens. Maybe it’s crashing at some point and they can’t complete?
I have just launched your experiment. I can confirm that credits are reserved at the start and freeed at the end, should the experiment be interrupted by Escape or should the tab be closed. I can also confirm that data is written when the experiment is interrupted by Escape but not when the tab is closed.
However I was unable to actually run your experiment. On Google Chrome and Firefox on Linux and on Windows, I can see the introduction screen but am unable to move forward. Is it working on your end?
Dear @apitiot, thank you very much for the confirmations. It is running on my end. In certain routines you need specific buttons to continue (to prevent students from proceeding on their own). These are: 1; 9; q; p; y; e. Could you please check if that works?! Thanx
Hi,
I did a second trial run with 3 friends that assured me that the waited until the test told them that it was safe to close the window, but i only got reserved credits and cannot access their data,
@cami94 most people’s experiments are certainly running fine to the end and saving data, so if yours isn’t we would need to work out what’s different. Is it a long study? Is there anything unusual about the data being stored, or the type of study? Are you redirecting subjects somewhere at the end? Basically, try to tell us anything that you can think of that distinguishes your study from basic examples that are working.
@jon it consists of many small parts, i think it’s about 13 routines, but all part’s is quite simple,
is there anything i can send you? unfortunately I have very little experience within this field… yesterday i had 154 reserved credits and today 22, and I know that a lot of people has done the test today, so I suspect that a lot of data has been lost…
Please let me know more in detail what I can send you so that you can help us!
I have just run your experiment. The log files were correctly saved in the data directory. As a matter of fact, I was your third participant today, and you had 6 yesterday.
The experiment results are saved in the database, since your experiment is configured for database saving rather than .csv. That is perhaps the cause of your concerns.
For instance, this is the last entry from my run: { "_id" : ObjectId("5ea18f10130a1fc76c1f4646"), "__projectId" : 20044, "__experimentName" : "openIAT", "__participant" : "ANONYMOUS", "__session" : "001", "__datetime" : "2020-04-23_14h46.53.043", "key_resp_4~keys" : "space", "key_resp_4~rt" : 3.6113399998284876, "participant" : "ANONYMOUS", "session" : "001", "date" : "2020-04-23_14h46.53.043", "expName" : "openIAT", "psychopyVersion" : "2020.1.2", "OS" : "Linux x86_64", "frameRate" : 60 }
Could I ask you to go to your experiment page from your dashboard and click on the Download Results button? You should be able to access all of your results in that way.
To change the way the results are saved, go to your experiment page and under Saving Format (on the right hand side), click on CSV. Your results will then be saved in .csv files.
Hi Alain,
thank you for checking on this, I have however 5 new reserved credits most of which was from today, and I I have more than 10 people that wrote to me saying they took the test today. All of whom I asked if they had waited until the screen said “thank you for your patience”. then gone blank.
I currently only have 36 replies accourding to my dashboard, out of which i am positive that at least 14 is my own and from our send test trial as I suspected that the answers weren’t saved correctly…
And it still doesn’t explain the more than 150 reserverd credits from the day befor yesterday, is it really no way that you have a backup of those that we can double check? that represent almost half of the replies we hope to receive…
when i download I only get from the 34 .csv objects with data, out of which 1 is “all data”
PLEASE let me know if there is anything else I can do or provide you with to take a look at this,
I am really afraid that we are loosing data that we will not be able to collect again
I am very surprised by the disconnect between what your participants are reporting to you and what the pavlovia logs are showing.
For instance, on the 23rd of April, only 9 sessions for your experiment were opened. 3 were completed, 1 was aborted (probably pressing ESC), and 5 hung (probably just closing the browser tab). That is somewhat different from what was reported to you.
From yesterday’s logs, 6 people started your experiments, two completed it, one aborted it, and the other 3 just gave up and closed the tab. At least as far as I can tell.
The 150 reserved credits are from 50 people starting your experiment but not finishing it on the 20th and 109 starting but not finishing on the 21st. Which participant service are you using? Sona perhaps?
Your experiment is long and repetitive: in my view, it is entirely possible that people get bored half way through and give up.
Do you have a way to ask someone to take the experiment for you, and tell me what happens then? There is not much I can do on my end since, as far as I can tell, all is working well. And, as you mentioned, you can run the experiment without issue and so can I.
Hi Alain,
As you mentioned this does not not add up, for example i know that at least 4 was completed yesterday as i was sitting beside or were on the phone with the participants, but i could only access the two completed. The test ran as usual, no glitch and all of the participant got to the point “thank you for your patience”
I have not used any participant services, i have shared the link on social media in groups that represent our target group.
is there anyway to access the data that was started but not finished? aborted test data is stored localy when doing it in builder, so it should store that data on the server as well? if we could access that we could identify where the error is - my suspicion is that it might be related to the ok button on the “thank you for your patience” window, making the test valid and complete but just not stored.
I have a hard time believing that between the 159 people did not complete the study between the 20th and 21st as they before the test had been informed of the design of the test and the time it will take
Please let me know if there is ANYWAY to access this data
As far as I can tell, only the last two participants completed it. At one time were you sat with your two participants?
Incidentally, when the first session started, the experiment’s saving format was DATABASE. I can also see that a request to change the experiment saving format from DATABASE to CSV was made later on at 2020-04-27 18:22:42 +0000. Did you do that?
Did you notice that you have one participant result from the DATABASE for 2020-04-26, and several from 2020-04-23, among other dates?
There is no way to access the data of sessions started but not finished at the moment, unfortunately.
Looking at the code of your experiment, it appears that you are checking whether the ESC key is pressed even in the thank you routine. If the participant presses ESC at that point, it will abort the session and no result will be saved. That is however not the issue here since we do see the aborted sessions in the logs and you did not have any yesterday.
I have no difficulty believing that over a hundred participants (students, I imagine) decided to give your experiment a go but were not motivated enough to finish it. It is already tricky enough to get participant to finish when there is a reward, but in the absence of a reward, and running your experiment completely anonymously over the internet, it is very easy for them to just go away as soon as they are bored.
yes it was I who changed this, as per your previous comment that it might be the cause, i thought i would try to change it to .csv and see if it made any difference - but it didn’t.
Which do you recommend that I use?
I was present/ on the phone with the 4 latest.
I would say that maybe 75-100 persons are students, the rest is actually business consultants all over the world from a big company, so I do still have a hard time believing that that many just decided to shut down the test…
FYI, we think that we have identified the problem, but we still have no solution.
we have noticed that it is most likely the choice of browser, although the test works in all browsers, when we have asked people to printscreen their last step some of them did not get the last dialog box that says “closing the session please wait a few moments”, but only a white screen.
Thus they close the window too early and their participation is not saved as a valid result and only reserves a credit.
Hi, I think I have a similar issue. So far, I appear to have 51 individuals who started the experiment. Of these, I only have 7 data stored, 5 aborted, but 41 are on reserved credits…isn’t it a very high proportion?
Thanks!
Hi,
I have it set to save any aborted sessions, however, when I download the results, the excel file appears blank in all aborted sessions.
The participant is indicated to press “space” to exit the experiment and wait for the dialog box with the confirmation of the data being uploaded correctly.
