Error using the shelf: the session is not opened

URL of experiment: Adrian Jusepeitis / cbtest · GitLab

Description of the problem: I am trying to code a prototype for comprehensive counterbalancing using the new features of “shelf”. Running some shelf-related code (updating a dictionary) in the “End of experiment” section of my code, I encounter the following error: when updating the value of the DICTIONARY record associated with key: [“counterbalancing”], error: “the session is not opened”. I suspect that the experiment is ending too fast for the shelf to operate. I was able to alleviate the problem by running the code in “Begin of routine” in the last routine of the experiment and forcing it to a certain duration using a text component. However, it is not 100% solved. Sometimes the error still appears. So I wonder, whether there is any way to force the ending of the experiment to wait for the shelf code to operate. Does anyone have an idea?

PS: To test my experiment, a dictionary record with the scope of the experiment is needed. It needs to be called “counterbalancing” and have the following content:

{
  "info": {
    "time_out": 2,
    "group": [
      1,
      2,
      3,
      4
    ],
    "n_wanted": [
      1,
      1,
      1,
      1
    ],
    "n_complete": [
      0,
      0,
      0,
      0
    ],
    "n_start": [
      0,
      0,
      0,
      0
    ],
    "active_starting_times": {
      "1": [],
      "2": [],
      "3": [],
      "4": []
    },
    "counter": 0
  }
}

Hello!

Firstly, thanks for interacting with the forum and telling us about this issue. Secondly, have you tried adding a code component to your final routine and instead putting this code in the “End Routine” tab?

Becca

Yes, that I also tried. It results in the same error. From what I tried, it seems like the closer to the end of the experiment the code is, the more likely the error becomes.

Hello @ajus ,

The issue stems from your making several calls to setDictionaryFieldValue in quick succession, in particular at the very end, in CounterbalancingRoutineRoutineEnd, with some of those calls not completing before the experiment ends, on occasion but not all of the time.

There are two reasons for this:

(a) As you may know, asynchronous calls return immediately, except when you use await, which you did here and there, but not everywhere.
(b) Shelf methods are currently throttled, to make sure that we are not overwhelming the server, during the beta version period. The throttling period is 500ms right now, but will go down in the coming weeks. That is to say: two consecutive calls to a shelf method will only be made 500ms apart.

A good way to think about it is that a call to setDictionaryFieldValue is really a request to the server to change the value. It basically tells the server: do change that value as soon as you can. The server will receive this request at some point after you have requested the change (subject to throttling), it will make that change, and it will inform the experiment that the change was made.
With throttling in place, even though you may make two calls to setDictionaryFieldValue back to back, they will only be submitted to the server 500ms apart from each other.

Coming back to your experiment:

  • A first improvement would be to use await systematically. This garantees that the experiment won’t end before all the requests have been processed by the server.

  • A second improvement would be to reduce the number of calls. In CounterbalancingRoutineRoutineEnd for instance, you make a change to n_complete and request a change on the shelf, and then if started is true, you make a change to active_starting_times and request a change on the shelf: that’s two calls which could be a single call, which means that there would not be any throttling. You can use the same optimisation in experimentInit.

  • A third possible improvement would be to use the Shelf provided counterbalancing approach, counterBalanceSelect. It is fast and takes care of concurrency on the server, which is very, very tricky to do at the experiment level. But perhaps it does not quite satisfy your needs? The current implementation is a straight-forward one at this stage. We will be improving upon it in the coming weeks, adding many features, and I’d be delighted to hear your opinion.

Alain

Hi Alain,

thank you for the in-depth response! Reducing the calls to the server to the minimum necessary and consistently using await made the code work robustly without errors.

Concerning your third suggestion: I was unsure as to what the scope of capabilities of the build-in counterbalancing is since the documentation that I found is at the moment rather sparse. My two main doubts were:

  • Does counterBalanceSelect select an available group at random or does it fill one group after the other?
  • Does it take into account participants that are active in one group but have not yet completed it?

Can you help me with these questions? These two things are important for me and the reason why I tried coding my own procedure.

Thanks again!
Adrian