Conditional audio replay (n-back) and corresponding keyboard component not transferring well to pavlovia

URL of experiment: Sign in · GitLab

Description of the problem:
I am trying to convert a Builder experiment (that works error-free in Builder!) to Pavlovia, and there is something funky going on in my conversion from python to js. I know about the auto → js translator (thank you!) and that I also need to update my code within my code components. However, there is still something up with my code and I can’t figure out what needs to be fixed.

This experiment will play participants a series of spoken sentences in a made-up language (note that for the purposes of piloting, we are running a shorter “debug” condition). Normally the sentences will play one after another. Sometimes these sentences will repeat. When this happens, the audio is supposed to stop and the participants are told to press “y” on their keyboard (“y” indicates that yes, they heard a repeated sentences). When “y” is pressed, a ding (ding.wav) is played and the audio should automatically move on, playing more sentences until a repeat emerges, upon which the cycle will begin again.

Originally, we had done this in Builder with a code snippet for an n-back routine, but this is proving too hard to convert to js in pavlovia. Our currently plan is expanding our condition files (see the individual condition file for each level - there are two in the debug version - in the “trials” folder) to manually include two of the same audio file back-to-back (instead of indicating one is an nback). We’ve tied a variable to a keyboard component (repeated_resp) and an audio component (ding) within the exposure routine.

We are currently stuck trying to define the correct variables we need to convert this n-back (or n-back-like) audio test and corresponding keyboard response into pavlovia. The experiment URL should be public and I’ve attached pictures of what our current keyboard response and code snippet of interest look like.

Thank you so very much!!

Try replacing “FINISHED” in the condition for repeated_resp with “PsychoJS.Status.FINISHED”

You may also need to change the status references in your code to be

x.status == PsychoJS.Status.FINISHED

or NOTPLAYED as the case may be.

Basically the way statuses work in PsychoJS is a little different and harder to reference.

Thank you very much!

I have made those fixes and really feel like we’re so close to solving it, but I am having issues figuring out where in my code I can “define” these variables that pavlovia is struggling with.

My current ding start condition is: repeated_resp.PyschoJS.status.FINISHED

My current repeated_resp start condition is:
exposure_sound.PsychoJS.status.FINISHED

The idea is that the participant can only give a keyboard response (or it can only “count”) after the audio trial is complete). Then, the ding can only play with the onset of that keyboard response (at the correct time).

I’ve attached my new code snippet below. My current error message is " TypeError: Cannot read property ‘status’ of undefined." It is referring to line 2611 in the code, which is attached below (the readout is in Sublime). Line 2611 refers to the ding component and looks different than the rest of the code but I don’t know how to solve that what I can access in the Builder.

Thank you again so much for your help!

OK, I would rewrite your code element as follows:

if((corrAns = "y") && (exposure_sound.status === PsychoJS.Status.FINISHED)){
   if(repeated_resp.getKeys["y"]){
        ding.play();
        ding.getDuration(2);
        if ((expInfo["auto"] === "1")){
            continueRoutine = false;
        }
    }
}

if (corrAns != "y") {
    ding.status = PsychoJS.Status.NOTPLAYED; //note: This may need to be NOT_STARTED, try both if it doesn't work
    if ((expInfo["auto"] === "1")){
        continueRoutine = false;
    }
}

Basically right now there’s a lot of stuff that looks a little off and I have a sneaking suspicion that it’s making it try to create new variables that don’t actually exist when it puts together the JS code. With this code component it will hopefully sort itself out.

Hi @jonathan.kominsky! Thank you very much for your help. I’ve adjusted my code accordingly and have been playing around with this over the past few days, but am still having some trouble, specifically with the continueRoutine function.

I’ve renamed my variables so that corrAns → repeat and “y” → “r” (not to be confused with the ‘y’ that is the actual keyboard press!) and have taken out some stuff that is redundant. In my conditions file, if the column “repeat” has an “r” in it, that sentence is a 1-back and the participant needs to click ‘y’ on the keyboard and they should hear a ding. Otherwise, the exposure files should just keep on playing.

Right now, the first sentence in the conditions file is an “r” sentence (I made it this way to debug) and I can click ‘y’. I don’t hear a ding, but the routine does end and the next sentence plays. However, the routine does NOT end (despite me telling it to) and I can’t do anything else.

I know that my keyboard component (repeated_resp) has an error; I’ve checked the box next to “force end of routine.” I presume this is ending the routine before the code even gets to the ding and this is why I am not hearing a ding before the next exposure file is plays. Importantly however, if I do NOT check that box, clicking ‘y’ does nothing. No ding, no next exposure sentence. The routine does not end.

Do you have any idea why this is the case? I’ve includes pictures of my revised code, as well as the builder components for ding and repeated_resp. It makes no sense to me why the ding component isn’t being called and why continue routine isn’t working either. I know there were issues with continue routine in the past, but I am using version 2020.1.

Thank you so much in advance for your help with this!!!

Might be how getKeys works online is different than I think. Let’s try this:

if((corrAns = "y") && (exposure_sound.status === PsychoJS.Status.FINISHED)){
  let repeatedRespKeys = repeated_resp.getKeys();
  if(repeatedRespKeys[0]=="y"){
        ding.play();
        ding.getDuration(2);
        if ((expInfo["auto"] == "1")){
            continueRoutine = false;
        }
    }
}

if (corrAns != "y") {
    ding.status = PsychoJS.Status.NOTPLAYED; //note: This may need to be NOT_STARTED, try both if it doesn't work
    if ((expInfo["auto"] === "1")){
        continueRoutine = false;
    }
}

EDIT: That won’t work for the condition element on “ding”, which you should leave blank. The code component is taking care of that for you.

Hi @jonathan.kominsky - thank you so much for your quick reply! I made the suggested update but unfortunately the result is the same as described in my previous post. I’ve tried to re-code this several ways now, and I get the same result every time.

It seems like the experiment will only progress through the routines via a key presses (not a code snippet declaring continueRoutine = false). Additionally, any audio starts either at the beginning of a routine or at a specific time (as told to by the sound component), not by anything written in a code snippet.

If I leave the condition element on “ding” blank, I get an error message. My coworker @fetch discussed this with @jon a few weeks ago here: frameDur is not defined error. However, putting a conditional variable into “ding” doesn’t work make the audio play. @jon, do you have any other suggestions to this problem?

The current experiment should be public here: https://gitlab.pavlovia.org/learningdevlab/v17_i_give_up (sorry for the name of the experiment - I named it in a moment of coding frustration)

    if ((repeat == "r") && (exposure_sound.psychoJS.STARTED == true))
        {
        let repeatedRespKeys = repeated_resp.getKeys();
        if (repeatedRespKeys[0] == 'y')
            {
            timetoDing = true;
            ding.play();
            ding.getDuration(2);
            continueRoutine = false;
            }
        }
    else if ((repeat !== "r") && (exposure_sound.psychoJS.STARTED == true))
        {
        timetoDing = false
        ding.status = PsychoJS.status.NOT_STARTED;
        continueRoutine = false;
        }

Hi there - I’m sorry to double-post but I wanted to provide an update. I’ve been trying to debug other parts of the experiment, including a progress bar that we are trying to update every trial (each trial, the progress bar should decrease in size until it disappears, at which point a break will start).

I came across these articles here Use thisRepN correct - Online experiment created with builder and here https://github.com/psychopy/psychojs/issues/49 regarding how TrialHandler’s attributes do not change throughout the experiment; rather, they are determined at the beginning of the experiment. @rob-linguistics13’s and @dvbridges’ final determination of the scheduler seems to work for TrialHandler (although I haven’t tried it myself - I have three different places in my experiment where I would need to use it and my priority is debugging the exposure routine), but I’m wondering – is this an issue for anything else? Could this relate to the issues I’m having with the conditional calling for my ding and keyboard components?

(also a very good chance I’m 110% off base but I figured I’d throw it out here just in case I am not!)

Thank you all so, so much!

Looking at the code I see a couple potential issues.

One is that you might be doing more than you need to with the code component. Your code component asks “ding” to play, but you are also setting a condition for “ding” to play in the builder. So if you look at the builder-compiled JS (https://gitlab.pavlovia.org/learningdevlab/v17_i_give_up/blob/master/html/all2afc_backbone.js) you’ll notice on lines 2627-2638:

    if ((timetoDing = true) && ding.status === PsychoJS.Status.NOT_STARTED) {
      // keep track of start time/frame for later
      ding.tStart = t;  // (not accounting for frame time here)
      ding.frameNStart = frameN;  // exact frame index
      
      psychoJS.window.callOnFlip(function(){ ding.play(); });  // screen flip
      ding.status = PsychoJS.Status.STARTED;
    }
    if (t >= (ding.getDuration() + ding.tStart)     && ding.status === PsychoJS.Status.STARTED) {
      ding.stop();  // stop the sound (if longer than duration)
      ding.status = PsychoJS.Status.FINISHED;
    }

and then in the same function, on lines 2663-2680:

if ((repeat == "r") && (exposure_sound.status === PsychoJS.Status.FINISHED))
        {
        let repeatedRespKeys = repeated_resp.getKeys();
        if (repeatedRespKeys[0] == 'y')
            {
            timetoDing = true;
            ding.play();
            ding.getDuration(2);
            ding.PsychoJS.status = STARTED;
            continueRoutine = false;
            }
        }
    else if ((repeat !== "r") && (exposure_sound.status === PsychoJS.Status.FINISHED))
        {
        timetoDing = false
        ding.PyschoJS.status = NOT_STARTED;
        continueRoutine = false;
        }

There’s a few problems. You’re setting the status to “STARTED” rather than “PsychoJS.Status.STARTED”, so it’s possible that it’s calling “ding” twice then timetoDing is set to true. You may not need to tell it to actually play ding in your code component at all if you leave that conditional in, it looks like it will just figure itself out.

On the other hand, if you want to control it exclusively through the code component, my recommendation would be to change the status to PsychoJS.Status.STARTED and add this from the auto-generated code:

      psychoJS.window.callOnFlip(function(){ ding.play(); });  // screen flip

I don’t know how much that actually makes a difference, but it can’t hurt.

I’m also, now that I’m looking at the compiled JS code, realizing a few problems in how I was thinking about getKeys. I have a better idea of why it’s not working now, it involves keyboard buffers being cleared and some naming conventions I wasn’t paying close enough attention to. I think you do want to turn “force end of routine” off, but then you should change the code component again (sorry, I think I’ve led you astray several times because I didn’t look closely enough at how keyboard components work):

if ((repeat == "r") && (exposure_sound.status === PsychoJS.Status.FINISHED))
        {
        if (repeated_resp.keys == 'y')
            {
            timetoDing = true;
            // include the following only if you want to control the sound entirely through the code component.
            ding.play();
            ding.getDuration(2);
            ding.PsychoJS.status = PsychoJS.Status.STARTED;
            psychoJS.window.callOnFlip(function(){ ding.play(); }); 
            // end lines to include
            //continueRoutine = false; I think this is ending the thing immediately, but in principle if all the components have the status "finished" then it should end the routine anyway
           //if not, we'll need another condition to check when ding ends and end it then.
            }
        }
    else if ((repeat !== "r") && (exposure_sound.status === PsychoJS.Status.FINISHED))
        {
        timetoDing = false
        ding.PyschoJS.status = PsychoJS.Status.FINISHED;
        continueRoutine = false;
        }
}

Right now, before getting to your code component, it’s doing the “getKeys” call, which is clearing the key buffer, so calling it again in the code component will return nothing. However, it should be storing the last key pressed as repeated_resp.keys. If the code above doesn’t work try repeated_resp.keys[0] == ‘y’ instead (I’m not sure if it’s an array or a string).

I think this will work but unfortunately do not have time to test it myself. Let me know if that at least changes the behavior.

Hi @jonathan.kominsky!

Thank you so much for your time taking a look at my code! I am extremely happy to report that I was able to solve this issue using the following code:

    if ((repeat == "r") && (exposure_sound.status === PsychoJS.Status.FINISHED))
        {
        if (repeated_resp.keys == 'y')
            {
            timetoDing = true;
            if (ding.status === PsychoJS.Status.FINISHED)
                {
                continueRoutine = false;
                }
            }
        }
    else if ((repeat !== "r") && (exposure_sound.status === PsychoJS.Status.FINISHED))
        {
        timetoDing = false;
        continueRoutine = false;
        }

My ding sound component is set to start if timetoDing==true and the keyboard response repeated_resp is conditional on if repeat==“r”.

By putting continueRoutine = False line in its own conditional statement, like you suggested, the routine now works just the way I want it to.

Thank you SO much for you help with this! I very much so appreciate it!!! Hopefully this can be of help to others using conditional key responses in online experiments as well.

1 Like