Waitblanking with multiple active threads

Coding
1

Hi there,

I am completely new to PsychoPy, so apologies if I’m asking obvious things, however nothing jumped out at me in the help files / forum searches / googling. What I’m hoping to find out is whether PsychoPy can solve my particular issue, and so would be worth investing a lot more time into working with the program in order to actually make it work.

I’m currently designing a series of experiments which are actually presented and controlled using a 3D gaming engine (Unity). This Unity-created experiment communicates via UDP with a helper app, coded in Python, for various purposes. In particular, event codes are sent from Unity to the Python app, which then forwards them to other experimental hardware.

Ideally, the code would be sent as close as possible to the beginning of the first screen refresh of the event, and if latency is unavoidable this should be as constant as possible. If I understand correctly, the standard solution implemented by various software packages (PsychoPy, Vision Egg, Eprime, etc) is to prime the code to be sent as soon as the next vertical refresh occurs. And in psychopy, this involves waitblanking being set to true, so that win.flip commands will essentially pause processing until this flip occurs.

So in the simplest case that I can see, Unity prepares a frame, sends the event code to Python, Python then waits until the next screen refresh and sends the code immediately after this.

However, to complicate matters, the Python app is also running various other threads, in particular checking for messages over a serial port which are coming in at 300hz, so much faster than a screen refresh (these are standard 60Hz monitors). If this is the case, am I going to get serious issues with waitblanking?

It does not strike me as a problem if the entire app “pauses” while waitblanking is doing its thing. Any incoming serial messages will be stored in the system buffer, and can be read once waitblanking is finished. It’s also not a problem if the other threads keep running as usual while just the thread handling the event code communication pauses to wait for the next blank. But… I am aware these are very dangerous waters, and I definitely lack some experience here, so any advice/comments would be appreciated.

On a related but side note: If I understand correctly, under the best of all possible circumstances, anyone using a consumer-grade monitor should expect up to 10ms jitter when using waitblanking (or any similar methods used by Vision Egg or Eprime or whatever). Is this correct? Can nothing be done without going back to CRTs?

Thanks,

Marcus

2

When using threads:

  • make sure that only one thread is allowed to draw/flip etc. If OpenGL is called from multiple threads mayhem can ensue
  • other threads can certainly run faster than 60Hz and they shouldn’t be forced to run only once per 1/60th second so you need to bear that in mind. e.g. check the parallel port at 300hz but in your main (60hz) thread you should bear in mind that 5-ish data values may have occured and choose what you do to handle that

Most consumer grade computer monitors won’t show the timing jitter you’re talking about. Hard to know from this information where the jitter lies (screen or graphics card). How are you measuring it?

3

Thanks for the reply.

One thing I did not mention is that I’m using two monitors here (one for the participant, one for the experimenter). The participant display (let’s call it 1) is being controlled by the Unity3D gaming engine. Ideally the experimenter display (2) would be showing information of various kinds that the experimenter would be interested in (trial number, accuracy, etc), and this would be controlled by PsychoPy. So PsychoPy has no direct control over Display 1. However, event codes should be synced to frame onsets on that display. (I realize this is a pretty unusual setup, and it may seem needlessly complicated, but it’s one that I think has the potential to be very powerful for many future experiments, allowing us to leverage the realtime 3D capabilities of powerful game engines AND the temporal precision of PsychoPy.)

So… in the psychopy code, what would be ideal is a thread that does nothing but wait for event codes on Display 1, and when it gets them it waits for the next flip, and then sends the event code on to our hardware without actually drawing anything? There would be other threads involved in making things happen on Display 2, and reading/writing various pieces of data, but none of that should affect the event code thread.

As for the timing jitter, I will be measuring it using photodiodes connected to an Arduino that gives accurate timestamps, but have not yet done so. Fingers crossed that it isn’t a problem!

Thanks again, sorry for the wall of text here.

4

I don’t understand. If PsychoPy isn’t drawing to that screen (specifically if it isn’t calling win.flip() on that screen) then you don’t get any info about its frame syncing. I’m afraid I don’t have any further advice for you. You need to think very carefully about how your threads will behave but these are hard things to debug without detailed knowledge of your exact set up (which I wouldn’t have time to learn about).

OK, if you’re measuring jitter on the arduino that psychopy isn’t detecting then it’s most likely your flat panel screen doing ‘post-processing’ to make your image look nice and this is adding a variable lag. As I say most flat screens I’ve used don’t have this problem but it’s hard to know in advance which are a problem. You’ll need to try with different monitors (but no stimulus delivery software can alter the issue)

5

Jon,

OK, thanks for the reply again. It looks like I may be able to get Unity to send out codes directly after frame blanks, in which case I don’t have to go through this esoteric way of doing things. Fingers crossed, anyways.

6

Jon,

Thanks again for your previous replies.

One more stab at this. I’m definitely going about this in a weird manner, and I’m doubtful that anyone has tried this, so I understand that a shrug might be the best you can give me. (And I also understand it’s not your job to be my tech support.) But that being said…

I’m now considering writing a PsychoPy app that opens a window on my Display 1, which is invisible to the user because it appears behind the Unity window. Prior to Unity computing anything in a frame where the stimulus is to appear, I send a code via UDP from Unity to Python, which then changes the colour of its hidden window (or does something else that involves a draw command, whatever is fastest), and sends an event code over the serial port to the experimental hardware immediately after the flip() command.

I realize this sounds like madness, but I have run a test using another python app receiving UDP messages from Unity, and the latency from send to receive time is generally very rapid, giving PsychoPy plenty of time to run its draw command and wait for the screen flip. However that being said, there are a lot of moving parts here and plenty of stages where things could go wrong.

The specific question, then, is simply whether there is an obvious problem with drawing to a window that is underneath another window? So far as I am aware, draw commands to these windows are still computed.

The more general question is whether there is anything else glaringly stupid about this plan, before I throw myself into a guaranteed failure. As I said, I understand that this isn’t a paid tech support forum and so I am more than ok with the answer being “I have no idea” or simply silence, but if there are any obivous things I haven’t considered or alternatives I’d be very grateful.