Sync while-loop iterations to frame rate

Coding
,
1

This is a cross post from here, for posterity sake, and because I think/hope that the solution lies in Psychopy. I’m using a webcam to record a 30-sec video; however, the resulting video ends up being ~43-44 seconds. Here is my code:

from psychopy import locale_setup, visual, core, event, data, gui, microphone
import cv2

#Set up Timer:
trialTimer = core.Clock()

#Begin Video Recording:
cap = cv2.VideoCapture(0 + cv2.CAP_DSHOW)
fourcc = cv2.VideoWriter_fourcc(*'XVID')
out = cv2.VideoWriter('{}.avi'.format(video_path),fourcc, 20.0, (640,480))

trialTimer.reset()

while trialTimer.getTime() < 30:
    ret, frame = cap.read()
    out.write(frame)

#Stop video recording:
cap.release()
out.release()
cv2.destroyAllWindows()

The third argument in the cv2.VideoWriter function specifies the frame rate, which is default at 20. This provides a smooth, fluid video, but one that is longer than 30 seconds. Adjusting this value to 29.433 makes the video 30 seconds, but it appears sped-up. The issue seems to be that I can’t constrain the while loop iterations to the frame rate or something else. Thus the frames are added to the video at a rate that I can’t control.

This post was helpful for getting the webcam recording to work; however, I made some changes since I’m not presenting the video, just saving it. I’ve tried a few things with no luck, and was wondering if someone has a solution to this issue.

Thanks for the help.

Dan

2

As indicated in the StackOverflow post, the issue is likely that the iterations of the while loop are not governed in any way. You should insert a counter in that loop to monitor how many frames you are grabbing within the 30 s loop, which is also likely to vary somewhat from one run to another. One solution would be to create a PsychoPy window and issue a win.flip() on every iteration. This would slow the loop to a hardware cycle (your screen refresh rate). Your counter should then show a reliable rate of, say, 60 Hz, assuming that the read and write functions can be completed within a 16.7 ms period. You could then tell the video writer to save the video at that frame rate, so the playback speed would then match the recorded speed.

Alternatively, you could use timers, but that gets tricky, as you’ll need to account for temporal slippage as the code runs. The naturally discretised time line provided by syncing to the screen refresh rate solves a lot of this for you.

3

Thanks @Michael

I should’ve indicated before that participants are viewing and describing images on a screen for 30 seconds. This means that in addition to video recording I’m also recording audio. I was able to adjust my while loop so that I can limit the video recording to 30 seconds; this entailed ending the loop once 600 frames are reached (with 20 fps). My lingering issue though is that the microphone recording is 21-sec instead of 30 (before it was 30-sec, but the video was 43-sec).

Here’s an updated (and more detailed) version of my code:

from __future__ import division, print_function
from psychopy import prefs
prefs.hardware['audioLib'] = ['pyo']
from psychopy import locale_setup, visual, core, event, data, gui, microphone, sound
import os, cv2

#Window:
win = visual.Window(size = (1200,1200), fullscr = True, pos = (0,0), units = "height", color = [-1,-1,-1])

#Turn off Mouse:
event.Mouse(visible = False)

#Set base variables:
width, height = [1200,1200]
text = visual.TextStim(win=win, text='', height=height/22, pos=(0,0), color="White")
microphone.switchOn()
mic = microphone.AdvAudioCapture()

for i in range(counter, len(stim_df)):
        image = visual.ImageStim(win=win, pos=(0,0), size=(0.55,0.55), image='Image{}'.format(i))

        #Begin audio recording:
        mic.reset()
        mic.record(sec=30, filename='audio_file{}'.format(i))
        
        #Begin Video Recording:
        cap = cv2.VideoCapture(0 + cv2.CAP_DSHOW)
        fourcc = cv2.VideoWriter_fourcc(*'XVID')
        out = cv2.VideoWriter('video_file{}.avi'.format(i), fourcc, 20.0, (640,480))

        #Present Image:
        theseKeys = event.clearEvents(eventType = "keyboard")
        image.setAutoDraw(True)

        frames = 0
        while True:
            ret, frame = cap.read()
            out.write(frame)
            win.flip()
            frames += 1

            if frames > 600:
                break

        #Stop audio & video recording after each trial:
        image.setAutoDraw(False)
        cap.release()
        out.release()
        cv2.destroyAllWindows()
        mic.stop()

The code above keeps my video at 30 seconds and appears smooth; however, the audio is only 21-seconds and can’t be synced to the video. I’m unsure which microphone parameter(s) to change to be at the same rate as the video capture.

4

Just wanted to ping this to see if anyone knows of a solution for syncing the video webcam and microphone capture

5

@dlevitas, one workaround is to capture the screen with some reliable software (I used MediaLooks for VirtualDub). The trick is that this software records both screen and desktop sound (as in streaming software nowadays), and they are in sync. That means you can fire a sync tone for the Microphone class in psychopy, which will be recorded both in your screen recording AND in sound channel you’re saving.
The picture shows a slightly more broad problem: syncing two multi-channel data streams to each other; still involving yours, too. Note how microphone capture is synced to video source. Also note that error will depend mostly on your video framerate.

DataCollectionScheme
DataCollectionScheme2268×907 56.4 KB