Adventures in testing an experiment with Unittest

Coding
1

Hello all. Auto-testing software is a great thing - it avoids hours of manual testing, allows you to keep all your testing requirements in code, and (most usefully) allows you to easily re-run a full set of tests after you make a small change to your code.

This is how we have been doing testing on a simple experiment which displays images, asks visual search questions and collects keypresses and RTs.

The first step is to make sure that your code can be easily called by the test framework. So we encapsulated the experiment inside a function:

def run_blocks(trials,noise,win,expInfo,incorrect,tone1,tone2,experiment_details,allPoints,n_blocks,trials_per_block):

…to which we pass all the necessary details.

The real experiment file, experiment.py, simply imports this function loads up the trial data and audio stimuli and calls this function.

Our testing file, test.py, imports this function too, and then defines a set of tests using the Unittest framework. Each one looks like this:

class reaction_time_zero(unittest.TestCase):
    @mock.patch('keyboard.wait_keys')
    @mock.patch('keyboard.get_keys')
    def runTest(self, mock_get_keys, mock_wait_keys):
        print 'TEST: ' + self.__class__.__name__

        experiment_details = {}
        trials = {}
        n = 1

        trials[1] = {}
        trials[1]['image_name'] = '2008_000177.jpg'
        trials[1]['present_absent'] = 'No Target'
        trials[1]['trial_type'] = 'Critical'
        trials[1]['tone_hz'] = 'tone1'
        trials[1]['tone_onset'] = 500

        mock_wait_keys.return_value = [['q', 0]]
        mock_get_keys.return_value = [['space', 0]]

        data = run_blocks(trials,noise,win,expInfo, incorrect, tone1, tone2, experiment_details,allPoints,1,n)

            t = data[1]
            print t
            self.assertEqual(t['RT_TO'], 0)
            self.assertEqual(t['tone_sdt'], 'HI')

Each test has three parts: setup, running and checking the results. This test just runs one trial: setup is done by setting the trial details:

        trials[1] = {}
        trials[1]['image_name'] = '2008_000177.jpg'
        trials[1]['present_absent'] = 'No Target'
        trials[1]['trial_type'] = 'Critical'
        trials[1]['tone_hz'] = 'tone1'
        trials[1]['tone_onset'] = 500

Next, we want to simulate keypresses. We do this by locating the function which would wait for keypresses (getKeys or waitKeys) and replacing it with a fake test function which instantly returns the key we specify. This is called mocking.

Now, we could set up a mock for getKeys with @mock.patch(‘psychopy.event.getKeys’), but it turns out that waitKeys also calls getKeys, so we wouldn’t be able to specify different behaviours for waitKeys and getKeys.

We solve this by simply wrapping getKeys in our own function, which in the normal experiment will pass straight through to getKeys:

def get_keys(keyList, timeStamped):
    return event.getKeys(keyList=keyList, timeStamped=timeStamped)

we can then mock this using @mock.patch(‘keyboard.get_keys’).

So we set up two mocks:


@mock.patch('keyboard.wait_keys')
@mock.patch('keyboard.get_keys')

which we then pass into the test function in reverse order (Python decorators, like these mock setups, are evaluated in reverse order):

def runTest(self, mock_get_keys, mock_wait_keys):

Next, we have to specify the behaviour of the mocks. We can specify a constant return value with

 mock_wait_keys.return_value = [['q', 0]]

or, if we want wait_keys to return a sequence of different values, we can use

mock_get_keys.side_effect = [ [],[['space', 1]] ]

(Returns nothing the first time it’s called, then a space. Be sure not to run out of return values.)

Next, we call the run_blocks function, passing it the trials we just set up. The experiment is now run and, due to the mocks, we don’t need to press any keys.

Finally, we assert a couple of things about the first trial:

self.assertEqual(t['RT_TO'], 0)
self.assertEqual(t['tone_sdt'], 'HI')

…and then the test is finished.

You can define multiple tests like this, each in their own class, and run them all with

if __name__ == '__main__':
     unittest.main()

Has anyone else used this approach? The satisfaction you get from watching your experiment complete a full hour’s run on its own - and the confidence that it’s actually collecting the right data, even after changes to the code - are great.

6 Likes
2

Very nice indeed, thank you! I wanted to try unittesting psychopy for a long time. Still not very clear with the idea of mocking – seems a bit like magic to me :slight_smile:

3

Mocking is just “replace this object with a fake object that responds to the call I expect the real object to receive.” This way you can use the mock object to check that the call was sent in the right way, and the mock doesn’t do what the real object would have done (for example, fire the missiles) which you don’t want to happen during test.

1 Like
4

This turned out to be exactly what I was looking for my project. Thanks suggesting this! By the way, do you also include accuracy tests, meaning that you have code to make sure that e.g. stimuli are presented when they should be?

5

I haven’t done that so far but it would be sensible