Pixel color reading procedure in online experiment

[toto]

Hello good people!

URL of experiment:

Sign in

Description of the problem:
In my experiment, participants have to click on a color wheel to choose a right color.

Desktop version works great: a radialStim is built at each trial. Then the ‘_getFrame’ function is applied to save the whole response screen as an NumPy array of RGB values.
Finally, the RGB values of the clicked pixel are received through simple indexing (I look at the image pixel located at the same x-y index as the chosen pixel x-y index on the window)

Also, it is possible to use the image file for response instead of the radialStim. The RGB value of the image pixels are extracted beforehand and saved in a NumPy array to be used for indexing when a pixel is chosen. It also works well.

Unfortunately, both solutions do not work online.

1. First, the ‘_getFrame’ - like functionality seems to be absent in psychoJS.

2. When I try to do a translation of a nested array with image RGB values from Python to JS in a builder - the system gets stuck (the array is rather massive (1920 X 1080 X 3)). Besides, I did not find a JS library that will extract the RGB values from an image in a straightforward way without implying the html mechanisms.

So I decided to use html functionality to draw the image on the canvas in the beginning of experiment and read-save the RGB values of it’s pixels into array to be used at response stage.

The problem is that I am not sure what is the right place to locate the following javascript code that manipulates the canvas :

          const canvas = document.querySelector('.myCanvas'); 
	  const width = canvas.width = 1920; // window.innerWidth;
	  const height = canvas.height = 1080; // window.innerHeight;
	  const ctx = canvas.getContext('2d');		
	  ctx.fillStyle = 'rgb(231,15, 144)';
          ctx.fillRect(0, 0, width, height);

	  let image = new Image(1920, 1080);
          image.src = 'wheel_v7s9_fullsize.png';
	  image.onload = function() {
          ctx.drawImage(image, 0, 0);
	  }
	  imageData = ctx.getImageData(0, 0, ctx.canvas.width, ctx.canvas.height)

When I put it in the body of index.html - the experiment fails at response screen (line 400) and I receive the following error:

core-2020.2.js:1846 ReferenceError: imageData is not defined
at Scheduler._currentTask (js_play.js:400)
at Scheduler._runNextTasks (util-2020.2.js:1570)
at Scheduler._runNextTasks (util-2020.2.js:1574)
at update (util-2020.2.js:1527)
window.onerror @ core-2020.2.js:1846
log4javascript.min.js:1

FATAL unknown | {}
BrowserConsoleAppender.append @ log4javascript.min.js:1

---

this is line 400, where it falls: var index = (x_corner + y_corner * imageData.width) * 4;

My knowledge of web technologies is definitely not enough to make it work …
Could somebody give me a direction please?

Thanks and happy holidays to all!

[thomas_pronk]

Happy holidays!

Kudos for your inventiveness, but the way you’re approaching it now is quite outside of the PsychoJS framework, so that could make things more difficult when you’d like to use the selected color in successive stages in your experiment. I’m asking around how you could do it inside of the framework. More soon…

Best, Thomas

[toto]

Hi, Thomas!
Thank you very very much!

I actually found a kind of awkward solution - I use imageio library to extract the rgb values into a nested list in python. The resulting structure is very massive and gets the system stuck. So I iterate over it , removing the background grey ([128,128,128]) pixels and adding the remaining colored pixels to the dictionary ({y, x: [r, g, b]}). Further, to make it even more lightweight, i remove most of the pixels from the left and right side of the wheel, leaving only two pixels for each row(y) - one from the left, and one from the right. the pixels from the top and bottom are left untouched.

__ = imageio.imread('wheel_v7s9_fullsize.png')
wheel_rgb = np.ndarray.tolist(__)
wheel_dict = {}
for y in range(1080):
    if y <= 200 or y >= 860:
        for x  in range(1920):
            if wheel_rgb[y][x] != [128, 128, 128]:
                wheel_dict.update({str([y, x]): wheel_rgb[y][x]})
    else:
        row_list = []
        for x in range(1920):
            if wheel_rgb[y][x] != [128, 128, 128,]: 
                row_list.append(wheel_rgb[y][x])
        row_list[0:20] = []
        row_list[20:-20] = []
        row_list[-19:] = []
        for i, value in enumerate(row_list):
            wheel_dict.update({str([y, i]): value})

It works meanwhile. I copy paste this ‘trimmed’ dictionary to the psychojs file as a js object and use the x,y values provided during the response stage as the keys to search for corresponding color value in it.
Do you think I can continue like this?

Thanks a lot !

[thomas_pronk]

Lovely solution! You’re providing fixes on a level I don’t have much to add to. So heck yeah, go for it

[toto]

Thanx a lot Thomas!