Textbox input Unicode issue

Builder
1

OS (e.g. Win10): MacOS 10.13.6
PsychoPy version (e.g. 1.84.x): v2020.2.5
Standard Standalone? (y/n) If not then what?: Y
What are you trying to achieve?:
Use new textinput Response function with Korean, Japanese, and Indian Input.

What did you try to make it work?:
Without font support, just blank box appears. After select proper font, the typed input string shows up in experiment screen. However, it cannot be saved, and the experiment hangs when it finished and trying to save the result.

What specifically went wrong when you tried that?:
Include pasted full error message if possible. “That didn’t work” is not enough information.

It seems the old ‘unicode input error’. I mentioned before. I guess the only the English and other Ratin-based language is usable for textbox.

Traceback (most recent call last):
File "ctypes/callbacks.c", line 234, in ‘calling callback function’
File “/Applications/PsychoPy_2002.app/Contents/Resources/lib/python3.6/pyglet/libs/darwin/cocoapy/runtime.py”, line 1121, in objc_method
result = f(py_self, *args)
File “/Applications/PsychoPy_2002.app/Contents/Resources/lib/python3.6/pyglet/window/cocoa/pyglet_view.py”, line 164, in pygletKeyUp
symbol = getSymbol(nsevent)
File “/Applications/PsychoPy_2002.app/Contents/Resources/lib/python3.6/pyglet/window/cocoa/pyglet_view.py”, line 52, in getSymbol
return charmap[chars[0].upper()]
KeyError: ‘\u315b’

2

I’m not quite clear what you mean by this (particularly as the error messages all refer to pyglet). Are you using the new TextBox2 stimulus to capture editable text automatically, or using the old approach, which required custom code, and the old TextStim class?

3

I am using textbox2 via builder GUI and facing the same problem
(the string input is visible with error log per every keypress), and the experiment freeze when it tries to save the result.

4

In your experiment settings, what formats are you saving in? If the text renders but won’t save then it may be that the unicode contents of the Textbox aren’t being stored properly in the .csv, .xlsx, etc. by whatever function is writing to those docs

5

The default csv setting.
I didn’t modified the save setting.

6

In that case it’s most likely that there’s something in the saveAsWideText function which uses some function which is incompatable with Koren characters… I’ll make some experiments to test this out to see where it breaks then get back to you!

7

I’m struggling to replicate this error - what characters and font did you use specifically? When I tried just now with the text 사이코 파이 (“PsychoPy” according to Google Translate) in Black Han Sans it saved just fine, so I’m guessing it’s either a specific character in your string or something to do with your font file

8

Hi, @TParsons,

Thank you for the investigation.
Are you using Windows or MacOS?
Since, as far as I know, MacOS use very strange Korean key input both OS level and Python backend of PsychoPy.

9

I use Windows 10, @Michael also raised that some Hindi characters don’t render in Textboxes on Mac too:


So I think you’re right that it’s something to do with how Macs handle key inputs.

I’ve added a test to our test suite to check that all unicode characters are saved correctly:


and that’s passing even on the Linux distro we run automatic testing on, so I’ll pass this to a Mac user and get them to try it out.
10

@TParsons
Thanks for the update.
The new version (2020.2.10) seems work in Hindi and Korean, but when you type more than certain numbers of strings, it will crash.

It’s due to pyglet issue (I am not sure it’s only Mac related or same thing happens in Linux and Windoows), not PsychoPy Itself.

See below error.
The error happens per every keypress of Non-alphabetic character.

strong text
@jon John, I guess the issue is related to the way Pyglet deal with keypress or the ‘middleware’ between Pyglet and PsychoPy.
Due to the errors, the textbox2 stops when the cumulated numbers of error in certain number.

Bests,
J.

Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/Applications/PsychoPy.app/Contents/Resources/lib/python3.6/pyglet/libs/darwin/cocoapy/runtime.py", line 1128, in objc_method
    result = f(py_self, *args)
  File "/Applications/PsychoPy.app/Contents/Resources/lib/python3.6/pyglet/window/cocoa/pyglet_view.py", line 199, in pygletKeyDown_
    self._window.dispatch_event('on_key_press', symbol, modifiers)
  File "/Applications/PsychoPy.app/Contents/Resources/lib/python3.6/pyglet/window/__init__.py", line 1333, in dispatch_event
    if EventDispatcher.dispatch_event(self, *args) != False:
  File "/Applications/PsychoPy.app/Contents/Resources/lib/python3.6/pyglet/event.py", line 422, in dispatch_event
    self._raise_dispatch_exception(event_type, args, getattr(self, event_type), exception)
  File "/Applications/PsychoPy.app/Contents/Resources/lib/python3.6/pyglet/event.py", line 476, in _raise_dispatch_exception
    raise exception
  File "/Applications/PsychoPy.app/Contents/Resources/lib/python3.6/pyglet/event.py", line 415, in dispatch_event
    if getattr(self, event_type)(*args):
  File "/Applications/PsychoPy.app/Contents/Resources/lib/python3.6/psychopy/visual/backends/pygletbackend.py", line 348, in onKey
    thisKey = pyglet.window.key.symbol_string(evt).lower()
  File "/Applications/PsychoPy.app/Contents/Resources/lib/python3.6/pyglet/window/key.py", line 145, in symbol_string
    if symbol < 1 << 32:
TypeError: '<' not supported between instances of 'NoneType' and 'int'
11

I get the same error as you do, but with Swedish/umlaut characters (å, ä, ö) on a Macbook with MacOS Mojave 10.14.6 and PsychoPy 2020.2.5. When I run the exact same experiment on a Windows 10 computer, there aren’t any issues.

On my Mac, I tried running the below script directly from the terminal, and in the terminal at the bottom of PsychoPy Builder’s code window:

import pyglet
window = pyglet.window.Window()
label = pyglet.text.Label('öööööääääääääååååååååå',
                      font_name='Arial',
                      font_size=36,
                      x=window.width//2, y=window.height//2,
                      anchor_x='center', anchor_y='center')

@window.event
def on_draw():
    window.clear()
    label.draw()

pyglet.app.run()

This actually works as expected and I’m not shown any errors. If I use ‘사이코 파이’ instead, the output doesn’t display the characters properly, instead cutting them off, but I still don’t get any errors.

So it seems like it’s not just the interaction between pyglet and MacOS itself that’s causing problems, unless I’m missing something. It looks more like there’s something about how PsychoPy processes the data before they are sent to pyglet, or how PsychoPy uses pyglet. The " KeyError: ‘\u315b’ " (I get the same, but with \xd6 or something like that) message makes it sound like characters have already been converted before the strings are handed off to pyglet.

Edit: I just noticed that the same error appears even if just using a regular keyboard component.

[...]
KeyError: '\xc5'
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/Applications/PsychoPy.app/Contents/Resources/lib/python3.6/pyglet/libs/darwin/cocoapy/runtime.py", line 1121, in objc_method
    result = f(py_self, *args)
  File "/Applications/PsychoPy.app/Contents/Resources/lib/python3.6/pyglet/window/cocoa/pyglet_view.py", line 164, in pygletKeyUp_
    symbol = getSymbol(nsevent)
  File "/Applications/PsychoPy.app/Contents/Resources/lib/python3.6/pyglet/window/cocoa/pyglet_view.py", line 52, in getSymbol
    return charmap[chars[0].upper()]
KeyError: '\xc5'