Unable to show unicode from excel files

Coding
1

I have written an experiment in which the subject has to choose between to unicode characters. The content of the text comes from an excel condition file. It works perfectly on Windows 7 and 8.1 but when I try to run it in a laptop with Windows 10 all the unicode characters are shown as squares. First time I tried using the latest psychopy version but then I downgraded to 1.83.04 because that is the one running on the 8.1 computer.
The script starts with

#!/usr/bin/env python
# -*- coding: utf-8 -*-

The encoding of the excel files have not changed.

Any ideas why the code does not work in the new laptop?

1 Like
2
  1. What happens with a .csv format conditions file?
  2. Try uploading a minimal example here so the problem can be reproduced.
3

If I save a unicode tex file in Excel instead of xlsx (I use unicode text because CSV option turns unicode into “?”) I got some errors:

################## Running: E:\jap_trial\jap_trial_lastrun.py ##################
pyo version 0.6.6 (uses single precision)
Traceback (most recent call last):
  File "E:\jap_trial\jap_trial_lastrun.py", line 76, in <module>
    trialList=data.importConditions(u'jap_loop.csv'),
  File "C:\Program Files (x86)\PsychoPy2\lib\site-packages\psychopy-1.83.04-py2.7.egg\psychopy\data.py", line 1980, in importConditions
    trialsArr = read_csv(fileUniv, encoding='utf-8')  # use pandas reader, which can handle commas in fields, etc
  File "C:\Program Files (x86)\PsychoPy2\lib\site-packages\pandas\io\parsers.py", line 474, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "C:\Program Files (x86)\PsychoPy2\lib\site-packages\pandas\io\parsers.py", line 250, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "C:\Program Files (x86)\PsychoPy2\lib\site-packages\pandas\io\parsers.py", line 566, in __init__
    self._make_engine(self.engine)
  File "C:\Program Files (x86)\PsychoPy2\lib\site-packages\pandas\io\parsers.py", line 705, in _make_engine
    self._engine = CParserWrapper(self.f, **self.options)
  File "C:\Program Files (x86)\PsychoPy2\lib\site-packages\pandas\io\parsers.py", line 1072, in __init__
    self._reader = _parser.TextReader(src, **kwds)
  File "pandas\parser.pyx", line 509, in pandas.parser.TextReader.__cinit__ (pandas\parser.c:4748)
  File "pandas\parser.pyx", line 663, in pandas.parser.TextReader._get_header (pandas\parser.c:6603)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xff in position 0: invalid start byte

Previusly when using the xlsx I got no errors but all the letters were squares.

I have made in the builder a simplified version with the same problem (runs on my Win 8.1 laptop but not in de Win 10):
jap_trial_lastrun.py (8.4 KB)
jap_trial.psyexp (5.8 KB)

I attach also the conditions
jap_loop.csv (30 Bytes)
jap_loop.xlsx (8.7 KB)

Thank you

4

The Excel file itself is fine, and pandas 0.18.1 loads it flawlessly for me in the Jupyter Notebook. In my terminal window I only get ?'s, though, so this appears to be a problem with the fonts used.

5

This shouldn’t be related to your problem.

6

I wrote it just in case some one suspected that was the problem

7

That actually helped!

I do not understand exactly why but it works now.

I installed Korean and Japanese fonts. I choosed Meiryo for the stimuli and worked. This is strange for me because under Excel all the strings are corretly shown in Arial but not in psychopy using the same font.

Now the instructions are shown in Arial and the stimuli in Meiryo (Meiryo also work for latin instructions but they do not fit in the original place)

Thank you

1 Like
8

Awesome. Don’t know why PsychoPy cannot use Arial, though…

9

My guess is that Arial font doesn’t really include korean or japanese characters. I imagine Excel detects that and switches (silently) to an alternative when it sees your characters. PsychoPy isn’t that smart!

10

Actually from this link I’ll change my theory:
http://www.wazu.jp/gallery/views/View_ArialUnicodeMS.html

Maybe there’s a version of Arial that supports kanji but that’s a different font file from the one psychopy is using.

1 Like
11

I do not really know. The thing is under linux (Mint 13 and Mint 14) and Windows 7 and 8.1 Arial works fine both with kanji and Korean. But using the same version of psychopy under Windows 10 it does not. It would be good to know why to avoid possible upcoming issues. On the other hand, I am quite happy because today I used the task on a real participant and everything was as expected.

1 Like
12

I have created an issue:
https://github.com/psychopy/psychopy/issues/1324

13

I wonder if Windows 10 ships with a different Arial font :slight_smile:

14

I am having unfortunately the same problem, but up to now, none of the suggested solutions worked for me.
My experiment uses .xlsx condition files. Changing them to .csv files does not work. The error I get is "UnicodeDecodeError: ‘utf8’ codec … (see picture attached).

Fehlermeldung_UnicodeDecodeError
Fehlermeldung_UnicodeDecodeError716×367 17.4 KB

The experiment runs fine on a laptop with windows 7 in version 1.83.04. But I am using a new configured computer (also Windows 7) and the PsychoPy Version 1.85.3.

Any ideas and suggestions why I get this error?

15

This doesn’t seem to be the same issue at all. I would start a new post and include more information, there’s not much here to go on.

16

@daniel.riggs1 is right, your problem is actually likely related to this thread: