[issue26226] Various test suite failures on Windows when computer name contains a non-ascii character

[ppperry]

Changes by ppperry :


--
title: Various test suite failures on Windows -> Various test suite failures on 
Windows when computer name contains a non-ascii character

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26226] Various test suite failures on Windows

[Emanuel Barry]

Emanuel Barry added the comment:

That is good to know, thanks Eryk for the explanation! I also think making 
Python set LC_CTYPE on startup on Windows would be good.

Martin's comment got me wondering, so I'm going to try to see if all modules 
properly got re-compiled, and re-run tests. I'll also run test_distutils 
through the command line to see if it passes. Following that, I'll branch off 
the relevant issues; and hopefully close this one too (I realize that it 
started going in all directions, it's hard to keep track even for me now).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26226] Various test suite failures on Windows

[Eryk Sun]

Eryk Sun added the comment:

time.strftime calls the CRT's strftime function, which the Windows universal 
CRT implements by calling wcsftime and encoding the result. The timezone name 
is actually stored as a char string (tzname), so wcsftime has to decode it via 
mbstowcs. 

The problem is that in the C locale tzname is an ANSI (1252) string while 
mbstowcs simply casts to wchar_t, which is the same as decoding as Latin-1. 
This works fine for "é" (U+00E9). But the right single quote character (U+2019) 
is "\x92" in 1252, and a simple cast maps it to the non-character U+0092. 

When the CRT's strftime encodes this back as an ANSI string, it maps U+0092 to 
the replacement character for 1252, a question mark. Similarly, time.tzname 
decodes the tzname ANSI strings using mbstowcs, with the same mismatch between 
LC_CTYPE and LC_TIME, resulting in the string "Est (heure d\x92été)"

In summary, the problem is that LC_TIME uses ANSI in the C locale, while 
LC_CTYPE uses Latin-1. A workaround (in most cases) is to delay importing the 
time module until after setting LC_CTYPE (also setting LC_TIME should cover all 
cases). For example:

>>> import sys, locale
>>> 'time' in sys.modules
False
>>> locale.setlocale(locale.LC_CTYPE, '')
'French_France.1252'
>>> import time
>>> time.tzname
('Est', 'Est (heure d’été)')
>>> time.strftime('%Z')
'Est (heure d’été)'

Note that Unix Python 3 sets LC_CTYPE at startup, so doing the same on Windows 
would actually improve cross-platform consistency.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26226] Various test suite failures on Windows

[Martin Panter]

Martin Panter added the comment:

test_distutils: Is the failure the same as before, or changed? Are you testing 
with new code that includes the Issue 27048 fix? It looks like test_distutils 
doesn’t support running directly via “unittest”. Perhaps try “python -m test -W 
test_distutils” or similar if you were previously only trying unittest.

Some of your new failures look like symptoms of running the test suite on 
out-of-date compiled modules. For test_ctypes, see Issue 27343; test_getargs2: 
Issue 26282; test_capi: Issue 26168.

strptime: Maybe the test suite is trying to report a string that contains both 
U+0092 and “é”, but cannot encode U+0092, and uses backslashreplace error 
handler.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26226] Various test suite failures on Windows

[Emanuel Barry]

Emanuel Barry added the comment:

About the strptime failure - it seems time.strftime returns "Est (heure d?été)" 
while _strptime.LocaleTime().timezone has "est (heure d\x92été)". I don't know 
why it's like that, but neither are correct - the character that goes there is 
a single quote ('). I'm also puzzled by the fact that 'é' appears properly 
despite not being ASCII.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26226] Various test suite failures on Windows

[Emanuel Barry]

Emanuel Barry added the comment:

Some of the tests that used to fail are now passing, however some are still 
failing (and new ones are also failing now).

Tests that are still failing:

test_code_module (TestInteractiveConsole; test_ps1 and test_ps2 failed)
test_codecencodings_iso2022 (blame git for changing line endings; will open a 
new issue about this)

New failures:

test_capi (_testcapi doesn't have test_buildvalue_N)
test_codecencodings_jp (passes when I run it by itself... ?)
test_ctypes (AssertionError: "phi" does not match "duplicate values for field 
'???'")
test_decimal (will open a new issue for that one, there are over 300 lines of 
failures/errors)
test_distutils (env changed, also no tests run when I try to run it by itself)
test_getargs2 (fails on import with 'from _testcapi import 
getargs_positional_only_and_keywords as getargs'; ImportError: cannot import 
name 'getargs_positional_only_and_keywords')

Tests previously hung up after that point. The tests continue now, so here are 
more failures:

test_logging (socket timeout)
test_os (many failed C++ assertions! [Visual C++ Runtime library] Will open a 
new issue about that)
test_random (line endings mismatch)
test_sax (again line endings mismatch)
test_sqlite (sqlite3.test.dbapi.CursorTests.CheckLastRowIDOnReplace fails with 
self.cu.lastrowid =! 1 [it's None])
test_strptime (that's a DST issue: AssertionError: False is not true : timezone 
est (heure d?été) not found in (frozenset({'gmt', 'est', 'utc'}), 
frozenset({'est (heure d\x92été)'})))
test_uuid (sounds like it's related to my hostname having non-ASCII characters)
test_warnings ('__main__' != '' [I'm running tests via the interactive prompt, 
this might be why])
test_xml_etree_c ('Windows fatal error: stack overflow' in the test that makes 
sure recursive repr doesn't overflow [but it doesn't fail when I run it 
individually, apparently])

I'll spawn new issues for the more serious/specific ones, but unless you'd 
rather have one issue per failure, I'd like to keep this one open (for now at 
least).

@R. David: 4 skips, but no failures in the email tests, so at least there's 
that!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26226] Various test suite failures on Windows

[R. David Murray]

R. David Murray added the comment:

It looks like this issue can be closed, but a new one should be opened for 
dealing with the newline issue under git on windows.  However, I'm surprised 
there weren't failures in the email tests.  Although I've been moving the email 
tests away from depending on data on the disk being in DOS format, I think 
there are some compat32 tests that still do so.

--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26226] Various test suite failures on Windows

[STINNER Victor]

STINNER Victor added the comment:

It looks like all issues are now fixed in the default branch, no? Can we close 
this issue?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26226] Various test suite failures on Windows

[Martin Panter]

Martin Panter added the comment:

Regarding the files getting messed up in Lib/test/cjkencodings, IMO the 
solution is to not configure Git to mess with newlines in files. ;) But perhaps 
a reasonable workaround might be to rename the relevant files to something like 
*.txt.bin. Perhaps that will bypass whatever logic Git uses.

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26226] Various test suite failures on Windows

[Ezio Melotti]

Ezio Melotti added the comment:

> I'm surprised by the test_codecencodings_iso2022 failures. Example:

==
FAIL: test_incrementaldecoder (test.test_codecencodings_iso2022.Test_ISO2022_KR)
--
Traceback (most recent call last):
  File "E:\GitHub\cpython\lib\test\multibytecodec_support.py", line 208, in 
test_incrementaldecoder
self.assertEqual(ostream.getvalue(), self.tstring[1])
AssertionError: b'\xe[334 
chars]\x80\n\xed\x9a\xa8\xec\x9c\xa8\xec\xa0\x81\xec[1668 chars]4.\n' != 
b'\xe[334 chars]\x80\r\n\xed\x9a\xa8\xec\x9c\xa8\xec\xa0\x81\x[1682 chars]\r\n'

I run into this problem during a sprint, with a machine that was also running 
Windows and cloned from the git repo.  Those bytes are read from txt files in 
the test/cjkencodings dir, and they are marked as binary files in .hgeol.  
Since git ignores .hgeol and treats them as text files, it also changes the 
newline to \r\n causing the failure.
The tests pass without problems after cloning with Mercurial.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26226] Various test suite failures on Windows

[Ezio Melotti]

Changes by Ezio Melotti :


--
nosy: +ezio.melotti
type:  -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26226] Various test suite failures on Windows

[STINNER Victor]

STINNER Victor added the comment:

For the test_httpservers failure, can you please try the following commands on 
your PC?

>>> socket.gethostname()
'selma'
>>> socket.gethostbyaddr(socket.gethostname())
('selma', [], ['2a01:e34:ec8d:4c70:3ea9:f4ff:fe65:c0c', 
'fe80::3ea9:f4ff:fe65:c0c'])

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26226] Various test suite failures on Windows

[STINNER Victor]

STINNER Victor added the comment:

I'm surprised by the test_codecencodings_iso2022 failures. Example:

==
FAIL: test_incrementaldecoder (test.test_codecencodings_iso2022.Test_ISO2022_KR)
--
Traceback (most recent call last):
  File "E:\GitHub\cpython\lib\test\multibytecodec_support.py", line 208, in 
test_incrementaldecoder
self.assertEqual(ostream.getvalue(), self.tstring[1])
AssertionError: b'\xe[334 
chars]\x80\n\xed\x9a\xa8\xec\x9c\xa8\xec\xa0\x81\xec[1668 chars]4.\n' != 
b'\xe[334 chars]\x80\r\n\xed\x9a\xa8\xec\x9c\xa8\xec\xa0\x81\x[1682 chars]\r\n'



It looks like Python uses the wrong encoding to decode stdout of child 
processes:

==
ERROR: test_build_ext (distutils.tests.test_build_ext.BuildExtTestCase)
--
Traceback (most recent call last):
  File "E:\GitHub\cpython\lib\distutils\tests\test_build_ext.py", line 61, in 
test_build_ext
cmd.run()
  File "E:\GitHub\cpython\lib\distutils\command\build_ext.py", line 338, in run
self.build_extensions()
  File "E:\GitHub\cpython\lib\distutils\command\build_ext.py", line 447, in 
build_extensions
self._build_extensions_serial()
  File "E:\GitHub\cpython\lib\distutils\command\build_ext.py", line 472, in 
_build_extensions_serial
self.build_extension(ext)
  File "E:\GitHub\cpython\lib\distutils\command\build_ext.py", line 532, in 
build_extension
depends=ext.depends)
  File "E:\GitHub\cpython\lib\distutils\_msvccompiler.py", line 306, in compile
self.initialize()
  File "E:\GitHub\cpython\lib\distutils\_msvccompiler.py", line 199, in 
initialize
vc_env = _get_vc_env(plat_spec)
  File "E:\GitHub\cpython\lib\distutils\_msvccompiler.py", line 92, in 
_get_vc_env
universal_newlines=True,
  File "E:\GitHub\cpython\lib\subprocess.py", line 636, in check_output
**kwargs).stdout
  File "E:\GitHub\cpython\lib\subprocess.py", line 705, in run
stdout, stderr = process.communicate(input, timeout=timeout)
  File "E:\GitHub\cpython\lib\subprocess.py", line 1062, in communicate
stdout = self.stdout.read()
  File "E:\GitHub\cpython\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 49: 
character maps to 

It would help to add a print() to lib\distutils\_msvccompiler.py:92 to get the 
executed command, then run manually the command, and see non-ASCII characters 
in the output.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26226] Various test suite failures on Windows

[Emanuel Barry]

Changes by Emanuel Barry :


--
stage:  -> needs patch
versions: +Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26226] Various test suite failures on Windows

[Emanuel Barry]

New submission from Emanuel Barry:

Compiled latest master and ran test suite (Py_DEBUG build). A few failures here 
and there, and some tests skipped. I haven't yet looked into getting the proper 
libraries to make some of the skipped tests execute, I'm trying to make the 
whole test suite pass first.

Attached file includes verbose output for all failed tests. The end of the file 
and the traceback at the end of the message here happen, then Python hangs and 
never resumes (tested for ~20 minutes).

Python 3.6.0a0 (default, Jan 27 2016, 10:49:09) [MSC v.1900 32 bit (Intel)] on w
in32
Type "help", "copyright", "credits" or "license" for more information.
>>> import test.regrtest
>>> test.regrtest.main_in_temp_cwd() # same as 'py -m test'
== CPython 3.6.0a0 (default, Jan 27 2016, 10:49:09) [MSC v.1900 32 bit (Intel)]
==   Windows-7-6.1.7601-SP1 little-endian
==   hash algorithm: siphash24 32bit
==   E:\GitHub\cpython\build\test_python_13304
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0,
dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, 
verbose=0,
bytes_warning=0, quiet=0, hash_randomization=1, isolated=0)
[  1/400] test_grammar
[  2/400] test_opcodes
[  3/400] test_dict
[  4/400] test_builtin
[  5/400] test_exceptions
[  6/400] test_types
[  7/400] test_unittest
[  8/400] test_doctest
[  9/400] test_doctest2
[ 10/400] test_support
[ 11/400] test___all__
[ 12/400] test___future__
[ 13/400] test__locale
[ 14/400] test__opcode
[ 15/400] test__osx_support
[ 16/400] test_abc
[ 17/400] test_abstract_numbers
[ 18/400] test_aifc
[ 19/400] test_argparse
[ 20/400] test_array
[ 21/400] test_asdl_parser
[ 22/400] test_ast
[ 23/400] test_asynchat
[ 24/400] test_asyncio
E:\GitHub\cpython\lib\asyncio\sslproto.py:327: ResourceWarning: unclosed 
transport

  warnings.warn("unclosed transport %r" % self, ResourceWarning)
test test_asyncio failed -- multiple errors occurred; run in verbose mode for 
details
[ 25/400/1] test_asyncore
[ 26/400/1] test_atexit
[ 27/400/1] test_audioop
[ 28/400/1] test_augassign
[ 29/400/1] test_base64
[ 30/400/1] test_bigaddrspace
[ 31/400/1] test_bigmem
[ 32/400/1] test_binascii
[ 33/400/1] test_binhex
[ 34/400/1] test_binop
[ 35/400/1] test_bisect
[ 36/400/1] test_bool
[ 37/400/1] test_buffer
[ 38/400/1] test_bufio
[ 39/400/1] test_bytes
[ 40/400/1] test_bz2
[ 41/400/1] test_calendar
[ 42/400/1] test_call
[ 43/400/1] test_capi
[ 44/400/1] test_cgi
[ 45/400/1] test_cgitb
[ 46/400/1] test_charmapcodec
[ 47/400/1] test_class
[ 48/400/1] test_cmath
[ 49/400/1] test_cmd
[ 50/400/1] test_cmd_line
[ 51/400/1] test_cmd_line_script
[ 52/400/1] test_code
[ 53/400/1] test_code_module
test test_code_module failed -- multiple errors occurred; run in verbose mode 
for details
[ 54/400/2] test_codeccallbacks
[ 55/400/2] test_codecencodings_cn
[ 56/400/2] test_codecencodings_hk
[ 57/400/2] test_codecencodings_iso2022
test test_codecencodings_iso2022 failed -- multiple errors occurred; run in 
verbose mode for details
[ 58/400/3] test_codecencodings_jp
[ 59/400/3] test_codecencodings_kr
[ 60/400/3] test_codecencodings_tw
[ 61/400/3] test_codecmaps_cn
[ 62/400/3] test_codecmaps_hk
[ 63/400/3] test_codecmaps_jp
[ 64/400/3] test_codecmaps_kr
[ 65/400/3] test_codecmaps_tw
[ 66/400/3] test_codecs
[ 67/400/3] test_codeop
[ 68/400/3] test_collections
[ 69/400/3] test_colorsys
[ 70/400/3] test_compare
[ 71/400/3] test_compile
[ 72/400/3] test_compileall
[ 73/400/3] test_complex
[ 74/400/3] test_concurrent_futures
[ 75/400/3] test_configparser
[ 76/400/3] test_contains
[ 77/400/3] test_contextlib
[ 78/400/3] test_copy
[ 79/400/3] test_copyreg
[ 80/400/3] test_coroutines
[ 81/400/3] test_cprofile
[ 82/400/3] test_crashers
[ 83/400/3] test_crypt
test_crypt skipped -- No module named '_crypt'
[ 84/400/3] test_csv
[ 85/400/3] test_ctypes
[ 86/400/3] test_curses
test_curses skipped -- Use of the 'curses' resource not enabled
[ 87/400/3] test_datetime
[ 88/400/3] test_dbm
[ 89/400/3] test_dbm_dumb
[ 90/400/3] test_dbm_gnu
test_dbm_gnu skipped -- No module named '_gdbm'
[ 91/400/3] test_dbm_ndbm
test_dbm_ndbm skipped -- No module named '_dbm'
[ 92/400/3] test_decimal
[ 93/400/3] test_decorators
[ 94/400/3] test_defaultdict
[ 95/400/3] test_deque
[ 96/400/3] test_descr
[ 97/400/3] test_descrtut
[ 98/400/3] test_devpoll
test_devpoll skipped -- test works only on Solaris OS family
[ 99/400/3] test_dictcomps
[100/400/3] test_dictviews
[101/400/3] test_difflib
[102/400/3] test_dis
[103/400/3] test_distutils

E:\GitHub\cpython\build\test_python_13304>exit 1

E:\GitHub\cpython\build\test_python_13304>exit 0
Warning -- files was modified by test_distutils
test test_distutils failed -- multiple errors occurred; run in verbose mode for 
details
[104/400/4] test_docxmlrpc
[105/400/4] test_dummy_thread
[106/400/4] test_dummy_threading
[107/400/4] test_dynamic
[108/400/4] test_dynamicclassattribute
[109/400/4] test_eintr
[110/400/4] test_email
[111/400/4] test_ensurepip
[112/400/4] 

[issue26226] Various test suite failures on Windows

[STINNER Victor]

STINNER Victor added the comment:

Python 2.7.11:

>>> import socket
>>> socket.gethostname()
'\xc9manuel-PC'

This one works on Python 3 because the Python function is implemented with a 
call to the Windows native API.

>>> socket.gethostbyaddr(socket.gethostname())
('\xc9manuel-PC.home', [], ['fe80::c9b7:5117:eea4:a104'])

This one fails on Python 3 because it uses the gethostbyaddr() C function and 
then decodes the hostname from UTF-8, whereas the hostname looks more to be 
encoded to ISO 8859-1 or something like that. IMHO it should be decoded from 
the ANSI code page.

I opened the issue #26227 to track this bug.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26226] Various test suite failures on Windows

[Emanuel Barry]

Emanuel Barry added the comment:

Well, it has a non-ASCII character in it, so I wouldn't be surprised if this 
was the issue :)

Latest master (3.6):

>>> import socket
>>> socket.gethostname()
'Émanuel-PC'
>>> socket.gethostbyaddr(socket.gethostname())
Traceback (most recent call last):
  File "", line 1, in 
socket.gaierror: [Errno 11004] getaddrinfo failed

Python 2.7.11:

>>> import socket
>>> socket.gethostname()
'\xc9manuel-PC'
>>> socket.gethostbyaddr(socket.gethostname())
('\xc9manuel-PC.home', [], ['fe80::c9b7:5117:eea4:a104'])

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com