[issue9988] test_warnings fails with PYTHONFSENCODING=latin-1 on UNIX/BSD

2010-10-13 Thread STINNER Victor 

STINNER Victor victor.stin...@haypocalc.com added the comment:

Fixed by r85430 (remove PYTHONFSENCODING), see #9992.

--
resolution:  - fixed
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9988
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9988] test_warnings fails with PYTHONFSENCODING=latin-1 on UNIX/BSD

2010-09-29 Thread STINNER Victor 

New submission from STINNER Victor victor.stin...@haypocalc.com:

$ PYTHONFSENCODING=latin-1 ./python Lib/test/test_warnings.py 
...
==
FAIL: test_nonascii (__main__.CEnvironmentVariableTests)
--
Traceback (most recent call last):
  File Lib/test/test_warnings.py, line 731, in test_nonascii
['ignore:DeprecaciónWarning'].encode('utf-8'))
AssertionError: b['ignore:Deprecaci\\udcf3nWarning'] != 
b['ignore:Deprecaci\xc3\xb3nWarning']

==
FAIL: test_nonascii (__main__.PyEnvironmentVariableTests)
--
Traceback (most recent call last):
  File Lib/test/test_warnings.py, line 731, in test_nonascii
['ignore:DeprecaciónWarning'].encode('utf-8'))
AssertionError: b['ignore:Deprecaci\\udcf3nWarning'] != 
b['ignore:Deprecaci\xc3\xb3nWarning']

--

The problem is that subprocess encodes PYTHONWARNINGS environment variable 
value with the filesystem encoding, whereas Py_main() decodes the variable 
value with the locale encoding.

History of how the variable is read in py3k:
 - #7301: r79880 creates this variable, use mbstowcs() and PySys_AddWarnOption()
 - #7301: r80066 uses setlocale(LC_ALL, ), and replaces mbstowcs() by 
_Py_char2wchar() (to support surrogates)
 - #8589: r81358 creates PySys_AddWarnOptionUnicode() and replaces 
_Py_char2wchar() by PyUnicode_DecodeFSDefault()
 - #8589: r84694 replaces PyUnicode_DecodeFSDefault() by _Py_char2wchar() + 
PyUnicode_FromWideChar() because the PyCodec machinery is not ready yet
 - #8589: r84731 uses PyUnicode_FromString() (utf-8) on Mac OS X

--
components: Tests, Unicode
messages: 117631
nosy: haypo
priority: normal
severity: normal
status: open
title: test_warnings fails with PYTHONFSENCODING=latin-1 on UNIX/BSD
versions: Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9988
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9988] test_warnings fails with PYTHONFSENCODING=latin-1 on UNIX/BSD

2010-09-29 Thread Antoine Pitrou 

Changes by Antoine Pitrou pit...@free.fr:


--
nosy: +brett.cannon

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9988
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9988] test_warnings fails with PYTHONFSENCODING=latin-1 on UNIX/BSD

2010-09-29 Thread Brett Cannon 

Brett Cannon br...@python.org added the comment:

OK, so who's messing up: subprocess or Py_main()?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9988
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9988] test_warnings fails with PYTHONFSENCODING=latin-1 on UNIX/BSD

2010-09-29 Thread Philip Jenvey 

Changes by Philip Jenvey pjen...@underboss.org:


--
nosy: +pjenvey

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9988
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9988] test_warnings fails with PYTHONFSENCODING=latin-1 on UNIX/BSD

2010-09-29 Thread STINNER Victor 

STINNER Victor victor.stin...@haypocalc.com added the comment:

 OK, so who's messing up: subprocess or Py_main()?

Well, this is the real question :-)

locale encoding is used to decode command line arguments (sys.argv), filesystem 
encoding is used to decode environment variables and to encode subprocess 
arguments and environment variables.

It means that we have something funny if a Python process creates a Python 
subprocess: child process arguments are encoded using the filesystem encoding, 
whereas the arguments are decoded using the locale encoding. If both encodings 
are different (eg. if PYTHONFSENCODING is used by at least the parent process), 
we have the bug similar to #4388. See also issue #8775.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9988
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9988] test_warnings fails with PYTHONFSENCODING=latin-1 on UNIX/BSD

2010-09-29 Thread Philip Jenvey 

Philip Jenvey pjen...@underboss.org added the comment:

It sounds like you had PYTHONWARNINGS using the fs encoding before r84694, but 
reverted it due to bootstrapping issues.

Indeed, the fs encoding isn't initialized until later in Py_InitializeEx. Maybe 
the PYTHONWARNINGS code should be moved there instead?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9988
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9988] test_warnings fails with PYTHONFSENCODING=latin-1 on UNIX/BSD

2010-09-29 Thread STINNER Victor 

STINNER Victor victor.stin...@haypocalc.com added the comment:

 Indeed, the fs encoding isn't initialized until later in
 Py_InitializeEx. Maybe the PYTHONWARNINGS code should be moved 
 there instead?

sys.warnopts should be filled early because it is used to initialize the 
_warnings module, and the _warnings module have to be initialized before 
loading another non-builtin module because importing a module may emit a 
warning.

 OK, so who's messing up: subprocess or Py_main()?

I opened issue #9992 to decide which encoding should be used to encode and 
decode command line arguments: locale or filesystem encoding.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9988
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9988] test_warnings fails with PYTHONFSENCODING=latin-1 on UNIX/BSD

2010-09-29 Thread STINNER Victor 

STINNER Victor victor.stin...@haypocalc.com added the comment:

 Maybe the PYTHONWARNINGS code should be moved there instead?

sys.warnoptions is read by the warnings module (not the _warnings module) when 
this module is loaded. The warnings module is loaded by Py_InitializeEx() if 
sys.warnoptions list is not empty.

It might be possible to read PYTHONWARNINGS env var after initfsencoding() but 
before loading the warnings module. But we have to ensure that 
Py_InitializeEx() can still be called multiple times.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9988
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com