Re: env parameter to popen won't accept Unicode on Windows - minor Unicode bug
John Nagle wrote: Benjamin wrote: On Jan 14, 6:26 pm, Bjoern Schliessmann usenet- [EMAIL PROTECTED] wrote: John Nagle wrote: It turns out that the strings in the env parameter have to be ASCII, not Unicode, even though Windows fully supports Unicode in CreateProcess. That's of course nonsense, they don't need to be ascii, they need to be byte-strings in whatever encoding you like. Are you sure it supports Unicode, not UTF8 or UTF16? Probably using something like uthestring.encode(utf16) will help. Otherwise: bugs.python.org John's understanding of the differences between unicode and it's encodings is a bit blurry, to say the least. Whatever translation is necessary should be done in popen, which has cases for Windows and POSIX. popen is supposed to be cross-platform to the extent possible. I think it's just something that didn't get fixed when Unicode support went in. Sure thing, python will just magically convert unicode to the encoding the program YOU invoke will expect. Right after we introduced the solve_my_problem() built-in-function. Any other wishes? If I write this simple program -- test.py --- import os import sys ENCODDINGS = ['utf-8', 'latin1'] os.env[MY_VARIABLE].encode(ENCODINGS[int(sys.argv[1])]) -- test.py --- how's python supposed to know that suprocess.call(python test.py 0, env=dict(MY_VARIABLE=u'foo')) needs to be UTF-8? Diez -- http://mail.python.org/mailman/listinfo/python-list
RE: env parameter to popen won't accept Unicode on Windows -minor Unicode bug
Diez B. Roggisch wrote: Sure thing, python will just magically convert unicode to the encoding the program YOU invoke will expect. Right after we introduced the solve_my_problem() built-in-function. Any other wishes? There's no reason to be rude. Anyway, at least on Windows it makes perfect sense for people to expect Unicode to be handled automatically. popen() knows that it is running on Windows, and it knows what encoding Windows needs for its environment (it's either UCS2 or UTF-16 for most Windows APIs). At least when it receives a unicode string, it has enough information to apply the conversion automatically, and doing so saves the caller from having to figure out what exact encoding is to be used. - Brian -- http://mail.python.org/mailman/listinfo/python-list
RE: env parameter to popen won't accept Unicode on Windows -minor Unicode bug
Brian Smith wrote: Diez B. Roggisch wrote: Sure thing, python will just magically convert unicode to the encoding the program YOU invoke will expect. Right after we introduced the solve_my_problem() built-in-function. Any other wishes? There's no reason to be rude. If you'd know John, you'd know there is. Anyway, at least on Windows it makes perfect sense for people to expect Unicode to be handled automatically. popen() knows that it is running on Windows, and it knows what encoding Windows needs for its environment (it's either UCS2 or UTF-16 for most Windows APIs). At least when it receives a unicode string, it has enough information to apply the conversion automatically, and doing so saves the caller from having to figure out what exact encoding is to be used. For once, the distinction between windows and other platforms is debatable. I admit that subprocess contains already quite a few platform specific aspects, but it's purpose is to abstract these away as much as possible. However, I'm not sure that just because there are wide-char windows apis available automatically means that using UCS2/UTF-16 would succeed. A look into the python sources (PC/_subprocess.c) reveals that someone already thought about this, but it seems that just setting a CREATE_UNICODE_ENVIRONMENT in the CreateProcess-function should have been easy enough to do it if there weren't any troubles to expect. Additionally, passing unicode to env would also imply that os.environ should yield unicode as well. Not sure how much code _that_ breaks. Diez -- http://mail.python.org/mailman/listinfo/python-list
RE: env parameter to popen won't accept Unicode on Windows -minor Unicode bug
Brian Smith wrote: popen() knows that it is running on Windows, and it knows what encoding Windows needs for its environment (it's either UCS2 or UTF-16 for most Windows APIs). At least when it receives a unicode string, it has enough information to apply the conversion automatically, and doing so saves the caller from having to figure out what exact encoding is to be used. So you propose Python should employ a hidden automatism that automagically guesses the right encoding? Why not leave it explicitly/consistently and let the user decide? What will happen if a future Windows changes its encoding? Will we need another magic routine to tell it apart? Regards, Björn -- BOFH excuse #353: Second-system effect. -- http://mail.python.org/mailman/listinfo/python-list
Re: env parameter to popen won't accept Unicode on Windows - minor Unicode bug
Diez B. Roggisch wrote: John Nagle wrote: Benjamin wrote: On Jan 14, 6:26 pm, Bjoern Schliessmann usenet- [EMAIL PROTECTED] wrote: John Nagle wrote: It turns out that the strings in the env parameter have to be ASCII, not Unicode, even though Windows fully supports Unicode in CreateProcess. That's of course nonsense, they don't need to be ascii, they need to be byte-strings in whatever encoding you like. Are you sure it supports Unicode, not UTF8 or UTF16? Probably using something like uthestring.encode(utf16) will help. Otherwise: bugs.python.org John's understanding of the differences between unicode and it's encodings is a bit blurry, to say the least. Who's this guy? Whatever translation is necessary should be done in popen, which has cases for Windows and POSIX. popen is supposed to be cross-platform to the extent possible. I think it's just something that didn't get fixed when Unicode support went in. I've been looking at the source code. There's _PyPopenCreateProcess in posixmodule.c.That one doesn't support passing an environment at all; see the call to Windows CreateProcess. Is that the one that Popen uses? Where is win32process in the source? It ought to be in Modules, but it's not. John Nagle -- http://mail.python.org/mailman/listinfo/python-list
Re: env parameter to popen won't accept Unicode on Windows -minor Unicode bug
Diez B. Roggisch wrote: Brian Smith wrote: Diez B. Roggisch wrote: Sure thing, python will just magically convert unicode to the encoding the program YOU invoke will expect. Right after we introduced the solve_my_problem() built-in-function. Any other wishes? There's no reason to be rude. If you'd know John, you'd know there is. ? Anyway, at least on Windows it makes perfect sense for people to expect Unicode to be handled automatically. popen() knows that it is running on Windows, and it knows what encoding Windows needs for its environment (it's either UCS2 or UTF-16 for most Windows APIs). At least when it receives a unicode string, it has enough information to apply the conversion automatically, and doing so saves the caller from having to figure out what exact encoding is to be used. For once, the distinction between windows and other platforms is debatable. I admit that subprocess contains already quite a few platform specific aspects, but it's purpose is to abstract these away as much as possible. However, I'm not sure that just because there are wide-char windows apis available automatically means that using UCS2/UTF-16 would succeed. A look into the python sources (PC/_subprocess.c) reveals that someone already thought about this, but it seems that just setting a CREATE_UNICODE_ENVIRONMENT in the CreateProcess-function should have been easy enough to do it if there weren't any troubles to expect. The problem is that only the NT-derived Microsoft systems talk Unicode. The DOS/Win16/Win9x family did not. But they did have CreateProcess. So the current code will handle Win9x, but not Unicode. When do we drop support for Win9x? It probably has to happen in Python 3K, since that's Unicode-everywhere. John Nagle -- http://mail.python.org/mailman/listinfo/python-list
Re: env parameter to popen won't accept Unicode on Windows -minor Unicode bug
John Nagle wrote: The problem is that only the NT-derived Microsoft systems talk Unicode. The DOS/Win16/Win9x family did not. But they did have CreateProcess. So the current code will handle Win9x, but not Unicode. Please explain, I don't understand. If you try using Windows system functions in older Windows versions, umystring will fail, too. Those functions need byte strings, not Unicode string instances. The latter have to be encoded to byte strings to pass them. Regards, Björn -- BOFH excuse #70: nesting roaches shorted out the ether cable -- http://mail.python.org/mailman/listinfo/python-list
env parameter to popen won't accept Unicode on Windows - minor Unicode bug
I passed a dict for the env variable to Popen with Unicode strings for the dictionary values. Got: File D:\Python24\lib\subprocess.py, line 706, in _execute_child TypeError: environment can only contain strings It turns out that the strings in the env parameter have to be ASCII, not Unicode, even though Windows fully supports Unicode in CreateProcess. John Nagle -- http://mail.python.org/mailman/listinfo/python-list
Re: env parameter to popen won't accept Unicode on Windows - minor Unicode bug
John Nagle wrote: It turns out that the strings in the env parameter have to be ASCII, not Unicode, even though Windows fully supports Unicode in CreateProcess. Are you sure it supports Unicode, not UTF8 or UTF16? Probably using something like uthestring.encode(utf16) will help. Regards, Björn -- BOFH excuse #31: cellular telephone interference -- http://mail.python.org/mailman/listinfo/python-list
Re: env parameter to popen won't accept Unicode on Windows - minor Unicode bug
On Jan 14, 6:26 pm, John Nagle [EMAIL PROTECTED] wrote: I passed a dict for the env variable to Popen with Unicode strings for the dictionary values. Got: File D:\Python24\lib\subprocess.py, line 706, in _execute_child TypeError: environment can only contain strings It turns out that the strings in the env parameter have to be ASCII, not Unicode, even though Windows fully supports Unicode in CreateProcess. John Nagle -- http://mail.python.org/mailman/listinfo/python-list
Re: env parameter to popen won't accept Unicode on Windows - minor Unicode bug
On Jan 14, 6:26 pm, Bjoern Schliessmann usenet- [EMAIL PROTECTED] wrote: John Nagle wrote: It turns out that the strings in the env parameter have to be ASCII, not Unicode, even though Windows fully supports Unicode in CreateProcess. Are you sure it supports Unicode, not UTF8 or UTF16? Probably using something like uthestring.encode(utf16) will help. Otherwise: bugs.python.org Regards, Björn -- BOFH excuse #31: cellular telephone interference -- http://mail.python.org/mailman/listinfo/python-list
Re: env parameter to popen won't accept Unicode on Windows - minor Unicode bug
Benjamin wrote: On Jan 14, 6:26 pm, Bjoern Schliessmann usenet- [EMAIL PROTECTED] wrote: John Nagle wrote: It turns out that the strings in the env parameter have to be ASCII, not Unicode, even though Windows fully supports Unicode in CreateProcess. Are you sure it supports Unicode, not UTF8 or UTF16? Probably using something like uthestring.encode(utf16) will help. Otherwise: bugs.python.org Whatever translation is necessary should be done in popen, which has cases for Windows and POSIX. popen is supposed to be cross-platform to the extent possible. I think it's just something that didn't get fixed when Unicode support went in. John Nagle -- http://mail.python.org/mailman/listinfo/python-list
