Re: [Python-Dev] Proposal: new environment variable PYTHONSTDOUTENCODING
Hi Martin, On Tue, May 20, 2008 at 10:22:37AM +0200, Martin v. Löwis wrote: In particular, setting this environment variable would also disable the detection of whether stdout is a terminal. In this case, it seems to me that existing programs that start python as a non-interactive subprocess, via a mecanism like os.popen2() or the equivalent in other languages, will receive bogus data (the python banner) and/or hang in unexpected ways (the subprocess waiting for more input after the prompt). A bientot, Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Proposal: new environment variable PYTHONSTDOUTENCODING
I'd like to propose a new environment variable PYTHONSTDOUTENCODING. This is meant to solve various problems that people had with Python not detecting their terminal encoding correctly; it would override any detection that Python would use for determining the encoding of stdout (and stdin - but that's less relevant in 2.x). In particular, setting this environment variable would also disable the detection of whether stdout is a terminal. This is desirable for cases as the pydev eclipse plugin, where Python currently fails to detect that the output is a terminal (and technically, what Eclipse provides is not a terminal, but just a pipe, as you can't do pseudoterms in Java). This would have the additional effect that the encoding also gets in effect when redirecting stdout to a file. Whether or not this is a good thing might be debatable; giving the user the control over it (to set or clear that variable) is a good thing, IMO. Naming contest: it probably would be the longest of the PYTHON* variables. I would not want to call it PYTHONENCODING, or PYTHONSTDENCODING, though, because people might infer that it affects sys.getdefaultencoding(), which it shouldn't. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal: new environment variable PYTHONSTDOUTENCODING
On 2008-05-20 10:22, Martin v. Löwis wrote: I'd like to propose a new environment variable PYTHONSTDOUTENCODING. This is meant to solve various problems that people had with Python not detecting their terminal encoding correctly; it would override any detection that Python would use for determining the encoding of stdout (and stdin - but that's less relevant in 2.x). How is this relevant for 2.x ? In 2.x, stdin and stdout are just files without any io wrappers around them. Writing Unicode to stdout will still use the default encoding ASCII to convert it to an 8-bit string. All other 8-bit strings will be passed to stdout as-is. For 3.x, I'd like to see a PYTHONSTDINENCODING, because the current way of relying on the terminal encoding does work well... it then falls back to ASCII, which prevents entering e.g. German Umlauts. In particular, setting this environment variable would also disable the detection of whether stdout is a terminal. This is desirable for cases as the pydev eclipse plugin, where Python currently fails to detect that the output is a terminal (and technically, what Eclipse provides is not a terminal, but just a pipe, as you can't do pseudoterms in Java). This would have the additional effect that the encoding also gets in effect when redirecting stdout to a file. Whether or not this is a good thing might be debatable; giving the user the control over it (to set or clear that variable) is a good thing, IMO. Naming contest: it probably would be the longest of the PYTHON* variables. I would not want to call it PYTHONENCODING, or PYTHONSTDENCODING, though, because people might infer that it affects sys.getdefaultencoding(), which it shouldn't. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 20 2008) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal: new environment variable PYTHONSTDOUTENCODING
On Tue, May 20, 2008 at 10:41 AM, M.-A. Lemburg [EMAIL PROTECTED] wrote: On 2008-05-20 10:22, Martin v. Löwis wrote: I'd like to propose a new environment variable PYTHONSTDOUTENCODING. This is meant to solve various problems that people had with Python not detecting their terminal encoding correctly; it would override any detection that Python would use for determining the encoding of stdout (and stdin - but that's less relevant in 2.x). How is this relevant for 2.x ? In 2.x, stdin and stdout are just files without any io wrappers around them. Writing Unicode to stdout will still use the default encoding ASCII to convert it to an 8-bit string. All other 8-bit strings will be passed to stdout as-is. You're forgetting about print; in Python 2.x, when stdout is connected to a terminal, the locale settings (typically the LANG, LC_ALL and LC_CTYPE environment variables) are taken into account when 'print' writes to sys.stdout. -- Thomas Wouters [EMAIL PROTECTED] Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal: new environment variable PYTHONSTDOUTENCODING
On Tue, May 20, 2008 at 12:16 PM, Thomas Wouters [EMAIL PROTECTED] wrote: You're forgetting about print; in Python 2.x, when stdout is connected to a terminal, the locale settings (typically the LANG, LC_ALL and LC_CTYPE environment variables) are taken into account when 'print' writes to sys.stdout. Isn't it then enough to make sure your locale setting are correct? (Never had any problems myself, if works great in Ubuntu). -- Lennart Regebro: Zope and Plone consulting. http://www.colliberty.com/ +33 661 58 14 64 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal: new environment variable PYTHONSTDOUTENCODING
On Tue, May 20, 2008 at 10:22:37AM +0200, Martin v. L?wis wrote: I'd like to propose a new environment variable PYTHONSTDOUTENCODING. This is meant to solve various problems that people had with Python not detecting their terminal encoding correctly; it would override any detection that Python would use for determining the encoding of stdout (and stdin - but that's less relevant in 2.x). Is it to override locale settings in case the user wants a different encoding? for such cases as redirected stdout, or windows console (which has an OEM encoding that differs from the locale encoding)? Naming contest: it probably would be the longest of the PYTHON* variables. I would not want to call it PYTHONENCODING, or PYTHONSTDENCODING, though, because people might infer that it affects sys.getdefaultencoding(), which it shouldn't. PYTHONIOENCODING? Oleg. -- Oleg Broytmannhttp://phd.pp.ru/[EMAIL PROTECTED] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal: new environment variable PYTHONSTDOUTENCODING
On 2008-05-20 12:16, Thomas Wouters wrote: On Tue, May 20, 2008 at 10:41 AM, M.-A. Lemburg [EMAIL PROTECTED] wrote: On 2008-05-20 10:22, Martin v. Löwis wrote: I'd like to propose a new environment variable PYTHONSTDOUTENCODING. This is meant to solve various problems that people had with Python not detecting their terminal encoding correctly; it would override any detection that Python would use for determining the encoding of stdout (and stdin - but that's less relevant in 2.x). How is this relevant for 2.x ? In 2.x, stdin and stdout are just files without any io wrappers around them. Writing Unicode to stdout will still use the default encoding ASCII to convert it to an 8-bit string. All other 8-bit strings will be passed to stdout as-is. You're forgetting about print; in Python 2.x, when stdout is connected to a terminal, the locale settings (typically the LANG, LC_ALL and LC_CTYPE environment variables) are taken into account when 'print' writes to sys.stdout. Thanks for reminding me. I had forgotten about that special case. So sys.stdout.write(unicode) will always use the default encoding, whereas print unicode uses the sys.stdout.encoding, correct ? Hmm, wouldn't it be better to always use .encoding and also make it adjustable from Python (it is adjustable from C) ?! PYTHONSTDOUTENCODING could then provide the default to sys.stdout.encoding. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 20 2008) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal: new environment variable PYTHONSTDOUTENCODING
On Tue, May 20, 2008 at 12:48 PM, Oleg Broytmann [EMAIL PROTECTED] wrote: On Tue, May 20, 2008 at 10:22:37AM +0200, Martin v. L?wis wrote: I'd like to propose a new environment variable PYTHONSTDOUTENCODING. This is meant to solve various problems that people had with Python not detecting their terminal encoding correctly; it would override any detection that Python would use for determining the encoding of stdout (and stdin - but that's less relevant in 2.x). Is it to override locale settings in case the user wants a different encoding? for such cases as redirected stdout, or windows console (which has an OEM encoding that differs from the locale encoding)? Naming contest: it probably would be the longest of the PYTHON* variables. I would not want to call it PYTHONENCODING, or PYTHONSTDENCODING, though, because people might infer that it affects sys.getdefaultencoding(), which it shouldn't. PYTHONIOENCODING? What about PYTHONLANG ? or something that tries to reflect which environment variables are used for this ? (LC_CTYPE - PYTHONCTYPE ? if the code uses just LC_CTYPE) http://www.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap08.html#tag_08_02 Just for my own knowledge: why it has to be one word ? can't it be PYTHON_LANG ? Tarek Oleg. -- Oleg Broytmannhttp://phd.pp.ru/[EMAIL PROTECTED] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal: new environment variable PYTHONSTDOUTENCODING
Writing Unicode to stdout will still use the default encoding ASCII to convert it to an 8-bit string. That's not true. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal: new environment variable PYTHONSTDOUTENCODING
You're forgetting about print; in Python 2.x, when stdout is connected to a terminal, the locale settings (typically the LANG, LC_ALL and LC_CTYPE environment variables) are taken into account when 'print' writes to sys.stdout. Isn't it then enough to make sure your locale setting are correct? (Never had any problems myself, if works great in Ubuntu). It's much more difficult on OS X, for example, which doesn't really support the concept of locales (at least prior to 10.5). There are other odd cases, like the Eclipse one I mentioned. Setting the locale doesn't help there. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal: new environment variable PYTHONSTDOUTENCODING
PYTHONSTDOUTENCODING could then provide the default to sys.stdout.encoding. Right - that's the proposal. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal: new environment variable PYTHONSTDOUTENCODING
On 2008-05-20 20:23, Martin v. Löwis wrote: Writing Unicode to stdout will still use the default encoding ASCII to convert it to an 8-bit string. That's not true. Are you sure ? setenv LC_ALL de_DE.utf8 python2.5 Python 2.5 (r25:51908, May 9 2007, 00:53:06) u = u'äöü' sys.stdout.write(u) Traceback (most recent call last): File stdin, line 1, in module UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128) print u äöü Only print will set the Py_PRINT_RAW flag to trigger the conversion from Unicode to 8-bit strings using .encoding in PyFile_WriteObject(). If not set, the default encoding is used. I'm not exactly sure why, since using .encoding would be useful in all cases. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 20 2008) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal: new environment variable PYTHONSTDOUTENCODING
I'm not exactly sure why, since using .encoding would be useful in all cases. Right, I think it should use the file's encoding also for .write. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal: new environment variable PYTHONSTDOUTENCODING
Is it to override locale settings in case the user wants a different encoding? for such cases as redirected stdout, or windows console (which has an OEM encoding that differs from the locale encoding)? On Windows, the setlocale mechanism isn't used at all, since it doesn't support nl_langinfo (let alone CODESET). And yes, that is to override whatever determination Python would make on its own. Naming contest: it probably would be the longest of the PYTHON* variables. I would not want to call it PYTHONENCODING, or PYTHONSTDENCODING, though, because people might infer that it affects sys.getdefaultencoding(), which it shouldn't. PYTHONIOENCODING? Unprecise in a different way (as it is meant to apply only to stdout, not to all IO), but shorter. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal: new environment variable PYTHONSTDOUTENCODING
What about PYTHONLANG ? or something that tries to reflect which environment variables are used for this ? (LC_CTYPE - PYTHONCTYPE ? if the code uses just LC_CTYPE) It's not meant to name a locale, but an encoding. In fact, that the encoding is tied to the locale is IMO a misconception in the POSIX locale machinery. http://www.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap08.html#tag_08_02 Just for my own knowledge: why it has to be one word ? can't it be PYTHON_LANG ? No technical reason - just because PYTHONPATH, PYTHONHOME, PYTHONUNBUFFERED, PYTHONVERBOSE, PYTHONSTARTUP, and PYTHONCASEOK don't have underscores, either. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal: new environment variable PYTHONSTDOUTENCODING
On Tue, May 20, 2008 at 10:22:03PM +0200, Martin v. L?wis wrote: PYTHONIOENCODING? Unprecise in a different way (as it is meant to apply only to stdout, not to all IO), but shorter. I don't think you can make it both precise and short. If you want to be precise and have both PYTHON and STDOUT - shorten ENCODING to ENC. If you agree to sacrifice PYTHON - make it PYSTDOUTENCODING. Oleg. -- Oleg Broytmannhttp://phd.pp.ru/[EMAIL PROTECTED] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
