[issue6135] subprocess seems to use local encoding and give no choice

Gregory P. Smith added the comment: New changeset 7f95c8c319c1ee593b130d0eb1d4947d9d7e008a by Gregory P. Smith (Miss Islington (bot)) in branch '3.6': bpo-6135: Fix subprocess.check_output doc to mention changes in 3.6 (GH-5564) (GH-5573)

[issue6135] subprocess seems to use local encoding and give no choice

Gregory P. Smith added the comment: New changeset 4e7a964aaf4374fa2f6b45cf5161fa6cd53aec19 by Gregory P. Smith (Miss Islington (bot)) in branch '3.7': bpo-6135: Fix subprocess.check_output doc to mention changes in 3.6 (GH-5564) (GH-5572)

[issue6135] subprocess seems to use local encoding and give no choice

Change by miss-islington : -- pull_requests: +5391 ___ Python tracker ___

[issue6135] subprocess seems to use local encoding and give no choice

Change by miss-islington : -- pull_requests: +5390 ___ Python tracker ___

[issue6135] subprocess seems to use local encoding and give no choice

Gregory P. Smith added the comment: New changeset fc1ce810f1da593648b4d19e7d582a235ec1dd37 by Gregory P. Smith (Brice Gros) in branch 'master': bpo-6135: Fix subprocess.check_output doc to mention changes in 3.6 (GH-5564)

[issue6135] subprocess seems to use local encoding and give no choice

Change by aicirbs : -- pull_requests: +5389 ___ Python tracker ___ ___ Python-bugs-list

[issue6135] subprocess seems to use local encoding and give no choice

Andrew Clegg added the comment: RFE submitted as issue31756 , thanks -- ___ Python tracker ___

[issue6135] subprocess seems to use local encoding and give no choice

Nick Coghlan added the comment: This discussion should probably be submitted as a new RFE (requesting "text" as a more obvious and beginner friendly alias for universal_newlines), but I'll also add a note regarding precedent for a simple binary/text toggle: the mode

[issue6135] subprocess seems to use local encoding and give no choice

Andrew Clegg added the comment: I meant the former; I'll look a bit more at the documentation and submit an issue/patch. As regards the 'text' flag - universal_newlines is actually exactly that already. I've just checked the code of subprocess.py and the

[issue6135] subprocess seems to use local encoding and give no choice

Steve Dower added the comment: > The commit for this bug (720f0cf580e2) introduces encoding and errors > arguments but doesn't actually document what the values of these should be Do you mean it doesn't document that they take the normal encoding/errors arguments that

[issue6135] subprocess seems to use local encoding and give no choice

Andrew Clegg added the comment: The commit for this bug (720f0cf580e2) introduces encoding and errors arguments but doesn't actually document what the values of these should be. In the case of the encoding it could be reasonably guessed, but the only way to

[issue6135] subprocess seems to use local encoding and give no choice

Steve Dower added the comment: > looks like only way to specify console's codepage is with `encoding=os.device_encoding(2)` which will have to be used for 99% of cases That's true, but it only applies when the subprocess is using the same call (GetConsoleCP) to determine what encoding to use

[issue6135] subprocess seems to use local encoding and give no choice

Dāvis added the comment: And looks like only way to specify console's codepage is with `encoding=os.device_encoding(2)` which will have to be used for 99% of cases. I don't see other way... -- ___ Python tracker

[issue6135] subprocess seems to use local encoding and give no choice

Dāvis added the comment: Still can't specify encoding for getstatusoutput and getoutput. Also it uses wrong encoding by default, most of time it will be console's codepage instead of ANSI codepage which is used now. -- nosy: +davispuh ___ Python

[issue6135] subprocess seems to use local encoding and give no choice

Steve Dower added the comment: I also added more tests, specifically for ansi and oem encodings on Windows and the errors argument on all platforms. -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker

[issue6135] subprocess seems to use local encoding and give no choice

Roundup Robot added the comment: New changeset 720f0cf580e2 by Steve Dower in branch 'default': Issue #6135: Adds encoding and errors parameters to subprocess https://hg.python.org/cpython/rev/720f0cf580e2 -- nosy: +python-dev ___ Python tracker

[issue6135] subprocess seems to use local encoding and give no choice

Martin Panter added the comment: Maybe good to adjust the other mentions of universal_newlines, e.g. for check_output(). The Posix version of the multiple-pipe _communicate() method probably needs adjusting too. Test case: >>> check_output(("true",), encoding="ascii", input="") # Should be

[issue6135] subprocess seems to use local encoding and give no choice

Steve Dower added the comment: More doc updates - shouldn't be a difficult review this time, but I always like getting multiple opinions on doc changes. -- Added file: http://bugs.python.org/file44408/6135_5.patch ___ Python tracker

[issue6135] subprocess seems to use local encoding and give no choice

Steve Dower added the comment: > Steve: please use this simpler flag to avoid TextIOWrapper details in > subprocess.py The TextIOWrapper details are already specified in multiple places in the documentation. Should we remove all of those and write it more like: "if *encoding*, *errors* or

[issue6135] subprocess seems to use local encoding and give no choice

STINNER Victor added the comment: > Why do you need to call getpreferredencoding()? I proposed to do that, but I prefer your simple flag: > text_mode = encoding or errors or universal_newlines Steve: please use this simpler flag to avoid TextIOWrapper details in subprocess.py ;-) --

[issue6135] subprocess seems to use local encoding and give no choice

Eryk Sun added the comment: Why do you need to call getpreferredencoding()? Isn't that already the default if you call TextIOWrapper with encoding as None? For example: text_mode = encoding or errors or universal_newlines self.stdin = io.open(p2cwrite, 'wb', bufsize) if text_mode:

[issue6135] subprocess seems to use local encoding and give no choice

STINNER Victor added the comment: Steve Dower added the comment: > Sure, that's easy enough. Any other concerns once I've made that change? If errors enables Unicode and the doc is updated, the patch will LTGM :-) -- ___ Python tracker

[issue6135] subprocess seems to use local encoding and give no choice

Steve Dower added the comment: Sure, that's easy enough. Any other concerns once I've made that change? -- ___ Python tracker ___

[issue6135] subprocess seems to use local encoding and give no choice

STINNER Victor added the comment: 6135_4.patch: Hum, what if you only set errors? I suggest to use TextIOWrapper but use the locale encoding: if (universal_newlines or errors) and not encoding: encoding = getpreferredencoding(False) -- ___ Python

[issue6135] subprocess seems to use local encoding and give no choice

Steve Dower added the comment: Addressed more feedback. -- Added file: http://bugs.python.org/file44404/6135_4.patch ___ Python tracker ___

[issue6135] subprocess seems to use local encoding and give no choice

Steve Dower added the comment: > universal_newlines uses the locale encoding which is a good choice in most cases. Well, some cases, and only really by accident. I won't hide the parameter, but it is promoted as "frequently used" and I'll make sure to document encoding before

[issue6135] subprocess seems to use local encoding and give no choice

STINNER Victor added the comment: Steve Dower added the comment: > Given specifying an encoding will do the same thing as universal_newlines > would have, should I just "hide" references to universal_newlines in the doc? > (i.e. only mention it under a versionchanged banner, rather than >

[issue6135] subprocess seems to use local encoding and give no choice

Nick Coghlan added the comment: Ah, excellent point about "encoding='utf-8'" already being a less cryptic replacement for universal_newlines=True, so consider my questions withdrawn :) As far as universal_newlines goes, that needs to remain documented for the sake of folks reading code that

[issue6135] subprocess seems to use local encoding and give no choice

Steve Dower added the comment: Given specifying an encoding will do the same thing as universal_newlines would have, should I just "hide" references to universal_newlines in the doc? (i.e. only mention it under a versionchanged banner, rather than front-and-centre) --

[issue6135] subprocess seems to use local encoding and give no choice

STINNER Victor added the comment: Nick Coghlan added the comment: > a. How do folks feel about providing a new "text" parameter to replace the > cryptic "universal_newlines=True" that would explicitly be equivalent to > "universal newlines with sys.getdefaultencoding()"? If it's just

[issue6135] subprocess seems to use local encoding and give no choice

STINNER Victor added the comment: Martin Panter added the comment: > Also, should encoding=... or errors=... be an error if > universal_newlines=False (the default)? Right. But if encoding or errors is used, universal_newlines value should be set automatically to True. For example, I expect

[issue6135] subprocess seems to use local encoding and give no choice

STINNER Victor added the comment: Steve: > You may be right about leaving out the opener API. The only use of it right > now is for separate encodings, but I don't know how valuable that is. My proposal is: Popen(cmd, stdin={'encoding': 'oem'}, stdout={'encoding': 'ansi'}) The dict would just

[issue6135] subprocess seems to use local encoding and give no choice

Nick Coghlan added the comment: A couple of high level questions: a. How do folks feel about providing a new "text" parameter to replace the cryptic "universal_newlines=True" that would explicitly be equivalent to "universal newlines with sys.getdefaultencoding()"? b. Given (a), what if the

[issue6135] subprocess seems to use local encoding and give no choice

Martin Panter added the comment: Would be nice to see tests for getstatusoutput() and the errors parameter. Also you need more error handling e.g. if the encoding is unsupported, I think the internal pipe files won’t be cleaned up. Also, should encoding=... or errors=... be an error if

[issue6135] subprocess seems to use local encoding and give no choice

Steve Dower added the comment: You may be right about leaving out the opener API. The only use of it right now is for separate encodings, but I don't know how valuable that is. I'll pull it out tomorrow and just leave the encoding parameter. -- ___

[issue6135] subprocess seems to use local encoding and give no choice

Steve Dower added the comment: Added tests. -- Added file: http://bugs.python.org/file44389/6135_2.patch ___ Python tracker ___

[issue6135] subprocess seems to use local encoding and give no choice

STINNER Victor added the comment: I'm not sure that about "advanced API" to specify an encoding per stream, or even change other parameters like buffering or newlines. I suggest to start with the least controversal part: add encoding and errors and only accept a string. More patches can come

[issue6135] subprocess seems to use local encoding and give no choice

Steve Dower added the comment: Initial patch attached - tests to follow. -- Added file: http://bugs.python.org/file44386/6135_1.patch ___ Python tracker

[issue6135] subprocess seems to use local encoding and give no choice

Steve Dower added the comment: 'mbcs' is exactly equivalent to what 'ansi' would be, so that's just a matter of knowing the name. I'm okay with adding an alias for it though. -- ___ Python tracker

[issue6135] subprocess seems to use local encoding and give no choice

STINNER Victor added the comment: > * add 'oem' encoding to make it easy to choose (but don't try and guess when > it is used) I suggest to open a separated issue for that. By the way, you might also add "ansi"? See also the aliasmbcs() function of the site module. Note: I never liked that

[issue6135] subprocess seems to use local encoding and give no choice

Steve Dower added the comment: I'll prepare a patch for the following: * add encoding and errors parameters to subprocess functions * add 'oem' encoding to make it easy to choose (but don't try and guess when it is used) EITHER * allow string or 2-tuple (stdin, stdout/err) or 3-tuple (stdin,

[issue6135] subprocess seems to use local encoding and give no choice

Martin Panter added the comment: Here is a tested illustration of how I would suggest to manually handle encoding with check_output(): >>> text_input = "©" >>> args = ("iconv", "--from-code", "ISO_8859-1", "--to-code", "UTF-8") >>> bytes_output = check_output(args,

[issue6135] subprocess seems to use local encoding and give no choice

Nick Coghlan added the comment: Since this issue was first filed, the "opener" parameter was added to the open() builtin: https://docs.python.org/3/library/functions.html#open I mention that, as the problem there is similar to the problem here: wanting to construct a buffered text-mode

[issue6135] subprocess seems to use local encoding and give no choice

Martin Panter added the comment: I added some review comments for Florian’s patches. Also, I suspect the code will not affect the text encoding when communicate() is used with multiple PIPE streams. -- ___ Python tracker

[issue6135] subprocess seems to use local encoding and give no choice

Martin Panter added the comment: This seems like a request for extra feature(s), rather than a bug report. FWIW I agree with Victor’s hesitation in . I doubt it is worth adding special support to have multiple pipes that all use text mode

[issue6135] subprocess seems to use local encoding and give no choice

Changes by Martin Panter vadmium...@gmail.com: -- nosy: +vadmium ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6135 ___ ___ Python-bugs-list

[issue6135] subprocess seems to use local encoding and give no choice

Joseph Perry added the comment: I've found a workaround by specifying the enviroment variable: my_env = os.environ my_env['PYTHONIOENCODING'] = 'utf-8' p = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE, env=my_env) I've attached

[issue6135] subprocess seems to use local encoding and give no choice

Chris Jerdonek added the comment: only one Popen instance (for the iconv call), but different encodings for stdin and stdout. Isn't that the exception rather than the rule? I think it actually makes sense, in at least 99.83% of cases ;-), to have a common encoding setting for all streams.

[issue6135] subprocess seems to use local encoding and give no choice

Chris Jerdonek added the comment: To my previous comment, issue 15648 shows the case where I was able to change the encoding for stdout in the child process but not stderr (which would require supporting two encodings in Popen to handle). -- ___

[issue6135] subprocess seems to use local encoding and give no choice

Éric Araujo mer...@netwok.org added the comment: So the users can control the encoding, and this is a doc bug. -- title: subprocess seems to use local 8-bit encoding and gives no choice - subprocess seems to use local encoding and give no choice ___

[issue6135] subprocess seems to use local encoding and give no choice

R. David Murray rdmur...@bitdance.com added the comment: If you decide this is only a doc bug, please see also related issue 12832. -- nosy: +r.david.murray ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6135

[issue6135] subprocess seems to use local encoding and give no choice

Antoine Pitrou pit...@free.fr added the comment: So the users can control the encoding, and this is a doc bug. Not really. People can control the encoding in the child process (and only if it's a Python 3 process of course). They can't control the encoding in the parent's subprocess pipes and