[issue22555] Tracking issue for adjustments to binary/text boundary handling

STINNER Victor added the comment: > https://vstinner.github.io/python30-listdir-undecodable-filenames.html Oh, thanks for mentioning my series of articles. It's also nice to see that we are now able to close this 4 years old issue! -- ___ Python

[issue22555] Tracking issue for adjustments to binary/text boundary handling

Nick Coghlan added the comment: Adding a link to the first post in a series of articles from Victor Stinner regarding the evolution over time of the text encoding assumptions in Python 3's operating system interfaces: https://vstinner.github.io/python30-listdir-undecodable-filenames.html

[issue22555] Tracking issue for adjustments to binary/text boundary handling

Nick Coghlan added the comment: Correction: I just rejected my proposed wsgiref in issue 22264 as failing to make a sufficient case for their practical utility, so that one is closed as well :) -- ___ Python tracker

[issue22555] Tracking issue for adjustments to binary/text boundary handling

Nick Coghlan added the comment: With PEPs 538 and 540 merged for Python 3.7 (so we'll almost always use UTF-8 instead of ASCII when the platform nominates the C or POSIX locale as the currently active one), and Windows previously switching to assuming UTF-8 instead of mbcs for binary

[issue22555] Tracking issue for adjustments to binary/text boundary handling

Nick Coghlan added the comment: Added another issue to the tracking list: * Automatically decode binary data in json.loads: issue #17909 -- dependencies: +Autodetecting JSON encoding ___ Python tracker

[issue22555] Tracking issue for adjustments to binary/text boundary handling

Nick Coghlan added the comment: Likely to be resolved, or at least significantly updated, for 3.6 due to PEP 528 and PEP 529: * Using sys.stdin consistently at the default interactive prompt: issue 1602 * Improved Unicode handling in the Windows console: issue 17620 * Allowing text encoding

[issue22555] Tracking issue for adjustments to binary/text boundary handling

Steve Dower added the comment: The thing about bogus assumptions is that Python should paper over those anyway. I can guarantee there's production code out there with the same assumptions. How do we make this work? No idea in the context of the bytes/str filename convention differences.

[issue22555] Tracking issue for adjustments to binary/text boundary handling

Nick Coghlan added the comment: In discussing the Windows aspects of the bytes/text boundary handling issues with Brett & Steve recently, I realised I hadn't clearly defined what "fixed" looked like from my perspective. The attached test case is an initial attempt at that. It currently fails

[issue22555] Tracking issue for adjustments to binary/text boundary handling

Steve Dower added the comment: Right now all of the tests fail on Windows by default (cp437 for me). If I change the default IO encoding to utf-8 (hacked into pylifecycle.c, since PYTHONIOENCODING is ignored by subprocesses using -E), the four "Misconfigured" tests crash at the os.fsencode()

[issue22555] Tracking issue for adjustments to binary/text boundary handling

Nick Coghlan added the comment: Thanks. I suspect some of the Windows problems are indeed due to bogus assumptions in my draft tests, but at the same time, folks should be able to invoke subprocesses with Unicode values without needing extensive knowledge of platform specific Unicode handling

[issue22555] Tracking issue for adjustments to binary/text boundary handling

Nick Coghlan added the comment: The Fedora RFE at https://bugzilla.redhat.com/show_bug.cgi?id=902094 to provide a C.UTF-8 locale by default has been addressed for Fedora 24 (the current Fedora Rawhide). This means the "LANG=C.UTF-8 python3" replacement for the ASCII-centric "LANG=C python3"

[issue22555] Tracking issue for adjustments to binary/text boundary handling

Nick Coghlan added the comment: For historical purposes, also linking the change in issue #19977 to enable surrogateescape by default on stdin and stdout when the OS claims the locale encoding is ASCII. -- dependencies: +Use "surrogateescape" error handler for sys.stdin and sys.stdout

[issue22555] Tracking issue for adjustments to binary/text boundary handling

Changes by Ethan Furman et...@stoneleaf.us: -- nosy: -ethan.furman ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22555 ___ ___ Python-bugs-list

[issue22555] Tracking issue for adjustments to binary/text boundary handling

Changes by Drekin dre...@gmail.com: -- nosy: +Drekin ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22555 ___ ___ Python-bugs-list mailing list

[issue22555] Tracking issue for adjustments to binary/text boundary handling

Changes by Ethan Furman et...@stoneleaf.us: -- nosy: +ethan.furman ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22555 ___ ___ Python-bugs-list

[issue22555] Tracking issue for adjustments to binary/text boundary handling

Changes by Berker Peksag berker.pek...@gmail.com: -- nosy: +berker.peksag ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22555 ___ ___

[issue22555] Tracking issue for adjustments to binary/text boundary handling

Nick Coghlan added the comment: I just went through the still-open issues referenced from here, and recommended deferring further consideration of all of the remaining items to 3.6: * utilities for clearing out surrogates from strings: issue 18814 * treating wsgistr as a serialisation format:

[issue22555] Tracking issue for adjustments to binary/text boundary handling

Nick Coghlan added the comment: PEP 461 landed, restoring binary interpolation support: https://hg.python.org/cpython/rev/8d802fb6ae32 There are also some relevant around standardising the C.UTF-8 locale currently available on some Linux systems: Fedora RFE:

[issue22555] Tracking issue for adjustments to binary/text boundary handling

Nick Coghlan added the comment: Slavek et al - you folks may be interested in this one, as it tracks several issues that I consider relevant to the Python 2 - 3 migration effort. Redoing the list in a way that should render the strike-throughs for closed issues: * Improved Windows console

[issue22555] Tracking issue for adjustments to binary/text boundary handling

Changes by Martin Panter vadmium...@gmail.com: -- nosy: +vadmium ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22555 ___ ___ Python-bugs-list

[issue22555] Tracking issue for adjustments to binary/text boundary handling

Nick Coghlan added the comment: Assigning to myself, since there's nothing specifically to *do* for this bug, it's just to make it easier to track the status of the various other RFEs it depends on. -- assignee: - ncoghlan type: - enhancement ___

[issue22555] Tracking issue for adjustments to binary/text boundary handling

Changes by Barry A. Warsaw ba...@python.org: -- nosy: +barry ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22555 ___ ___ Python-bugs-list mailing

[issue22555] Tracking issue for adjustments to binary/text boundary handling

New submission from Nick Coghlan: See PEP 478 for the PEP level items targeting 3.5: http://www.python.org/dev/peps/pep-0478/ This is a tracking issue to help me keep track of some lower level items that didn't make the release PEP: * Improved Windows console Unicode support (see

[issue22555] Tracking issue for adjustments to binary/text boundary handling

Nick Coghlan added the comment: PEP 461 binary interpolation implementation issue: http://bugs.python.org/issue20284 -- dependencies: +Add codecs.convert_surrogateescape to clean surrogate escaped strings, Add wsgiref.util.dump_wsgistr load_wsgistr, Allow backslashreplace error