Re: [Python-ideas] PEP 505: None-aware operators

2018-07-23 Thread Steve Dower
On 23Jul2018 1530, David Mertz wrote: Of course I don't mean that if implemented the semantics would be ambiguous... rather, the proper "swallowing" of different kinds of exceptions is not intuitively obvious, not even to you, Steve.  And if some decision was reached and documented, it would re

Re: [Python-ideas] PEP 505: None-aware operators

2018-07-23 Thread Steve Dower
On 23Jul2018 1145, Antoine Pitrou wrote: Le 23/07/2018 à 12:38, Steve Dower a écrit : General comment to everyone (not just Antoine): these arguments have zero value to me. Feel free to keep making them, but I am uninterested. So you're uninterested in learning from past mistakes?

Re: [Python-ideas] PEP 505: None-aware operators

2018-07-23 Thread Steve Dower
On 23Jul2018 1129, Antoine Pitrou wrote: Le 23/07/2018 à 12:25, Steve Dower a écrit : On 23Jul2018 , Antoine Pitrou wrote: On Mon, 23 Jul 2018 10:51:31 +0100 Steve Dower wrote: Which is the most important operator? - Personally, I think '?.&#x

Re: [Python-ideas] PEP 505: None-aware operators

2018-07-23 Thread Steve Dower
On 23Jul2018 , Antoine Pitrou wrote: On Mon, 23 Jul 2018 10:51:31 +0100 Steve Dower wrote: Which is the most important operator? - Personally, I think '?.' is the most valuable. For me, it's the most contentious. The fact that a sing

Re: [Python-ideas] PEP 505: None-aware operators

2018-07-23 Thread Steve Dower
Responding to a few more ideas that have come up here. Again, apologies for not directing them to the original authors, but I want to focus on the ideas that are leading towards a more informed decision, and not getting distracted by providing customised examples for people or getting into side

Re: [Python-ideas] PEP 505: None-aware operators

2018-07-23 Thread Steve Dower
On 23Jul2018 0151, Steven D'Aprano wrote: What if there was a language supported, non-hackish way to officially delay evaluation of expressions until explicitly requested? The current spelling for this is "lambda: delayed-expression" and the way to request the value is "()". :) (I'm not even

Re: [Python-ideas] Python docs page: In what ways is None special

2018-07-23 Thread Steve Dower
On 23Jul2018 1003, Jonathan Fine wrote: This arises out of PEP 505 - None-aware operators. I thought, a page on how None is special would be nice. I've not found such a page on the web. We do have === https://docs.python.org/3/library/constants.html None The sole value of the type NoneType. None

Re: [Python-ideas] PEP 505: None-aware operators

2018-07-20 Thread Steve Dower
On 20Jul2018 1119, Brendan Barnwell wrote: In this situation I lean toward "explicit is better than implicit" --- if you want to compare against None, you should do so explicitly --- and "special cases aren't special enough to break the rules" --- that is, None is not special enough to warrant

Re: [Python-ideas] PEP 505: None-aware operators

2018-07-20 Thread Steve Dower
Just for fun, I decided to go through some recently written code by some genuine Python experts (without their permission...) to see what changes would be worth taking. So I went to the sources of our github bots. Honestly, I only found three places that were worth changing (though I'm now kind

Re: [Python-ideas] PEP 505: None-aware operators

2018-07-19 Thread Steve Dower
Thanks everyone for the feedback and discussion so far. I want to address some of the themes, so apologies for not quoting individuals and for doing this in one post instead of twenty. -- * "It looks like line noise" Thanks for the feedback. There's nothing constructive for me to take f

Re: [Python-ideas] PEP 505: None-aware operators

2018-07-18 Thread Steve Dower
Thanks! Bit of discussion below about precedence, but thanks for spotting the typos. On 18Jul2018 1318, MRAB wrote: On 2018-07-18 18:43, Steve Dower wrote: Grammar changes --- The following rules of the Python grammar are updated to read::   augassign

[Python-ideas] PEP 505: None-aware operators

2018-07-18 Thread Steve Dower
evision$ Last-Modified: $Date$ Author: Mark E. Haase , Steve Dower Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 18-Sep-2015 Python-Version: 3.8 Abstract Several modern programming languages have so-called "``null``-coalescing" or "``null``- aware

Re: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three!

2018-04-08 Thread Steve Dower
# Dict display data = { key_a: local_a := 1, key_b: local_b := 2, key_c: local_c := 3, } Isn’t this a set display with local assignments and type annotations? :o) (I’m -1 on all of these ideas, btw. None help readability for me, and I read much more code than I wr

Re: [Python-ideas] New PEP proposal -- Pathlib Module ShouldContain All File Operations -- version 2

2018-03-23 Thread Steve Dower
I had a colleague complaining to me the other day about having to search multiple packages for the right function to move a file (implying: with the same semantics as drag-drop). If there isn’t a pathtools library on PyPI yet, this would certainly be valuable for newer developers. My view on P

Re: [Python-ideas] Complicate str methods

2018-02-08 Thread Steve Dower
Easily fixed by installing one of the alternate regex libraries. re performance and its edge cases have been discussed endlessly. Please look it up before restarting that discussion. Top-posted from my Windows phone From: Franklin? Lee Sent: Thursday, February 8, 2018 2:46 To: Serhiy Storchaka

Re: [Python-ideas] Possible Enhancement to py Launcher - set default

2018-02-07 Thread Steve Dower
Checking the Version (!=SysVersion) property should be enough (and perhaps we need to set it properly on install). The launcher currently only works with PythonCore entries anyway, so no need to worry about other distros. PEP 514 allows for other keys to be added as well (it specifies a minimum

Re: [Python-ideas] Format mini-language for lakh and crore

2018-01-29 Thread Steve Dower
Someone would have to check, but presumably the CRT on Windows is converting the natively thread-local locale into a process-wide locale for POSIX compatibility, which means it can probably be easily bypassed without having to use specific overloads. Top-posted from my Windows phone From: Nath

Re: [Python-ideas] Windows Best Fit Encodings

2018-01-19 Thread Steve Dower
On 20Jan2018 0518, M.-A. Lemburg wrote: do you know of a definite resource for Windows code pages on MSDN or another official MS website ? I don't know of anything sorry, and my quick search didn't turn up anything public. But I can at least confirm that the internal table for cp1252 has the

Re: [Python-ideas] Support WHATWG versions of legacy encodings

2018-01-11 Thread Steve Dower
On 12Jan2018 0342, Random832 wrote: On Thu, Jan 11, 2018, at 04:55, Serhiy Storchaka wrote: The way of solving this issue in Python is using an error handler. The "surrogateescape" error handler is specially designed for lossless reversible decoding. It maps every unassigned byte in the range 0x

Re: [Python-ideas] Make functions, methods and descriptor types living in the types module

2018-01-11 Thread Steve Dower
I certainly have code that joins __module__ with __name__ to create a fully-qualified name (with special handling for those builtins that are not in builtins), and IIUC __qualname__ doesn't normally include the module name either (it's intended for nested types/functions). Can we make it visib

Re: [Python-ideas] Allow to compile debug extension againstreleasePython in Windows

2018-01-10 Thread Steve Dower
pping through both Python and C code simultaneously (https://docs.microsoft.com/en-us/visualstudio/python/debugging-mixed-mode). It might make your life a little easier. Top-posted from my Windows phone From: Ivan Pozdeev Sent: Wednesday, January 10, 2018 5:47 To: Steve Dower; python-ideas@pytho

Re: [Python-ideas] Allow to compile debug extension against releasePython in Windows

2018-01-07 Thread Steve Dower
It’s not a good idea. You end up with two different C runtimes in memory that cannot communicate, and many things will not work properly. If you compile your debug build extension with the non-debug CRT (/MD rather than /MDd) you will lose asserts, but otherwise it will work fine and the quoted

Re: [Python-ideas] Looking for input to help with the pip situation

2017-11-15 Thread Steve Dower
taller behaviour, and adding pythonX and pythonX.Y commands would be useful (and Guido would presumably delegate resolving that to Steve Dower as the Windows installer maintainer). If you write such a PEP, please also research and write up the issues with modifying PATH on Windows (they're largel

Re: [Python-ideas] PEP draft: context variables

2017-10-13 Thread Steve Dower
On 13Oct2017 1132, Yury Selivanov wrote: On Fri, Oct 13, 2017 at 1:45 PM, Ethan Furman wrote: On 10/13/2017 09:48 AM, Steve Dower wrote: On 13Oct2017 0941, Yury Selivanov wrote: Actually, capturing context at the moment of coroutine creation (in PEP 550 v1 semantics) will not work at all

Re: [Python-ideas] PEP draft: context variables

2017-10-13 Thread Steve Dower
On 13Oct2017 0941, Yury Selivanov wrote: On Fri, Oct 13, 2017 at 3:25 AM, Nick Coghlan wrote: [..] However, considering that coroutines are almost always instantiated at the point where they're awaited, I do concede that creation time context capture would likely also work out OK for the corout

Re: [Python-ideas] PEP draft: context variables

2017-10-11 Thread Steve Dower
On 11Oct2017 0458, Koos Zevenhoven wrote: ​Exactly. You did say it less politely than I did, but this is exactly how I thought about it. And I'm not sure people got it the first time. Yes, perhaps a little harsh. However, if I released a refactoring tool that moved function calls that far, peo

Re: [Python-ideas] PEP draft: context variables

2017-10-10 Thread Steve Dower
Nick: “I like Yury's example for this, which is that the following two examples are currently semantically equivalent, and we want to preserve that equivalence:     with decimal.localcontext() as ctx:     ctc.prex = 30         for i in gen():            pass     g = gen()     with decimal.lo

Re: [Python-ideas] Add pathlib.Path.write_json andpathlib.Path.read_json

2017-03-27 Thread Steve Dower
It was enough of a benefit for text (and I never forget the argument order for writing text to a file, unlike json.dump(file_or_data?, data_or_file?) ) +1 Top-posted from my Windows Phone -Original Message- From: "Paul Moore" Sent: ‎3/‎27/‎2017 5:57 To: "Ram Rachum" Cc: "python-ideas"

Re: [Python-ideas] Adding an 'errors' argument to print

2017-03-26 Thread Steve Dower
On 26Mar2017 0707, Nick Coghlan wrote: Perhaps it would be worth noting in the table of error handlers at https://docs.python.org/3/library/codecs.html#error-handlers that backslashreplace is used by the `ascii()` builtin and the associated format specifiers backslashreplace is also the default

Re: [Python-ideas] site.py uses os.sep to determine platform

2017-02-12 Thread Steve Dower
It depends on what the code beneath the if is doing. "Windows" may be a shorthand for "uses backslash", in which case the check is correct. We should certainly match the check to the intent, but we need to determine the intent first. Cheers, Steve Top-posted from my Windows Phone -Origina

Re: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator

2017-02-10 Thread Steve Dower
On 10Feb2017 1400, Stephan Hoyer wrote: An important note is that ideally, we would still have way of indicating that Spam.func should exists in on the Spam class itself, even if it doesn't define the implementation. I suppose an abstractmethod overwritten by the later definition might do the tri

Re: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator

2017-02-10 Thread Steve Dower
y Windows Phone -Original Message- From: "Markus Meskanen" Sent: ‎2/‎10/‎2017 10:18 To: "Paul Moore" Cc: "Python-Ideas" ; "Steve Dower" Subject: Re: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator

Re: [Python-ideas] Fwd: Define a method or function attributeoutside of a class with the dot operator

2017-02-10 Thread Steve Dower
Since votes seem to be being counted and used for debate purposes, I am -1 to anything that encourages or condones people adding functionality to classes outside of the class definition. (Monkeypatching in my mind neither condones or encourages, and most descriptions come with plenty of caveats

Re: [Python-ideas] Using Python for end user applications

2017-02-07 Thread Steve Dower
You can leave python.exe out of your distribution to avoid it showing up on PATH, or if your stub explicitly LoadLibrary's vcruntime140.dll and then python36.dll you should be able to put them wherever you like. I think what we really want is a self-extractor that "installs" into the user's App

Re: [Python-ideas] math.nextafter

2017-02-04 Thread Steve Dower
These days, the subset of C99 supported by MSVC is "most" of it, so feel free to start off by assuming the best, at least for new features (the version we use for 2.7 obviously is not improving). Cheers, Steve Top-posted from my Windows Phone -Original Message- From: "tritium-l...@sdam

Re: [Python-ideas] Unified TLS API for Python

2017-02-03 Thread Steve Dower
On 02Feb2017 0601, Cory Benfield wrote: 4. Eventually, integrating the two backends above into the standard library so that it becomes possible to reduce the reliance on OpenSSL. This would allow future Python implementations to ship with all of their network protocol libraries supporting platfo

Re: [Python-ideas] pathlib suggestions

2017-01-25 Thread Steve Dower
On 25Jan2017 0816, Petr Viktorin wrote: On 01/25/2017 04:33 PM, Todd wrote: But what if the .tar.gz file is called "spam-4.2.5-final.tar.gz"? Existing tools like glob and endswith() can deal with the ".tar.gz" extension reliably, but "fullsuffix" would, arguably, not give the ans

Re: [Python-ideas] PEP 540: Add a new UTF-8 mode

2017-01-06 Thread Steve Dower
Passing universal_newlines will use whatever locale.getdefaultencoding() returns (which at least on Windows is useless enough that I added the encoding and errors parameters in 3.6). So it sounds like it'll only actually do Unicode on Linux if enough of the planets have aligned, which is what Vi

Re: [Python-ideas] New PyThread_tss_ C-API for CPython

2016-12-24 Thread Steve Dower
Right. Platforms that have a defined invalid value don't need the struct, and so they can define the type differently. It just means we also need to provide a macro for testing whether it's been created or not, and users should genuinely treat the value as opaque. Cheers, Steve Top-posted from

Re: [Python-ideas] Enhancing vars()

2016-12-12 Thread Steve Dower
I'm +1. This bites me far too often. > in the past developers were encouraged to put only "useful" attributes in __dir__. Good. If I'm getting vars() I really only want the useful ones. If I need interesting/secret ones then I'll getattr for them. Cheers, Steve Top-posted from my Windows Phone

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

2016-11-28 Thread Steve Dower
On 28Nov2016 1433, Steve Dower wrote: On 28Nov2016 1419, Nathaniel Smith wrote: I'd suggest that we additional specify that if we find a foo.missing.py, then the code is executed but -- unlike a regular module load -- it's not automatically inserted into sys.modules["foo"]

Re: [Python-ideas] PEP: Distributing a Subset of the Standard Library

2016-11-28 Thread Steve Dower
On 28Nov2016 1419, Nathaniel Smith wrote: I'd suggest that we additional specify that if we find a foo.missing.py, then the code is executed but -- unlike a regular module load -- it's not automatically inserted into sys.modules["foo"]. That seems like it could only create confusion. And it doesn

Re: [Python-ideas] Alternative to PEP 532: delayed evaluation ofexpressions

2016-11-12 Thread Steve Dower
Would it be helped by an explicit "free variable" marker? (I'm sure I've seen someone demo a prototype of this): >>> data_frame.subset($height > 100 and $age < 5) Which essentially translates into: >>> data_frame.subset(lambda **a: a["height"] > 100 and a["age"] < 5) Maybe the generated thunk

Re: [Python-ideas] Proposal for default character representation

2016-10-15 Thread Steve Dower
FWIW, Python 3.6 should print this in the console just fine. Feel free to upgrade whenever you're ready. Cheers, Steve -Original Message- From: "Mikhail V" Sent: ‎10/‎12/‎2016 16:07 To: "M.-A. Lemburg" Cc: "python-ideas@python.org" Subject: Re: [Python-ideas] Proposal for default char

Re: [Python-ideas] (Windows-only - calling Steve Dower) Is Python forWindows using PGO? If not consider this a suggestion.

2016-09-18 Thread Steve Dower
; Subject: [Python-ideas] (Windows-only - calling Steve Dower) Is Python forWindows using PGO? If not consider this a suggestion. Hello, Is Python for Windows using PGO (Profile Guided Optimizations)? If not consider this a suggestion. Best regards, JM ___

Re: [Python-ideas] (Windows-only - calling Steve Dower) Consider addinga symlink to pip in the same location as the py launcher

2016-09-18 Thread Steve Dower
rom my Windows Phone -Original Message- From: "João Matos" Sent: ‎9/‎17/‎2016 3:57 To: "python-ideas@python.org" Subject: [Python-ideas] (Windows-only - calling Steve Dower) Consider addinga symlink to pip in the same location as the py launcher Hello, If Py3.5 is

Re: [Python-ideas] Let’s make escaping in f-literals impossible

2016-08-29 Thread Steve Dower
On 29Aug2016 1433, Eric V. Smith wrote: On 8/29/2016 5:26 PM, Ethan Furman wrote: Update the PEP, then it's a bugfix. ;) Heh. I guess that's true. But it's sort of a big change, so shipping beta 1 with the code not agreeing with the PEP rubs me the wrong way. Or, I could stop worrying and ty

Re: [Python-ideas] Let’s make escaping in f-literals impossible

2016-08-19 Thread Steve Dower
On 19Aug2016 1157, Guido van Rossum wrote: I don't think we should take action now. Would it make sense, as a precaution, to declare the PEP provisional for one release? Then we can develop a sense of whether the current approach causes real problems. We could also emit some kind of warning if

Re: [Python-ideas] Let’s make escaping in f-literals impossible

2016-08-18 Thread Steve Dower
On 18Aug2016 1215, Terry Reedy wrote: On 8/18/2016 12:50 PM, Steve Dower wrote: I don't think f'{x.partition('-')[0]}' is any less readable as a result of the reused quotes, I find it hard to not read f'{x.partition(' + ')[0]}' as string concatena

Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread Steve Dower
On 18Aug2016 1036, Terry Reedy wrote: On 8/18/2016 11:25 AM, Steve Dower wrote: In this case, we would announce in 3.6 that using bytes as paths on Windows is no longer deprecated, My understanding is the the first 2 fixes refine the deprecation rather than reversing it. And #3 simply

Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread Steve Dower
On 18Aug2016 1018, MRAB wrote: Could we use still call it 'mbcs', but use 'surrogateescape'? surrogateescape is used for escaping undecodable values when you want to represent arbitrary bytes in Unicode. It's the wrong direction for this situation - we are starting with valid Unicode and en

Re: [Python-ideas] Let’s make escaping in f-literals impossible

2016-08-18 Thread Steve Dower
On 18Aug2016 0950, Steve Dower wrote: I'm generally inclined to agree, especially as someone who is very likely to be implementing syntax highlighting and completion support within f-literals. I also really don't like the subject line. "Do not require string escapes within e

Re: [Python-ideas] Let’s make escaping in f-literals impossible

2016-08-18 Thread Steve Dower
I'm generally inclined to agree, especially as someone who is very likely to be implementing syntax highlighting and completion support within f-literals. I stepped out of the original discussion near the start as it looked like we were going to end up with interleaved strings and normal expr

Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread Steve Dower
On 18Aug2016 0900, Chris Angelico wrote: On Fri, Aug 19, 2016 at 1:54 AM, Steve Dower wrote: On 18Aug2016 0829, Chris Angelico wrote: The second call to glob doesn't have any Unicode characters at all, the way I see it - it's all bytes. Am I completely misunderstanding this?

Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread Steve Dower
On 18Aug2016 0829, Chris Angelico wrote: The second call to glob doesn't have any Unicode characters at all, the way I see it - it's all bytes. Am I completely misunderstanding this? You're not the only one - I think this has been the most common misunderstanding. On Windows, the paths as st

Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread Steve Dower
Summary for python-dev. This is the email I'm proposing to take over to the main mailing list to get some actual decisions made. As I don't agree with some of the possible recommendations, I want to make sure that they're represented fairly. I also want to summarise the background leading to

Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread Steve Dower
as intended to discourage. (As soon as you open(..., 'r') you're only affected by this change if you explicitly encode again with mbcs.) Top-posted from my Windows Phone -Original Message- From: "Stephen J. Turnbull" Sent: ‎8/‎17/‎2016 19:43 To: "Steve Dowe

Re: [Python-ideas] Fix default encodings on Windows

2016-08-17 Thread Steve Dower
On 17Aug2016 0901, Nick Coghlan wrote: On 17 August 2016 at 02:06, Chris Barker wrote: So the Solution is to either: (A) get everyone to use Unicode "properly", which will work on all platforms (but only on py3.5 and above?) or (B) kludge some *nix-compatible support for byte paths into Wi

Re: [Python-ideas] Fix default encodings on Windows

2016-08-17 Thread Steve Dower
On 17Aug2016 0235, Stephen J. Turnbull wrote: Paul Moore writes: > On 16 August 2016 at 16:56, Steve Dower wrote: > > This discussion is for the developers who insist on using bytes > > for paths within Python, and the question is, "how do we best > > represe

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Steve Dower
I've just created http://bugs.python.org/issue27781 with a patch removing use of the *A API from posixmodule.c and changing the default FS encoding to utf-8. Since we're still discussing whether the encoding should be utf-8 or something else, let's keep that here. But if you want to see how th

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Steve Dower
On 16Aug2016 1915, Brendan Barnwell wrote: On 2016-08-16 17:14, Steve Dower wrote: The existence of bugs in other applications is not a good reason to help people create new bugs. I haven't been following all the details in this thread, but isn't the whole purpose of this propo

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Steve Dower
On 16Aug2016 1650, Victor Stinner wrote: 2016-08-17 1:27 GMT+02:00 Steve Dower : filenameb = os.listdir(b'.')[0] # Python 3.5 encodes Unicode (UTF-16) to the ANSI code page # what if Python 3.7 encodes Unicode (UTF-16) to UTF-8? print("filename bytes: %a" %

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Steve Dower
On 16Aug2016 1603, Victor Stinner wrote: 2016-08-16 17:56 GMT+02:00 Steve Dower : 2. Windows file system encoding is *always* UTF-16. There's no "assuming mbcs" or "assuming ACP" or "assuming UTF-8" or "asking the OS what encoding it is". We know ex

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Steve Dower
On 16Aug2016 1006, Sven R. Kunze wrote: Question is: are we in a hurry? Has somebody too little time to wait for the "Right Thing" to happen? Not really in a hurry. It's just that I decided to attack a few of the encoding issues on Windows and this was one of them. Ideally I'd want to get th

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Steve Dower
I just want to clearly address two points, since I feel like multiple posts have been unclear on them. 1. The bytes API was deprecated in 3.3 and it is listed in https://docs.python.org/3/whatsnew/3.3.html. Lack of mention in the docs is an unfortunate oversight, but it was certainly announced

Re: [Python-ideas] Fix default encodings on Windows

2016-08-16 Thread Steve Dower
sted from my Windows Phone -Original Message- From: "Paul Moore" Sent: ‎8/‎16/‎2016 3:54 To: "Nick Coghlan" Cc: "python-ideas" Subject: Re: [Python-ideas] Fix default encodings on Windows On 15 August 2016 at 19:26, Steve Dower wrote: > Passing path_as_byt

Re: [Python-ideas] Fix default encodings on Windows

2016-08-15 Thread Steve Dower
On 15Aug2016 1819, eryk sun wrote: On Mon, Aug 15, 2016 at 6:26 PM, Steve Dower wrote: (Frankly I don't mind what encoding we use, and I'd be quite happy to force bytes paths to be UTF-16-LE encoded, which would also round-trip invalid surrogate pairs. But that would pre

Re: [Python-ideas] Fix default encodings on Windows

2016-08-15 Thread Steve Dower
On 15Aug2016 1126, Steve Dower wrote: My proposal is to remove all use of the *A APIs and only use the *W APIs. That completely removes the (already deprecated) use of bytes as paths. I then propose to change the (unused on Windows) sys.getfsdefaultencoding() to 'utf-8' and handle b

Re: [Python-ideas] Fix default encodings on Windows

2016-08-15 Thread Steve Dower
On 15Aug2016 0954, Random832 wrote: On Mon, Aug 15, 2016, at 12:35, Steve Dower wrote: I'm still not sure we're talking about the same thing right now. For `open(path_as_bytes).read()`, are we talking about the way path_as_bytes is passed to the file system? Or the codec used to

Re: [Python-ideas] Fix default encodings on Windows

2016-08-15 Thread Steve Dower
- From: "Random832" Sent: ‎8/‎15/‎2016 6:41 To: "Steve Dower" ; "Stephen J. Turnbull" Cc: "Victor Stinner" ; "python-ideas" Subject: Re: [Python-ideas] Fix default encodings on Windows On Mon, Aug 15, 2016, at 09:23, Steve Dower wrote: > I

Re: [Python-ideas] Fix default encodings on Windows

2016-08-15 Thread Steve Dower
om my Windows Phone -Original Message- From: "Stephen J. Turnbull" Sent: ‎8/‎14/‎2016 22:06 To: "Steve Dower" Cc: "Victor Stinner" ; "python-ideas" ; "Random832" Subject: RE: [Python-ideas] Fix default encodings on Windows Steve Dower writes:

Re: [Python-ideas] Fix default encodings on Windows

2016-08-14 Thread Steve Dower
om my Windows Phone -Original Message- From: "Victor Stinner" Sent: ‎8/‎14/‎2016 9:20 To: "Steve Dower" Cc: "Stephen J. Turnbull" ; "python-ideas" ; "Random832" Subject: Re: [Python-ideas] Fix default encodings on Windows > The last

Re: [Python-ideas] Fix default encodings on Windows

2016-08-13 Thread Steve Dower
The last point is correct: if you get bytes from a file system API, you should be able to pass them back in without losing information. CP_ACP (a.k.a. the *A API) does not allow this, so I'm proposing using the *W API everywhere and encoding to utf-8 when the user wants/gives bytes. Top-posted

Re: [Python-ideas] Fix default encodings on Windows

2016-08-13 Thread Steve Dower
On 13Aug2016 0523, Random832 wrote: On Sat, Aug 13, 2016, at 04:12, Stephen J. Turnbull wrote: Steve Dower writes: > ISTM that changing sys.getfilesystemencoding() on Windows to > "utf-8" and updating path_converter() (Python/posixmodule.c; I think this proposal requires the

Re: [Python-ideas] Fix default encodings on Windows

2016-08-13 Thread Steve Dower
Just a heads-up that I've assigned http://bugs.python.org/issue1602 to myself and started a patch for the console changes. Let's move the console discussion back over there. Hopefully it will show up in 3.6.0b1, but if you're prepared to apply a patch and test on Windows, feel free to grab my

Re: [Python-ideas] Fix default encodings on Windows

2016-08-12 Thread Steve Dower
I was thinking we would end up using the console API for input but stick with the standard handles for output, mostly to minimize the amount of magic switching we have to do. But since we can just switch the entire stream object in __std*__ once at startup if nothing is redirected it probably is

Re: [Python-ideas] Fix default encodings on Windows

2016-08-11 Thread Steve Dower
Unless someone else does the implementation, I'd rather add a utf8-readsig encoding that initially only skips a utf8 BOM - notably, you always get the same encoding, it just sometimes skips the first three bytes. I think we can change this later to detect and switch to utf16 without it being di

Re: [Python-ideas] Fix default encodings on Windows

2016-08-10 Thread Steve Dower
On 10Aug2016 1630, Random832 wrote: On Wed, Aug 10, 2016, at 19:04, eryk sun wrote: Using 'mbcs' doesn't work reliably with arbitrary bytes paths in locales that use a DBCS codepage such as 932. Er... utf-8 doesn't work reliably with arbitrary bytes paths either, unless you intend to use surro

Re: [Python-ideas] Fix default encodings on Windows

2016-08-10 Thread Steve Dower
On 10Aug2016 1431, Chris Angelico wrote: On Thu, Aug 11, 2016 at 6:09 AM, Random832 wrote: On Wed, Aug 10, 2016, at 15:22, Steve Dower wrote: Why? What's the use case? [byte paths] Allowing library developers who support POSIX and Windows to just use bytes everywhere to represent

Re: [Python-ideas] Fix default encodings on Windows

2016-08-10 Thread Steve Dower
On 10Aug2016 1226, Random832 wrote: On Wed, Aug 10, 2016, at 15:08, Steve Dower wrote: Testing with obscure filenames and strings is where help will be needed most :) How about filenames with invalid surrogates? For added fun, consider that the file system encoding is normally used with

Re: [Python-ideas] Fix default encodings on Windows

2016-08-10 Thread Steve Dower
On 10Aug2016 1146, Random832 wrote: On Wed, Aug 10, 2016, at 14:10, Steve Dower wrote: To summarise the proposals (remembering that these would only affect Python 3.6 on Windows): * change sys.getfilesystemencoding() to return 'utf-8' * automatically decode byte paths assuming they

Re: [Python-ideas] Fix default encodings on Windows

2016-08-10 Thread Steve Dower
On 10Aug2016 1144, Paul Moore wrote: I presume you'd be targeting 3.7 for this change. Does 3.6 seem too aggressive? I think I have time to implement the changes before beta 1, as it's mostly changing default values and mopping up resulting breaks. (Doing something like reimplementing files

[Python-ideas] Fix default encodings on Windows

2016-08-10 Thread Steve Dower
I suspect there's a lot of discussion to be had around this topic, so I want to get it started. There are some fairly drastic ideas here and I need help figuring out whether the impact outweighs the value. Some background: within the Windows API, the preferred encoding is UTF-16. This is a 16-