Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread eryk sun
On Thu, Aug 18, 2016 at 2:32 AM, Stephen J. Turnbull wrote: > > So it's not just invalid surrogate *pairs*, it's invalid surrogates of > all kinds. This means that it's theoretically possible (though I > gather that it's unlikely in the extreme) for a real Windows filename > to indistinguishable

Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread Steve Dower
"You consistently ignore Makefiles, .ini, etc." Do people really do open('makefile', 'rb'), extract filenames and try to use them without ever decoding the file contents? I've honestly never seen that, and it certainly looks like the sort of thing Python 3 was intended to discourage. (As soon a

Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread Steve Dower
Summary for python-dev. This is the email I'm proposing to take over to the main mailing list to get some actual decisions made. As I don't agree with some of the possible recommendations, I want to make sure that they're represented fairly. I also want to summarise the background leading to

Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread Chris Angelico
On Fri, Aug 19, 2016 at 1:25 AM, Steve Dower wrote: open('test\uAB00.txt', 'wb').close() import glob glob.glob('test*') > ['test\uab00.txt'] glob.glob(b'test*') > [b'test?.txt'] > > The Unicode character in the second call to glob is missing information. You > can observe the s

[Python-ideas] Let’s make escaping in f-literals impossible

2016-08-18 Thread Philipp A.
Hi, I originially posted this via google groups, which didn’t make it through to the list proper, sorry! Read it here please: https://groups.google.com/forum/#!topic/python-ideas/V1U6DGL5J1s My arguments are basically: 1. f-literals are semantically not strings, but expressions. 2. Their es

Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread Random832
On Thu, Aug 18, 2016, at 11:29, Chris Angelico wrote: > glob.glob('test*') > > ['test\uab00.txt'] > glob.glob(b'test*') > > [b'test?.txt'] > > > > The Unicode character in the second call to glob is missing information. > > Apologies if this is just noise, but I'm a little confused by t

Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread Steve Dower
On 18Aug2016 0829, Chris Angelico wrote: The second call to glob doesn't have any Unicode characters at all, the way I see it - it's all bytes. Am I completely misunderstanding this? You're not the only one - I think this has been the most common misunderstanding. On Windows, the paths as st

Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread Chris Angelico
On Fri, Aug 19, 2016 at 1:54 AM, Steve Dower wrote: > On 18Aug2016 0829, Chris Angelico wrote: >> >> The second call to glob doesn't have any Unicode characters at all, >> the way I see it - it's all bytes. Am I completely misunderstanding >> this? > > > You're not the only one - I think this has

Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread Steve Dower
On 18Aug2016 0900, Chris Angelico wrote: On Fri, Aug 19, 2016 at 1:54 AM, Steve Dower wrote: On 18Aug2016 0829, Chris Angelico wrote: The second call to glob doesn't have any Unicode characters at all, the way I see it - it's all bytes. Am I completely misunderstanding this? You're not the

Re: [Python-ideas] Let’s make escaping in f-literals impossible

2016-08-18 Thread Chris Angelico
On Fri, Aug 19, 2016 at 1:05 AM, Philipp A. wrote: > the embedded expressions are just normal python. the embedded strings just > normal strings. you can simply switch between both using <{> and > <[format]}>. > The trouble with that way of thinking is that, to a human, the braces contain somethi

Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread Chris Angelico
On Fri, Aug 19, 2016 at 2:07 AM, Steve Dower wrote: > I think so, though I find the wording a little awkward (and on rereading, my > original wording was pretty bad). How about: > > "The second call to glob has replaced the Unicode character with '?', which > means the actual filename cannot be re

Re: [Python-ideas] Let’s make escaping in f-literals impossible

2016-08-18 Thread Random832
On Thu, Aug 18, 2016, at 12:17, Chris Angelico wrote: > The trouble with that way of thinking is that, to a human, the braces > contain something. They don't "uncontain" it. Those braced expressions > are still part of a string; they just have this bit of magic that gets > them evaluated. Consider

Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread eryk sun
On Thu, Aug 18, 2016 at 4:07 PM, Steve Dower wrote: > On 18Aug2016 0900, Chris Angelico wrote: >> >> On Fri, Aug 19, 2016 at 1:54 AM, Steve Dower >> wrote: >>> >>> On 18Aug2016 0829, Chris Angelico wrote: The second call to glob doesn't have any Unicode characters at all, the

Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread Chris Angelico
On Fri, Aug 19, 2016 at 2:39 AM, eryk sun wrote: > They're all just characters in the context of Unicode, so I think it's > clearest to use the character code, e.g.: > > The second call to glob has replaced the U+AB00 character with '?', > which means ... Technically the character has bee

Re: [Python-ideas] Let’s make escaping in f-literals impossible

2016-08-18 Thread Steve Dower
I'm generally inclined to agree, especially as someone who is very likely to be implementing syntax highlighting and completion support within f-literals. I stepped out of the original discussion near the start as it looked like we were going to end up with interleaved strings and normal expr

Re: [Python-ideas] Let’s make escaping in f-literals impossible

2016-08-18 Thread Steve Dower
On 18Aug2016 0950, Steve Dower wrote: I'm generally inclined to agree, especially as someone who is very likely to be implementing syntax highlighting and completion support within f-literals. I also really don't like the subject line. "Do not require string escapes within expressions in f-lit

Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread MRAB
On 2016-08-16 16:56, Steve Dower wrote: I just want to clearly address two points, since I feel like multiple posts have been unclear on them. 1. The bytes API was deprecated in 3.3 and it is listed in https://docs.python.org/3/whatsnew/3.3.html. Lack of mention in the docs is an unfortunate ove

Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread Random832
On Thu, Aug 18, 2016, at 13:18, MRAB wrote: > > If you see an alternative choice to those listed above, feel free to > > contribute it. Otherwise, can we focus the discussion on these (or any > > new) choices? > > > Could we use still call it 'mbcs', but use 'surrogateescape'? Er, this discussion

Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread Steve Dower
On 18Aug2016 1018, MRAB wrote: Could we use still call it 'mbcs', but use 'surrogateescape'? surrogateescape is used for escaping undecodable values when you want to represent arbitrary bytes in Unicode. It's the wrong direction for this situation - we are starting with valid Unicode and en

Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread Terry Reedy
On 8/18/2016 11:25 AM, Steve Dower wrote: In this case, we would announce in 3.6 that using bytes as paths on Windows is no longer deprecated, My understanding is the the first 2 fixes refine the deprecation rather than reversing it. And #3 simply applies it. -- Terry Jan Reedy _

Re: [Python-ideas] Let’s make escaping in f-literals impossible

2016-08-18 Thread Brett Cannon
On Thu, 18 Aug 2016 at 08:32 Philipp A. wrote: > [SNIP] > Brett Cannon schrieb am Mi., 17. Aug. 2016 um > 20:28 Uhr: > >> They are still strings, there is just post-processing on the string >> itself to do the interpolation. >> > > Sounds hacky to me. I’d rather see a proper parser for them, whi

Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread Steve Dower
On 18Aug2016 1036, Terry Reedy wrote: On 8/18/2016 11:25 AM, Steve Dower wrote: In this case, we would announce in 3.6 that using bytes as paths on Windows is no longer deprecated, My understanding is the the first 2 fixes refine the deprecation rather than reversing it. And #3 simply applie

Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread eryk sun
On Thu, Aug 18, 2016 at 4:44 PM, Chris Angelico wrote: > On Fri, Aug 19, 2016 at 2:39 AM, eryk sun wrote: >> They're all just characters in the context of Unicode, so I think it's >> clearest to use the character code, e.g.: >> >> The second call to glob has replaced the U+AB00 character with

Re: [Python-ideas] Let’s make escaping in f-literals impossible

2016-08-18 Thread Terry Reedy
On 8/18/2016 12:50 PM, Steve Dower wrote: > I'm generally inclined to agree, especially as someone who is very > likely to be implementing syntax highlighting and completion support > within f-literals. I consider these separate issues. IDLE currently provides filename completion support within

Re: [Python-ideas] Let’s make escaping in f-literals impossible

2016-08-18 Thread Random832
On Thu, Aug 18, 2016, at 15:15, Terry Reedy wrote: > This is the crux of this thread. Is an f-string a single string that > contains magically handled code, or interleaved strings using { and } as > closing and opening quotes (which is backwards from their normal > function of being opener and

Re: [Python-ideas] Let’s make escaping in f-literals impossible

2016-08-18 Thread Steve Dower
On 18Aug2016 1215, Terry Reedy wrote: On 8/18/2016 12:50 PM, Steve Dower wrote: I don't think f'{x.partition('-')[0]}' is any less readable as a result of the reused quotes, I find it hard to not read f'{x.partition(' + ')[0]}' as string concatenation. That's a fair counter-example. Though f

Re: [Python-ideas] Allow manual creation of DirEntry objects

2016-08-18 Thread Brendan Moloney
Thanks, opened an issue here: http://bugs.python.org/issue27796 -Brendan From: [email protected] [[email protected]] on behalf of Guido van Rossum [[email protected]] Sent: Wednesday, August 17, 2016 7:20 AM To: Nick Coghlan; Brendan Moloney Cc: Victor Stinne

Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread Chris Barker
On Thu, Aug 18, 2016 at 6:23 AM, Steve Dower wrote: > "You consistently ignore Makefiles, .ini, etc." > > Do people really do open('makefile', 'rb'), extract filenames and try to > use them without ever decoding the file contents? > I'm sure they do :-( But this has always confused me - back in

Re: [Python-ideas] Let’s make escaping in f-literals impossible

2016-08-18 Thread Greg Ewing
Chris Angelico wrote: f"This is a number: {13:0\u07c4}" If I understand correctly, the proposal intends to make it easier for a syntax hightlighter to treat f"This is a number: {foo[42]:0\u07c4}" as f"This is a number: {foo[42] :0\u07c4}" ----

Re: [Python-ideas] Let’s make escaping in f-literals impossible

2016-08-18 Thread Steven D'Aprano
On Fri, Aug 19, 2016 at 02:17:29AM +1000, Chris Angelico wrote: > Format codes are just text, I really think that is wrong. They're more like executable code. https://www.python.org/dev/peps/pep-0498/#expression-evaluation "Just text" implies it is data: result = "function(arg)" like the

Re: [Python-ideas] Let’s make escaping in f-literals impossible

2016-08-18 Thread Steven D'Aprano
On Thu, Aug 18, 2016 at 12:26:26PM -0400, Random832 wrote: > There's a precedent. "$()" works this way in bash - call it a recursive > parser context or whatever you like, but the point is that "$(command > "argument with spaces")" works fine, and humans don't seem to have any > trouble with it.

Re: [Python-ideas] Let’s make escaping in f-literals impossible

2016-08-18 Thread Eric V. Smith
On 8/18/2016 3:15 PM, Terry Reedy wrote: On 8/18/2016 12:50 PM, Steve Dower wrote: I find it hard to not read f'{x.partition(' + ')[0]}' as string concatenation. and it will certainly be easier for highlighters to handle (assuming they're doing anything more complicated than simply displaying t

Re: [Python-ideas] Let’s make escaping in f-literals impossible

2016-08-18 Thread Chris Angelico
On Fri, Aug 19, 2016 at 10:18 AM, Steven D'Aprano wrote: > On Fri, Aug 19, 2016 at 02:17:29AM +1000, Chris Angelico wrote: > >> Format codes are just text, > > I really think that is wrong. They're more like executable code. > > https://www.python.org/dev/peps/pep-0498/#expression-evaluation > > "

Re: [Python-ideas] Let’s make escaping in f-literals impossible

2016-08-18 Thread Terry Reedy
On 8/18/2016 3:30 PM, Steve Dower wrote: On 18Aug2016 1215, Terry Reedy wrote: On 8/18/2016 12:50 PM, Steve Dower wrote: I don't think f'{x.partition('-')[0]}' is any less readable as a result of the reused quotes, Why are you reusing the single quote', which needs the escaping that you don'

Re: [Python-ideas] Let’s make escaping in f-literals impossible

2016-08-18 Thread Terry Reedy
On 8/18/2016 8:27 PM, Eric V. Smith wrote: On 8/18/2016 3:15 PM, Terry Reedy wrote: Without the escapes, existing f-unaware highlighters like IDLE's will be broken in that they will highlight the single f-string as two strings with differently highlighted content in the middle. For f'{x.parti

Re: [Python-ideas] Let’s make escaping in f-literals impossible

2016-08-18 Thread Terry Reedy
On 8/18/2016 8:18 PM, Steven D'Aprano wrote: On Fri, Aug 19, 2016 at 02:17:29AM +1000, Chris Angelico wrote: Format codes are just text, I really think that is wrong. They're more like executable code. https://www.python.org/dev/peps/pep-0498/#expression-evaluation I agree with you here.

Re: [Python-ideas] Fix default encodings on Windows

2016-08-18 Thread Terry Reedy
On 8/18/2016 1:39 PM, Steve Dower wrote: On 18Aug2016 1036, Terry Reedy wrote: On 8/18/2016 11:25 AM, Steve Dower wrote: In this case, we would announce in 3.6 that using bytes as paths on Windows is no longer deprecated, My understanding is the the first 2 fixes refine the deprecation rathe

Re: [Python-ideas] Let’s make escaping in f-literals impossible

2016-08-18 Thread Steven D'Aprano
On Thu, Aug 18, 2016 at 08:27:50PM -0400, Eric V. Smith wrote: > Right. Because all strings (regardless of prefixes) are first parsed as > strings, and then have their prefix "operator" applied, it's easy for a > parser to ignore any sting prefix character. Is that why raw strings can't end wit

Re: [Python-ideas] Let’s make escaping in f-literals impossible

2016-08-18 Thread אלעזר
בתאריך יום ו׳, 19 באוג' 2016, 08:29, מאת Terry Reedy ‏: > On 8/18/2016 8:18 PM, Steven D'Aprano wrote: > > On Fri, Aug 19, 2016 at 02:17:29AM +1000, Chris Angelico wrote: > > > >> Format codes are just text, > > > > I really think that is wrong. They're more like executable code. > > > > https://w

Re: [Python-ideas] Let’s make escaping in f-literals impossible

2016-08-18 Thread Serhiy Storchaka
On 19.08.16 08:07, Terry Reedy wrote: On 8/18/2016 3:30 PM, Steve Dower wrote: On 18Aug2016 1215, Terry Reedy wrote: On 8/18/2016 12:50 PM, Steve Dower wrote: I don't think f'{x.partition('-')[0]}' is any less readable as a result of the reused quotes, Why are you reusing the single quote',