Re: [Python-Dev] Bytes path support

[Paul Moore]

On 20 August 2014 07:53, Ben Finney  wrote:
> "Stephen J. Turnbull"  writes:
>
>> Marko Rauhamaa writes:
>>  > Unix programmers, though, shouldn't be shielded from bytes.
>>
>> Nobody's trying to do that.  But Python users should be shielded from
>> Unix programmers.
>
> +1 QotW

That quote is actually almost a "hidden extra Zen of Python" IMO :-)
Both parts of it.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Nick Coghlan]

On 20 Aug 2014 04:18, "Marko Rauhamaa"  wrote:
>
> Tres Seaver :
>
> > On 08/19/2014 01:43 PM, Ben Hoyt wrote:
> >> Fair enough. I don't quite understand, though -- why is the "official
> >> policy" to kill something that's "essential" on *nix?
> >
> > ISTM that the policy is based on a fantasy that "it looks like text to
> > me in my use cases, so therefore it must be text for everyone."
>
> What I like about Python is that it allows me to write native linux code
> without having to make portability compromises that plague, say, Java. I
> have select.epoll(). I have os.fork(). I have socket.TCP_CORK. The
> "textualization" of Python3 seems part of a conscious effort to make
> Python more Java-esque.

It's not just the JVM that says text and binary APIs should be separate -
it's every widely used operating system services layer except POSIX. The
POSIX way works well *if* everyone reliably encodes things as UTF-8 or
always uses encoding detection, but its failure mode is unfortunately
silent data corruption.

That said, there's a lot of Python software that is POSIX specific, where
bytes paths would be the least of the barriers to porting to Windows or
Jython. I'm personally +1 on consistently allowing binary paths in lower
level APIs, but disallowing them in higher level explicitly cross platform
abstractions like pathlib.

Regards,
Nick.

>
>
> Marko
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Antoine Pitrou]

Le 20/08/2014 07:08, Nick Coghlan a écrit :


It's not just the JVM that says text and binary APIs should be separate
- it's every widely used operating system services layer except POSIX.
The POSIX way works well *if* everyone reliably encodes things as UTF-8
or always uses encoding detection, but its failure mode is unfortunately
silent data corruption.

That said, there's a lot of Python software that is POSIX specific,
where bytes paths would be the least of the barriers to porting to
Windows or Jython. I'm personally +1 on consistently allowing binary
paths in lower level APIs, but disallowing them in higher level
explicitly cross platform abstractions like pathlib.


I fully agree with Nick's position here.

To elaborate specifically about pathlib, it doesn't handle bytes paths 
but allows you to generate them if desired:

https://docs.python.org/3/library/pathlib.html#operators

Adding full bytes support to pathlib would have added a lot of 
complication and fragility in the implementation *and* in the API (is it 
allowed to combine str and bytes paths? should they have separate 
classes?), for arguably little benefit.


I think if you want low-level features (such as unconverted bytes paths 
under POSIX), it is reasonable to point you to low-level APIs.


Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Brett Cannon]

On Wed Aug 20 2014 at 9:02:25 AM Antoine Pitrou  wrote:

> Le 20/08/2014 07:08, Nick Coghlan a écrit :
> >
> > It's not just the JVM that says text and binary APIs should be separate
> > - it's every widely used operating system services layer except POSIX.
> > The POSIX way works well *if* everyone reliably encodes things as UTF-8
> > or always uses encoding detection, but its failure mode is unfortunately
> > silent data corruption.
> >
> > That said, there's a lot of Python software that is POSIX specific,
> > where bytes paths would be the least of the barriers to porting to
> > Windows or Jython. I'm personally +1 on consistently allowing binary
> > paths in lower level APIs, but disallowing them in higher level
> > explicitly cross platform abstractions like pathlib.
>
> I fully agree with Nick's position here.
>
> To elaborate specifically about pathlib, it doesn't handle bytes paths
> but allows you to generate them if desired:
> https://docs.python.org/3/library/pathlib.html#operators
>
> Adding full bytes support to pathlib would have added a lot of
> complication and fragility in the implementation *and* in the API (is it
> allowed to combine str and bytes paths? should they have separate
> classes?), for arguably little benefit.
>
> I think if you want low-level features (such as unconverted bytes paths
> under POSIX), it is reasonable to point you to low-level APIs.
>

+1 from me as well. Allowing the low-level stuff work on bytes but keeping
high-level actually high-level keeps with our consenting adults policy as
well as making things possible, but not at the detriment of the common
case.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Terry Reedy]

On 8/20/2014 9:01 AM, Antoine Pitrou wrote:

Le 20/08/2014 07:08, Nick Coghlan a écrit :


It's not just the JVM that says text and binary APIs should be separate
- it's every widely used operating system services layer except POSIX.
The POSIX way works well *if* everyone reliably encodes things as UTF-8
or always uses encoding detection, but its failure mode is unfortunately
silent data corruption.

That said, there's a lot of Python software that is POSIX specific,
where bytes paths would be the least of the barriers to porting to
Windows or Jython. I'm personally +1 on consistently allowing binary
paths in lower level APIs, but disallowing them in higher level
explicitly cross platform abstractions like pathlib.


I fully agree with Nick's position here.

To elaborate specifically about pathlib, it doesn't handle bytes paths
but allows you to generate them if desired:
https://docs.python.org/3/library/pathlib.html#operators

Adding full bytes support to pathlib would have added a lot of
complication and fragility in the implementation *and* in the API (is it
allowed to combine str and bytes paths? should they have separate
classes?), for arguably little benefit.


I am glad you did not recreate the madness of pre 3.0 Python in that regard.


I think if you want low-level features (such as unconverted bytes paths
under POSIX), it is reasonable to point you to low-level APIs.


Do our docs somewhere explain the idea that files names are conceptually 
*names*, not arbitrary bytes; explain the concept of low-level versus 
high-level API' and point to the two types of APIs in Python?


--
Terry Jan Reedy


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Greg Ewing]

Antoine Pitrou wrote:
I think if you want low-level features (such as unconverted bytes paths 
under POSIX), it is reasonable to point you to low-level APIs.


The problem with scandir() in particular is that there is
currently *no* low-level API exposed that gives the same
functionality.

If scandir() is not to support bytes paths, I'd suggest
exposing the opendir() and readdir() system calls with
bytes path support.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Nick Coghlan]

On 21 Aug 2014 08:19, "Greg Ewing"  wrote:
>
> Antoine Pitrou wrote:
>>
>> I think if you want low-level features (such as unconverted bytes paths
under POSIX), it is reasonable to point you to low-level APIs.
>
>
> The problem with scandir() in particular is that there is
> currently *no* low-level API exposed that gives the same
> functionality.
>
> If scandir() is not to support bytes paths, I'd suggest
> exposing the opendir() and readdir() system calls with
> bytes path support.

scandir is low level (the entire os module is low level). In fact, aside
from pathlib, I'd consider pretty much every API we have that deals with
paths to be low level - that's a large part of the reason we needed pathlib!

Cheers,
Nick.

>
> --
> Greg
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Chris Barker]

>
>  but disallowing them in higher level
>> > explicitly cross platform abstractions like pathlib.
>>
>
I think the trick here is that posix-using folks claim that filenames are
just bytes, and indeed they can be passed around with a char*, so they seem
to be.

but you can't possible do anything other than pass them around if you
REALLY think they are just bytes.

So really, people treat them as
"bytes-in-some-arbitrary-encoding-where-at-least the-slash-character-(and
maybe a couple others)-is-ascii-compatible"

If you assume that, then you could write a pathlib that would work. And in
practice, I expect a lot of designed only for posix code works that way.
But of course, this gets ugly if you go to a platform where filenames are
not "bytes-in-some-arbitrary-encoding-where-at-least
the-slash-character-(and maybe a couple others)-is-ascii-compatible", like
windows.

I'm not sure if it's worth having a pathlib, etc. that uses this assumption
-- but it could help us all write code that actually works with this screwy
lack of specification.

 Antoine Pitrou wrote:

> To elaborate specifically about pathlib, it doesn't handle bytes paths
> but allows you to generate them if desired:
> https://docs.python.org/3/library/pathlib.html#operators


but that uses

os.fsencode:  Encode filename to the filesystem encoding

As I understand it, the whole problem with some posix systems is that there
is NO filesystem encoding -- i.e. you can't know for sure what encoding a
filename is in. So you need to be able to pass the bytes through as they
are.

(At least as I read Armin Ronacher's blog)

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Nick Coghlan]

On 21 Aug 2014 09:06, "Chris Barker"  wrote:

>
> As I understand it, the whole problem with some posix systems is that
there is NO filesystem encoding -- i.e. you can't know for sure what
encoding a filename is in. So you need to be able to pass the bytes through
as they are.
>
> (At least as I read Armin Ronacher's blog)

Armin lets his astonishment at the idea we'd expect Linux vendors to fix
their broken OS get the better of him at times - he thinks the
responsibility lies entirely with us to work around its quirks and
limitations :)

The "surrogateescape" codec is our main answer to the unreliability of the
POSIX encoding model - fsdecode will squirrel away arbitrary bytes in the
private use area, and then fsencode will restore them again later. That
works for the simple round tripping case, but we currently lack good
default tools for "cleaning" strings that may contain surrogates (or even
scanning a string to see if surrogates are present).

One idea I had along those lines is a surrogatereplace error handler (
http://bugs.python.org/issue22016) that emitted an ASCII question mark for
each smuggled byte, rather than propagating the encoding problem.

Cheers,
Nick.

>
> -Chris
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Ethan Furman]

On 08/20/2014 03:31 PM, Nick Coghlan wrote:


On 21 Aug 2014 08:19, "Greg Ewing" mailto:greg.ew...@canterbury.ac.nz>> wrote:


Antoine Pitrou wrote:


I think if you want low-level features (such as unconverted bytes paths under 
POSIX), it is reasonable to point you to low-level APIs.



The problem with scandir() in particular is that there is
currently *no* low-level API exposed that gives the same
functionality.

If scandir() is not to support bytes paths, I'd suggest
exposing the opendir() and readdir() system calls with
bytes path support.


scandir is low level (the entire os module is low level). In fact, aside from 
pathlib, I'd consider pretty much every
API we have that deals with paths to be low level - that's a large part of the 
reason we needed pathlib!


If scandir is low-level, and the low-level API's are the ones that should support bytes paths, then scandir should 
support bytes paths.


Is that what you meant to say?

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Nick Coghlan]

On 21 August 2014 09:33, Ethan Furman  wrote:
> On 08/20/2014 03:31 PM, Nick Coghlan wrote:
>> On 21 Aug 2014 08:19, "Greg Ewing" > > wrote:
>>>
>>>
>>> Antoine Pitrou wrote:


 I think if you want low-level features (such as unconverted bytes paths
 under POSIX), it is reasonable to point you to low-level APIs.
>>>
>>>
>>>
>>> The problem with scandir() in particular is that there is
>>> currently *no* low-level API exposed that gives the same
>>> functionality.
>>>
>>> If scandir() is not to support bytes paths, I'd suggest
>>> exposing the opendir() and readdir() system calls with
>>> bytes path support.
>>
>>
>> scandir is low level (the entire os module is low level). In fact, aside
>> from pathlib, I'd consider pretty much every
>> API we have that deals with paths to be low level - that's a large part of
>> the reason we needed pathlib!
>
>
> If scandir is low-level, and the low-level API's are the ones that should
> support bytes paths, then scandir should support bytes paths.
>
> Is that what you meant to say?

Yes. The discussions around PEP 471 *deferred* discussions of bytes
and file descriptor support to their own RFEs (not needing a PEP),
they didn't decide definitively not to support them. So Serhiy's
thread is entirely pertinent to that question.

Note that adding bytes support still *should not* hold up the initial
PEP 471 implementation - it should be done as a follow on RFE.

Cheers,
Nick.


-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Ethan Furman]

On 08/20/2014 05:15 PM, Nick Coghlan wrote:

On 21 August 2014 09:33, Ethan Furman  wrote:

On 08/20/2014 03:31 PM, Nick Coghlan wrote:


scandir is low level (the entire os module is low level). In fact, aside
from pathlib, I'd consider pretty much every
API we have that deals with paths to be low level - that's a large part of
the reason we needed pathlib!


If scandir is low-level, and the low-level API's are the ones that should
support bytes paths, then scandir should support bytes paths.

Is that what you meant to say?


Yes. The discussions around PEP 471 *deferred* discussions of bytes
and file descriptor support to their own RFEs (not needing a PEP),
they didn't decide definitively not to support them. So Serhiy's
thread is entirely pertinent to that question.


Thanks for clearing that up.  I hate feeling confused.  ;)



Note that adding bytes support still *should not* hold up the initial
PEP 471 implementation - it should be done as a follow on RFE.


Agreed.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 4000 to explicitly declare we won't be doing a Py3k style compatibility break again?

[Joseph Martinot-Lagarde]

Le 18/08/2014 03:02, Guido van Rossum a écrit :

On Sun, Aug 17, 2014 at 6:29 AM, Barry Warsaw mailto:ba...@python.org>> wrote:

On Aug 16, 2014, at 07:43 PM, Guido van Rossum wrote:

 >(Don't understand this to mean that we should never deprecate things.
 >Deprecations will happen, they are necessary for the evolution of any
 >programming language. But they won't ever hurt in the way that
Python 3
 >hurt.)

It would be useful to explore what causes the most pain in the 2->3
transition?  IMHO, it's not the deprecations or changes such as print ->
print().  It's the bytes/str split - a fundamental change to core
and common
data types.  The question then is whether you foresee any similar
looming
pervasive change? [*]


I'm unsure about what's the single biggest pain moving to Python 3. In
the past I would have said that it's for sure the bytes/str split (which
both the biggest pain and the biggest payoff).


The pain was even bigger because in addition to the change in underlying 
types, the names of the types were not compatible between the python 
versions. I often try to write compatible code between python2 and 3, 
and I can't use "str" because it has not the same meaning in both 
versions, I can not use "unicode" because it disappeared in python3, and 
I can't use "byte" because it doesn't exist in python2. Add __str__ and 
__unicode__ to the mix and then you get the real pain.


Actually "str" is still usefull in the cases where a library is 
byte-only in python2 and unicode-only in python3 (hello, 
locale.setlocale()).


Joseph

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Ben Hoyt]

>> If scandir is low-level, and the low-level API's are the ones that should
>> support bytes paths, then scandir should support bytes paths.
>>
>> Is that what you meant to say?
>
> Yes. The discussions around PEP 471 *deferred* discussions of bytes
> and file descriptor support to their own RFEs (not needing a PEP),
> they didn't decide definitively not to support them. So Serhiy's
> thread is entirely pertinent to that question.
>
> Note that adding bytes support still *should not* hold up the initial
> PEP 471 implementation - it should be done as a follow on RFE.

I agree with this (that scandir is low level and should support
bytes). As it happens, I'm implementing bytes support as well -- what
with the path_t support in posixmodule.c and the listdir
implementation to go on, it's not really any harder. So I think we'll
have it right off the bat.

BTW, the Windows implementation of PEP 471 is basically done, and the
POSIX implementation is written but not working yet. And then there's
tests and docs.

-Ben
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Stephen J. Turnbull]

Nick Coghlan writes:

 > One idea I had along those lines is a surrogatereplace error handler (
 > http://bugs.python.org/issue22016) that emitted an ASCII question mark for
 > each smuggled byte, rather than propagating the encoding problem.

Please, don't.

"Smuggled bytes" are not independent events.  They tend to be
correlated *within* file names, and this handler would generate names
whose human semantics get lost (and there *are* human semantics,
otherwise the name would be str(some_counter)).  They tend to be
correlated across file names, and this handler will generate multiple
files with the same munged name (and again, the differentiating human
semantics get lost).

If you don't know the semantics of the intended file names, you can't
generate good replacement names.  This has to be an application-level
function, and often requires user intervention to get good names.

If you want to provide helper functions that applications can use to
clean names explicitly, that might be OK.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Cameron Simpson]

On 20Aug2014 16:04, Chris Barker - NOAA Federal  wrote:

 but disallowing them in higher level

> explicitly cross platform abstractions like pathlib.



I think the trick here is that posix-using folks claim that filenames are
just bytes, and indeed they can be passed around with a char*, so they seem
to be.

but you can't possible do anything other than pass them around if you
REALLY think they are just bytes.

So really, people treat them as
"bytes-in-some-arbitrary-encoding-where-at-least the-slash-character-(and
maybe a couple others)-is-ascii-compatible"


As someone who fought long and hard in the surrogate-escape listdir() wars, and 
was won over once the scheme was thoroughly explained to me, I take issue with 
these assertions: they are bogus or misleading.


Firstly, POSIX filenames _are_ just byte strings. The only forbidden character 
is the NUL byte, which terminates a C string, and the only special character is 
the slash, which separates pathanme components.


Second, a bare low level program cannot do _much_ more than pass them around.  
It certainly can do things like compute their basename, or other path related 
operations.


The "bytes in some arbitrary encoding where at least the slash character (and
maybe a couple others) is ascii compatible" notion is completely bogus. There's 
only one special byte, the slash (code 47). There's no OS-level need that it or 
anything else be ASCII compatible. I think characterisations such as the one 
quoted are activately misleading.


The way you get UTF-8 (or some other encoding, fortunately getting less and 
less common) is by convention: you decide in your environment to work in some 
encoding (say utf-8) via the locale variables, and all your user-facing text 
gets used in UTF-8 encoding form when turned into bytes for the filename calls 
because your text<->bytes methods say to do so.


I think we'd all agree it is nice to have a system where filenames are all 
Unicode, but since POSIX/UNIX predates it by decades it is a bit late to ignore 
the reality for such systems. I certainly think the Window-side Babel of code 
pages and multiple code systems is far far worse. (Disclaimer: not a Windows 
programmer, just based on hearing them complain.)


I'm +1000 on systems where the filesystem enforces Unicode (eg Plan 9 or Mac 
OSX, which forces a specific UTF-8 encoding in the bytes POSIX APIs - the 
underlying filesystems reject invalid byte sequences).


[...]

Antoine Pitrou wrote:

To elaborate specifically about pathlib, it doesn't handle bytes paths
but allows you to generate them if desired:
https://docs.python.org/3/library/pathlib.html#operators


but that uses
os.fsencode:  Encode filename to the filesystem encoding

As I understand it, the whole problem with some posix systems is that there
is NO filesystem encoding -- i.e. you can't know for sure what encoding a
filename is in. So you need to be able to pass the bytes through as they
are.


Yes and no. I made that argument too.

There's no _external_ "filesystem encoding" in the sense of something recorded 
in the filesystem that anyone can inspect. But there is the expressed locale 
settings, available at runtime to any program that cares to pay attention. It 
is a workable situation.


Oh, and I reject Nick's characterisation of POSIX as "broken". It's perfectly 
internally consistent. It just doesn't match what he wants. (Indeed, what I 
want, and I'm a long time UNIX fanboy.)


Cheers,
Cameron Simpson 

God is real, unless declared integer.   - Johan Montald, jo...@ingres.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com