Re: [Python-Dev] My first post here ~ do you need more Python core developers on Windows?

[Burkhard Meier]

Thank you for the warm welcome and the links. I will definitely check them
out.

Burkhard

On Mon, Apr 18, 2016 at 1:16 AM, Victor Stinner 
wrote:

> 2016-04-18 7:23 GMT+02:00 Burkhard Meier :
> > My name is Burkhard Meier and I wrote the "Python GUI Programming
> Cookbook"
> > published by Packt.
> >
> > It is available on Amazon and PacktPub.com.
>
> Welcome!
>
> > Maybe I can become more involved in the Python community as a Python
> > developer on Windows .
>
> You can use the Developer Guide to start:
> https://docs.python.org/devguide/
>
> See also the Python menthors to get help on a dedicated and private
> mailing list:
> http://pythonmentors.com/
>
> Sadly yes, we have many open issues specific to Windows. I'm trying to
> sometimes give time to fix some of them, but I'm less interested than
> in open source operating systems ;-)
>
> Victor
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Wes Turner]

On Apr 18, 2016 3:19 PM, "Ethan Furman"  wrote:
>
> On 04/18/2016 12:54 PM, Wes Turner wrote:
>
>> Don't we *have* to always support bytes because other programs can
>> create filenames containing bytes?
>
>
> Yes, but not every function has to support bytes.

Because there's no function overloading in Python, we then must have
explicit typing conditionals.

I haven't the time to dig through and compare this with the other fine
solutions presented; is there a reason that a proxy/facade PrimitiveType
wouldn't solve for this?

class TextThing:
  __init__(self, data):
  self.data = data
  self.type_ = type(data)
   __getattr__(self, key):
   return getattr(self.data, key)


>
>
> --
> ~Ethan~
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-ideas] pep 7 line break suggestion differs from pep 8

[Guido van Rossum]

[ideas to bcc]

I'm not as excited about this as I am about the PEP 8 change.

PEP 8 affects most Python programmers.

But PEP 7 is really just for CPython and its extensions, and I don't think
it has found anything like as widespread a following as PEP 8.

I worry that if we change this in PEP 7 we'll just see either massing
inconsistent code or endless diffs that do nothing but change the
formatting (and occasionally introduce a bug).

And I don't think it would do as much good -- reading and understanding C
code is primarily a matter of knowing the language, and the audience is
much more heavily skewed towards experts.

IOW, -1.

On Mon, Apr 18, 2016 at 1:27 PM, Chris Angelico  wrote:

> On Tue, Apr 19, 2016 at 5:33 AM, Joseph Jevnik  wrote:
> > I saw that there was recently a change to pep 8 to suggest adding a line
> > break before a binary operator. Pep 7 suggests the opposite:
> >
> >> When you break a long expression at a binary operator, the operator goes
> >> at the end of the previous line, e.g.:
> >
> >> if (type->tp_dictoffset != 0 && base->tp_dictoffset == 0 &&
> >> type->tp_dictoffset == b_size &&
> >> (size_t)t_size == b_size + sizeof(PyObject *))
> >> return 0; /* "Forgive" adding a __dict__ only */
> >
> > I imagine that some of the reasons for making the change in pep 8 for
> > readability reasons will also
> > translate to C; maybe pep 7 should also be updated.
>
> I would agree with this. Passing it directly to python-dev as that's
> where the key decision makers are.
>
> ChrisA
> ___
> Python-ideas mailing list
> python-id...@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Ethan Furman]

On 04/18/2016 12:26 PM, Stephen J. Turnbull wrote:


I haven't looked at Antipathy, but I would guess from Ethan's
promotion of bytes paths and concern with efficiency that "bytes
antipaths" do *not* "go through" str to get to bytes, they already are
bytes (in the sense of class inheritance).


Couple points:

- Correct: if you create an antipathy.Path with bytes, you get a
  bytes path (bPath); if you create an antipathy.Path with str
  you get a str path (uPath)

- if you mix a bPath with a uPath, or bytes with a uPath, or str with
  a bPath -- an exception is raised (conversions are *not* implicit (on
  3.0, at least -- on 2.x you can activate that behavior if you want it)

- my concern with supporting bytes is primarily for the sake of the
  stdlib, and secondarily for anyone who needs to work with bytes; it
  really has no effect on my library (since antipathy uses subclasses
  of bytes/str)

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Ethan Furman]

On 04/18/2016 02:58 PM, Koos Zevenhoven wrote:


It's a matter of documentation whether it "supports" bytes
or not. In fact, that function (assuming the name os.fspath) could now
even be documented to support this:

 patharg = os.fspath(patharg, output_types = (str, pathlib.PurePath))  # :-)


While the os.fspath() function could be abused in such a way, we 
certainly wouldn't advertise it.  (Leave that to StackOverflow. ;)


--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Koos Zevenhoven]

On Mon, Apr 18, 2016 at 5:03 PM, Ethan Furman  wrote:
> On 04/18/2016 12:41 AM, Nick Coghlan wrote:
>
>> Given the variant you [Koos] suggested, what if we defined the API
>> semantics
>> like this:
>>
>>  # Offer the simplest possible API as the public vesion
>>  def fspath(pathlike) -> str:
>>  return os._raw_fspath(pathlike)
>>
>>  # Expose the complexity in the "private" variant
>>  def _raw_fspath(pathlike, *, output_types = (str,)) -> (str, bytes):
>>  # Short-circuit for instances of the output type
>>  if isinstance(pathlike, output_types):
>>  return pathlike
>>  # We'd have a tidier error message here for non-path objects
>>  result = pathlike.__fspath__()
>>  if not isinstance(result, output_types):
>>  raise TypeError("argument is not and does not provide an
>> acceptable pathname")
>>  return result
>
> My initial reaction was that this was overly complex, but after thinking
> about it a couple days I /really/ like it.  It has a reasonable default for
> the 99% real-world use-case, while still allowing for custom and exact
> tailoring (for the 99% stdlib use-case ;) .
>

While it does seem we finally might be nearly there :), this still
seems to need some further discussion.

As described in that long post of mine, I suppose some third-party
code may need the variations (A-C), while it seems that in the stdlib,
most places need (str, bytes), i.e. (A), except in pathlib, which
needs (str,), i.e. (B). I'm not sure what I think about making the
variations private, even if "hiding" the bytes version is, as I said,
an important role of the public function.

Except for that type hint, there is *nothing* in the function that
might mislead the user to think bytes paths are something important in
Python 3. It's a matter of documentation whether it "supports" bytes
or not. In fact, that function (assuming the name os.fspath) could now
even be documented to support this:

patharg = os.fspath(patharg, output_types = (str, pathlib.PurePath))  # :-)

So are we still going to end up with two functions or can we deal with one?
What should the typehint be? Something new in typing.py? How about
FSPath[...] as follows:

FSPath[bytes]  # bytes-based pathlike, including bytes
FSPath[str]   # str-based pathlike, including str

pathstring = typing.TypeVar('pathstring', str, bytes)  # could be
extended with PurePath or some path ABC

So the above variation might become:

def fspathname(pathlike: FSPath[pathstring],
   *, output_types: tuple = (str,)) -> pathstring:
# Short-circuit for instances of the output type
if isinstance(pathlike, output_types):
return pathlike
# We'd have a tidier error message here for non-path objects
result = pathlike.__fspath__()
if not isinstance(result, output_types):
raise TypeError("valid output type not provided via __fspath__")
return result

And similar type hints would apply to os.path functions. For instance,
os.path.dirname:

def dirname(p: FSPath[pathstring]) -> pathstring:
...

This would say pathstring all over and not give anyone any ideas about
bytes, unless they know what they're doing.

Complicated? Yes, typing is. But I think we will need this kind of
hints for os.path functions anyway.

-Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Brett Cannon]

On Mon, 18 Apr 2016 at 12:26 Stephen J. Turnbull  wrote:

> Brett Cannon writes:
>
>  > If we continue with the "str is an encoding of file paths",
>
> It's not.  It's a representation, but not an encoding.  In Python 3,
> encoding means a representation of a character string using bytes.
> It's using "encoding" generically for "representation" that makes your
> head hurt.
>

Well, it makes *your* head hurt; for me it helped clarify some things. :)


>
>  > you can then build from "bytes is an encoding of str" to get a
>  > pyramid of file path encodings: Path -> str -> bytes. I don't think
>  > this is in any way a controversial view.
>
> Perhaps not.  But it's not particularly useful. ;-)  Here's the
> pyramid I think about:
>
>  Path
> /\
>/  \
>   VV
> str <-> bytes
>
> That is, str and bytes are interchangeable *without* any knowledge of
> paths, which are on a higher level of complexity and abstraction.
> Although in pathlib, there's an assumption that paths are serialized
> to str which is (implicitly) serialized to bytes when talking to the
> OS, this is not necessarily true for other structured path classes, in
> particular it is not true for DirEntry (which is a "enhanced
> degenerate" path containing only one path segment but also other
> useful information about the filesystem object addressed)
>
> I haven't looked at Antipathy, but I would guess from Ethan's
> promotion of bytes paths and concern with efficiency that "bytes
> antipaths" do *not* "go through" str to get to bytes, they already are
> bytes (in the sense of class inheritance).
>
>  > But that's when I realized that adding __fspath__ support to
> os.fsdecode()
>  > and os.fsencode(), they become more coercion functions rather than
>  > encoding/decoding functions. It also means that os.fspath() has a place
>  > when you want to say "I only want to encode a file path to str" and
> avoid
>  > the decode bit that os.fsdecode() would do
>
> I don't understand what you're trying to say here.  fsdecode currently
> does not promise to decode anything, because it's polymorphic,
> accepting str and bytes.  fsdecode and fsencode already *are* coercion
> functions.
>

And they will continue to be coercion functions. My point is that since
they coerce there is no way to use them in a way to dictate that you don't
want any str/bytes encoding/decoding to occur without checking the
arguments going into the function (i.e. "no guessing about encodings,
please"). By providing os.fspath() I can say that I do not, under any
circumstances, want someone to guess at the encoding some bytes path is
under to get me a string and instead I want to start and end entirely in a
world of strings. IOW os.fspath() lets me work in such a way that the
instant bytes are introduced into my code for file paths it triggers a
TypeError.


>
> It's this kind of semantic confusion and broken nomenclature that is
> *why* I dislike these polymorphic functions and objects so much.  It
> is impossible to reason correctly about them.  We're stuck with
> invoking "practicality" and muddling through.  And the names mislead
> even experienced Pythonistas.
>

Yep, we are stuck with the names unless you want to propose a new name and
deprecate the old one.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-ideas] pep 7 line break suggestion differs from pep 8

[Chris Angelico]

On Tue, Apr 19, 2016 at 5:33 AM, Joseph Jevnik  wrote:
> I saw that there was recently a change to pep 8 to suggest adding a line
> break before a binary operator. Pep 7 suggests the opposite:
>
>> When you break a long expression at a binary operator, the operator goes
>> at the end of the previous line, e.g.:
>
>> if (type->tp_dictoffset != 0 && base->tp_dictoffset == 0 &&
>> type->tp_dictoffset == b_size &&
>> (size_t)t_size == b_size + sizeof(PyObject *))
>> return 0; /* "Forgive" adding a __dict__ only */
>
> I imagine that some of the reasons for making the change in pep 8 for
> readability reasons will also
> translate to C; maybe pep 7 should also be updated.

I would agree with this. Passing it directly to python-dev as that's
where the key decision makers are.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Question][Asyncio] Process + Threads + asyncio... has sense?

[cr0hn]

Oks. Thank you very much.


---
*Daniel García (cr0hn)*
Security researcher and ethical hacker

*Personal site*: http://cr0hn.com
*Linkedin*: https://www.linkedin.com/in/garciagarciadaniel
*Company*: http://abirtone.com
*Twitter*: @ggdaniel 

El día 18 de abril de 2016 a las 18:40:14, Guido van Rossum (
gu...@python.org) escrito:

> A better place for this question would be the tulip Google group:
> https://groups.google.com/forum/#!forum/python-tulip
>
> On Mon, Apr 18, 2016 at 3:05 AM, cr0hn  wrote:
>
>> Hi all,
>>
>> It's the first time I write in this list. Sorry if it's not the best
>> place for this question.
>>
>> After I read the Asyncio's documentation, PEPs, Guido/Jesse/David Beazley
>> articles/talks, etc, I developed a PoC library that mixes: Process +
>> Threads + Asyncio Tasks, doing an scheme like this diagram:
>>
>>  main -> Process 1 -> Thread 1.1 -> Task 1.1.1
>>   -> Task 1.1.2
>>   -> Task 1.1.3
>>
>> -> Thread 1.2
>>  -> Task 1.2.1
>>  -> Task 1.2.2
>>  -> Task 1.2.3
>>
>>  Process 2 -> Thread 2.1 -> Task 2.1.1
>>  -> Task 2.1.2
>>  -> Task 2.1.3
>>
>>   -> Thread 2.2
>>  -> Task 2.2.1
>>  -> Task 2.2.2
>>  -> Task 2.2.3
>>
>> In my local tests, this approach appear to improve (and simplify) the
>> concurrency/parallelism for some tasks but, before release the library at
>> github, I don't know if my aproach is wrong and I would appreciate your
>> opinion.
>>
>> Thank you very much for your time.
>>
>> Regards!
>>
>> --
>> Daniel García a.k.a. cr0hn - Security researcher and pentester
>> @ggdaniel
>> http://www.cr0hn.com/me/
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Ethan Furman]

On 04/18/2016 12:54 PM, Wes Turner wrote:


Don't we *have* to always support bytes because other programs can
create filenames containing bytes?


Yes, but not every function has to support bytes.

--
~Ethan~

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Wes Turner]

On Apr 18, 2016 2:50 PM, "Ethan Furman"  wrote:
>
> On 04/18/2016 12:25 PM, Stephen J. Turnbull wrote:
>
>> Koos Zevenhoven writes:
>
>
>>> After all, we want something that's *almost* exclusively str.
>>
>>
>> But we don't want that, AFAICT.  Some clearly want this API to be
>> unbiased against bytes in the same way the os APIs are unbiased[2],
>> because that's what we've got in the current proposal.
>
>
> Are we reading the same thread?  For my last several replies I am very
biased against bytes (and I know I'm not the only one).
>
> Just not so biased that I'm unwilling to let clients say, "No, I'm really
okay with getting bytes back".
>
> I really like Koos' ideas because they allow the client to say:
>
> - I only want str
> - I only want bytes
> - I'm okay with either
>
> If the client says "I'm okay with either" then I fully expect the client
to have code to properly handle str vs bytes after the fspath (or whatever
it's called) call.

Don't we *have* to always support bytes because other programs can create
filenames containing bytes?

>
> --
> ~Ethan~
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Ethan Furman]

On 04/18/2016 12:25 PM, Stephen J. Turnbull wrote:

Koos Zevenhoven writes:



After all, we want something that's *almost* exclusively str.


But we don't want that, AFAICT.  Some clearly want this API to be
unbiased against bytes in the same way the os APIs are unbiased[2],
because that's what we've got in the current proposal.


Are we reading the same thread?  For my last several replies I am very 
biased against bytes (and I know I'm not the only one).


Just not so biased that I'm unwilling to let clients say, "No, I'm 
really okay with getting bytes back".


I really like Koos' ideas because they allow the client to say:

- I only want str
- I only want bytes
- I'm okay with either

If the client says "I'm okay with either" then I fully expect the client 
to have code to properly handle str vs bytes after the fspath (or 
whatever it's called) call.


--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Random832]

On Mon, Apr 18, 2016, at 15:26, Stephen J. Turnbull wrote:
> in
> particular it is not true for DirEntry (which is a "enhanced
> degenerate" path containing only one path segment but also other
> useful information abot the filesystem object addressed)

DirEntry contains multiple path segments - it has the name, and the
directory path that was passed into scandir.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Stephen J. Turnbull]

Brett Cannon writes:

 > If we continue with the "str is an encoding of file paths",

It's not.  It's a representation, but not an encoding.  In Python 3,
encoding means a representation of a character string using bytes.
It's using "encoding" generically for "representation" that makes your
head hurt.

 > you can then build from "bytes is an encoding of str" to get a
 > pyramid of file path encodings: Path -> str -> bytes. I don't think
 > this is in any way a controversial view.

Perhaps not.  But it's not particularly useful. ;-)  Here's the
pyramid I think about:

 Path
/\
   /  \
  VV
str <-> bytes

That is, str and bytes are interchangeable *without* any knowledge of
paths, which are on a higher level of complexity and abstraction.
Although in pathlib, there's an assumption that paths are serialized
to str which is (implicitly) serialized to bytes when talking to the
OS, this is not necessarily true for other structured path classes, in
particular it is not true for DirEntry (which is a "enhanced
degenerate" path containing only one path segment but also other
useful information abot the filesystem object addressed)

I haven't looked at Antipathy, but I would guess from Ethan's
promotion of bytes paths and concern with efficiency that "bytes
antipaths" do *not* "go through" str to get to bytes, they already are
bytes (in the sense of class inheritance).

 > But that's when I realized that adding __fspath__ support to os.fsdecode()
 > and os.fsencode(), they become more coercion functions rather than
 > encoding/decoding functions. It also means that os.fspath() has a place
 > when you want to say "I only want to encode a file path to str" and avoid
 > the decode bit that os.fsdecode() would do

I don't understand what you're trying to say here.  fsdecode currently
does not promise to decode anything, because it's polymorphic,
accepting str and bytes.  fsdecode and fsencode already *are* coercion
functions.

It's this kind of semantic confusion and broken nomenclature that is
*why* I dislike these polymorphic functions and objects so much.  It
is impossible to reason correctly about them.  We're stuck with
invoking "practicality" and muddling through.  And the names mislead
even experienced Pythonistas.

Steve

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Stephen J. Turnbull]

I don't disagree with the basic analysis, but there are a number of
issues with motivational statements.

Koos Zevenhoven writes:

 > (B) "str-based only"
 > *Accept*: str, provided via __fspath__ as well as plain str.
 > *Return*: str.
 > *Audience*: relatively low-level code that works exclusively with str
 > paths but accepts specialized path objects as input.

Why "low-level"?  All code that stores paths persistently is likely to
store them in text files or database strings or the like, rather than
as Path (read: specialized path objects, not necessarily
pathlib.Path).  But if there is any low-level manipulation of the
paths to be done before storing, it would be done as Path.  Thus
high-level code might also want to accept Path transparently.

 > (C) "bytes-based only"
 > *Accept*: bytes, provided via __fspath__ as well as plain bytes.
 > *Return*: bytes.
 > *Audience*: low-level code that explicitly deals with paths as bytes
 > (probably to deal with undefined/ill-defined encodings).

No, if it's to deal with encoding issues, we wouldn't accept this.
PEP 383 eliminates that concern.  We accept bytes to support people
who are representing paths with bytes because they think that it's a
good idea and that encoding doesn't matter in their application.

 > (D) "coerce to str"
 > *Accept*: str and bytes, provided via __fspath__ as well as plain str
 > and bytes instances.
 > *Return*: str (coerced / decoded if needed).
 > *Audience*: code that deals explicitly with str but wants to 'try'
 > supporting bytes-based path inputs too via implicit decoding (even if
 > it may result in surrogate escapes, which one cannot for instance
 > print(...).)

No.  As Nick points out with respect to fsencode/fsdecode, it's not
a question of supporting known bytes via implicit decoding (that's
what __fspath__ does for the types that support it), but rather
of supporting ambiguity.  Best practice is to convert explicitly at
the boundary, because it's too likely that data with unexpected type
is just the wrong data.  

Printing surrogates can be done with errors=backslashreplace, and if
you're using fsdecode, you probably should use that, namereplace, or
xmlcharrefreplace.

 > (E) "coerce to bytes"
 > *Accept*: str and bytes, provided via __fspath__ as well as plain str
 > and bytes instances.
 > *Return*: bytes (coerced / encoded if needed).
 > *Audience*: low-level code that explicitly deals with bytes paths but
 > wants to accept str-based path inputs too via implicit encoding.

Again, it's a question of ambiguity, or perhaps sloppy programming
(eg, using str literals for paths in a bytes-oriented program).

Use cases D and E are basically "guessing when faced with ambiguity",
and fsencode and fsdecode are code smells because (as Nick claims)
they almost always conceal a situation where you don't know whether
you've got bytes or str (and it's way too much work to find out by
tracing them back to where they came from).

 > It seems to me we now "all" agree that __fspath__ should allow
 > str+bytes polymorphism.

I don't agree that we *should* allow polymorphism, because (purity)
paths are in the text domain[1] and (practicality) I don't believe that
use of os.fspath will be restricted to "low-level boundary code".  I
would be perfectly happy telling bytes users that the idiom is not
"os.fspath(maybe_direntry, allow_types=(bytes,))", but rather
"os.fsencode(os.fspath(maybe_direntry))", so that code in the text
domain can safely use os.fspath(maybe_direntry) without worrying that
it will raise because maybe_direntry.__fspath__() returns bytes.

This would allow pathlib.Path to handle arguments providing __fspath__
transparently.  With the current proposal, it would need to rule out
bytes before invoking os.fspath, or handle the exception, or leave the
exception to its caller.  None of these options are pleasant.

Unfortunately, as Nick points out, defining __fspath__ to return str
is very unpleasant because bytes applications will now have to guard
*everything* that might provide __fspath__ with that incantation
before passing to open and other APIs that store the path on the
object returned.  So we don't really have a choice about polymorphism
if we want to support both __fspath__ and bytes paths.

 > After all, we want something that's *almost* exclusively str.

But we don't want that, AFAICT.  Some clearly want this API to be
unbiased against bytes in the same way the os APIs are unbiased[2],
because that's what we've got in the current proposal.  Further, due
to the existing ambiguity in fsencode and fsdecode, we're extending
the field of ambiguity where bytes and str can mix indiscriminately.

If we are serious about "*almost* exclusively str" we should accept
that "exclusively str" is a very good approximation and much easier to
use correctly, and regretfully postpone inclusion of DirEntry in this
protocol to the future.  But that's not on the table, is it?


Footnotes: 
[1]  Representation on disk as (basically 

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Brett Cannon]

On Sun, 17 Apr 2016 at 06:59 Koos Zevenhoven  wrote:

> On Sun, Apr 17, 2016 at 11:03 AM, Stephen J. Turnbull
>  wrote:
> > Nick Coghlan writes:
> >
> >  > str and bytes aren't going to implement __fspath__ (since they're
> >  > only *sometimes* path objects), so asking people to call the
> >  > protocol method directly for any purpose would be a pain.
> >
> > It *should* be a pain.  People who need bytes should call fsencode,
> > people who need str should call fsdecode, and Ethan's antipathy checks
> > for bytes and str, then calls __fspath__ if needed.  Who's left?  Just
> > the bartender and the janitor, last call was hours ago.  OK, maybe
> > there are enough clients to make it worthwhile to provide the utility,
> > but it should be clearly marked as "double opt-in, for experts only
> > (consenting adults must show proof of insurance)".
>
> My doubts, expressed several times in these threads, about the need
> for a *public* os.fspath function to complement the __fspath__
> protocol, are now perhaps gone. I'll explain why (and how). The
> reasons for my doubts were that
>
> (1) The audience outside the stdlib for such a function should be
> small, because it is preferred to either use existing tools in
> os.path.* or pathlib (or similar) for manipulating paths.
>
> (2) There are just too many different possible versions of this
> function: rejecting str, rejecting bytes, coercion to str, coercion to
> bytes, and accepting both str and bytes. That's a total of 5 different
> cases. People also used to talk about versions that would not allow
> passing through objects that are already bytes or str. That would make
> it a total of 10 different versions!
> (in principle, there could be even more, but let's not go there :-).
> In other words, this argument was that it is probably best to
> implement whatever flavor is needed for the context, perhaps based on
> documented recipes.
>
>
> Regarding (2), we can first rule out half of the 10 cases---the ones
> that reject plain instances of bytes and/or str---because they would
> not be very useful as all the isinstance/hasattr checking etc. would
> be left to the caller. And here are the remaining five, explained
> based on what they accept as argument, what they return, and where
> they would be used:
>
> (A) "polymorphic"
> *Accept*: str and bytes, provided via __fspath__ as well as plain str
> and bytes instances.
> *Return*: str/bytes depending on input.
> *Audience*: the stdlib, including os.path.things, os.things,
> shutil.things, open, ... (some functions would need a C version).
> There may even be a small audience outside the stdlib.
>
> (B) "str-based only"
> *Accept*: str, provided via __fspath__ as well as plain str.
> *Return*: str.
> *Audience*: relatively low-level code that works exclusively with str
> paths but accepts specialized path objects as input.
>
> (C) "bytes-based only"
> *Accept*: bytes, provided via __fspath__ as well as plain bytes.
> *Return*: bytes.
> *Audience*: low-level code that explicitly deals with paths as bytes
> (probably to deal with undefined/ill-defined encodings).
>
> (D) "coerce to str"
> *Accept*: str and bytes, provided via __fspath__ as well as plain str
> and bytes instances.
> *Return*: str (coerced / decoded if needed).
> *Audience*: code that deals explicitly with str but wants to 'try'
> supporting bytes-based path inputs too via implicit decoding (even if
> it may result in surrogate escapes, which one cannot for instance
> print(...).)
>
> (E) "coerce to bytes"
> *Accept*: str and bytes, provided via __fspath__ as well as plain str
> and bytes instances.
> *Return*: bytes (coerced / encoded if needed).
> *Audience*: low-level code that explicitly deals with bytes paths but
> wants to accept str-based path inputs too via implicit encoding.
>
>
> Even if all options (A-E) probably have small audiences (compared to
> e.g. os.path.*), some of them have larger audiences than others. But
> all of them have at least *some* reasonable audience (as desribed
> above).
>
> Recently (well, a few days ago, but 'recently', considering the scale
> of these discussions anyway ;-), Nick pointed out something I hadn't
> realized---os.fsencode and os.fsdecode actually already implement
> coercion to bytes and str, respectively. With those two functions made
> compatible with the __fspath__ protocol [using (A) above], they would
> in fact *be* (D) and (E), respectively.
>
> Now, we only have options (A-C) left. They could all be implemented
> roughly as follows:
>
> def fspath(pathlike, *, output_types = (str,)):
>   if hasattr(pathlike, '__fspath__'):
> ret = pathlike.__fspath__()  # or pathlike.__fspath__ if it's not a
> method
>   else:
> ret = pathlike
>   if not isinstance(ret, output_types):
> raise TypeError("argument is not and does not provide an
> acceptable pathname")
>   return ret
>
> With an implementation like the above, (A) would correspond to
> output_types = 

Re: [Python-Dev] [Question][Asyncio] Process + Threads + asyncio... has sense?

[Guido van Rossum]

A better place for this question would be the tulip Google group:
https://groups.google.com/forum/#!forum/python-tulip

On Mon, Apr 18, 2016 at 3:05 AM, cr0hn  wrote:

> Hi all,
>
> It's the first time I write in this list. Sorry if it's not the best place
> for this question.
>
> After I read the Asyncio's documentation, PEPs, Guido/Jesse/David Beazley
> articles/talks, etc, I developed a PoC library that mixes: Process +
> Threads + Asyncio Tasks, doing an scheme like this diagram:
>
>  main -> Process 1 -> Thread 1.1 -> Task 1.1.1
>   -> Task 1.1.2
>   -> Task 1.1.3
>
> -> Thread 1.2
>  -> Task 1.2.1
>  -> Task 1.2.2
>  -> Task 1.2.3
>
>  Process 2 -> Thread 2.1 -> Task 2.1.1
>  -> Task 2.1.2
>  -> Task 2.1.3
>
>   -> Thread 2.2
>  -> Task 2.2.1
>  -> Task 2.2.2
>  -> Task 2.2.3
>
> In my local tests, this approach appear to improve (and simplify) the
> concurrency/parallelism for some tasks but, before release the library at
> github, I don't know if my aproach is wrong and I would appreciate your
> opinion.
>
> Thank you very much for your time.
>
> Regards!
>
> --
> Daniel García a.k.a. cr0hn - Security researcher and pentester
> @ggdaniel
> http://www.cr0hn.com/me/
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] [Question][Asyncio] Process + Threads + asyncio... has sense?

[cr0hn]

Hi all,

It's the first time I write in this list. Sorry if it's not the best place
for this question.

After I read the Asyncio's documentation, PEPs, Guido/Jesse/David Beazley
articles/talks, etc, I developed a PoC library that mixes: Process +
Threads + Asyncio Tasks, doing an scheme like this diagram:

 main -> Process 1 -> Thread 1.1 -> Task 1.1.1
  -> Task 1.1.2
  -> Task 1.1.3

-> Thread 1.2
 -> Task 1.2.1
 -> Task 1.2.2
 -> Task 1.2.3

 Process 2 -> Thread 2.1 -> Task 2.1.1
 -> Task 2.1.2
 -> Task 2.1.3

  -> Thread 2.2
 -> Task 2.2.1
 -> Task 2.2.2
 -> Task 2.2.3

In my local tests, this approach appear to improve (and simplify) the
concurrency/parallelism for some tasks but, before release the library at
github, I don't know if my aproach is wrong and I would appreciate your
opinion.

Thank you very much for your time.

Regards!

-- 
Daniel García a.k.a. cr0hn - Security researcher and pentester
@ggdaniel
http://www.cr0hn.com/me/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Updated PEP 509

[Oscar Benjamin]

On 18 April 2016 at 12:46, Jim J. Jewett  wrote:
>>
>> * I removed the dict[key]=value; dict[key]=value. It's really a
>> micro-optimization. I also fear that Raymond will complain because it
>> adds an if in the hot code of dict, and the dict type is very
>> important for Python performance.
>
> That is an acceptable answer.  Though I really do prefer explicitly
> *refusing to promise* either way when the replacement/replaced objects
> are ==.
>
> dicts (and other collections) already assume sensible ==, even
> explicitly allowing self-matches of objects that are not equal to
> themselves.  I don't like the idea of making new promises that violate
> (or rely on violations of) that sensible == assumption.

dicts make assumptions about the behaviour of __eq__ for the *keys*
but not for the *values* (on which no assumptions are made). The only
way to replace a key in a dict with another equal key (having a
well-behaved hash function) is to pop the key out and then insert the
new key so it's not possible to replace a key with another equal key
without bumping the version twice. So presumably you're referring to
the values here right?

The purpose of the PEP is to be able to guard for changes to
namespaces which are implemented as dicts. So if
builtins.__dict__['abs'] is replaced by foo then we don't care what
foo.__eq__ says about the situation: any optimisation that assumed
builtins.abs was not monkeypatched is invalidated. That's why the
version update is needed. Without it the version cannot be relied upon
as an optimisation guard. Consider:

class MyAbs:
def __eq__(self, other):
return True
def __call__(self, arg):
return - arg

builtins.abs = MyAbs()

--
Oscar
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Ethan Furman]

On 04/18/2016 12:41 AM, Nick Coghlan wrote:


Given the variant you [Koos] suggested, what if we defined the API semantics
like this:

 # Offer the simplest possible API as the public vesion
 def fspath(pathlike) -> str:
 return os._raw_fspath(pathlike)

 # Expose the complexity in the "private" variant
 def _raw_fspath(pathlike, *, output_types = (str,)) -> (str, bytes):
 # Short-circuit for instances of the output type
 if isinstance(pathlike, output_types):
 return pathlike
 # We'd have a tidier error message here for non-path objects
 result = pathlike.__fspath__()
 if not isinstance(result, output_types):
 raise TypeError("argument is not and does not provide an
acceptable pathname")
 return result


My initial reaction was that this was overly complex, but after thinking 
about it a couple days I /really/ like it.  It has a reasonable default 
for the 99% real-world use-case, while still allowing for custom and 
exact tailoring (for the 99% stdlib use-case ;) .


--
~Ethan~

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Updated PEP 509

[Jim J. Jewett]

On Sat, Apr 16, 2016 at 5:01 PM, Victor Stinner
 wrote:
> * I mentionned that version++ must be atomic, and that in the case of
> CPython, it's done by the GIL

Better; if those methods *already* hold the GIL, it is worth saying
"already", to indicate that the change is not expensive.

> * I removed the dict[key]=value; dict[key]=value. It's really a
> micro-optimization. I also fear that Raymond will complain because it
> adds an if in the hot code of dict, and the dict type is very
> important for Python performance.

That is an acceptable answer.  Though I really do prefer explicitly
*refusing to promise* either way when the replacement/replaced objects
are ==.

dicts (and other collections) already assume sensible ==, even
explicitly allowing self-matches of objects that are not equal to
themselves.  I don't like the idea of making new promises that violate
(or rely on violations of) that sensible == assumption.

-jJ
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 509: Add a private version to dict

[Jim J. Jewett]

On Fri, Apr 15, 2016 at 7:31 PM, Victor Stinner
 wrote:
> .2016-04-15 23:45 GMT+02:00 Jim J. Jewett :
...
>> I just worry that you may end up closing off even better optimizations
>> later, if you make too many promises about exactly how you will do
>> which ones.

>> Today, dict only cares about ==, and you (reasonably) think that full
>> == isn't always worth running ... but when it comes to which tests
>> *are* worth running, I'm not confident that the answers won't change
>> over the years.

> I checked, currently there is no unit test for a==b, only for a is b.
> I will add add a test for a==b but a is not b, and ensure that the
> version is increased.

Again, why?  Why not just say "If an object is replaced by something
equal to itself, the version_tag may not be changed.  While the
initial heuristics are simply to check for identity but not full
equality, this may change in future releases."

>> For example, if I know that my dict values are all 4-digit integers,
>> can I write:
>>
>> d[k]  = d[k] + 0
>>
>> and be assured that the version_tag will bump?  Or is that something
>> that a future optimizer might optimize out?

> Hum, I will try to clarify that.

I would prefer that you clarify it to say that while the initial patch
doesn't optimize that out, a future optimizer might.

> The problem with storing an identifier (a pointer in C) with no strong
> reference is when the object is destroyed, a new object can likely get
> the same identifier. So it's likely that "dict[key] is old_value_id"
> can be true even if dict[key] is now a new object.

Yes, but it shouldn't actually be destroyed until it is removed from
the dict, which should change version_tag, so that there will be no
need to compare it.

> Do you want to modify the PEP 509 to fix this issue? Or you don't
> understand why the PEP 509 cannot be used to fix the issue? I'm
> lost...

I believe it *does* fix the issue in some (but not all) cases.

-jJ
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] My first post here ~ do you need more Python core developers on Windows?

[Victor Stinner]

2016-04-18 7:23 GMT+02:00 Burkhard Meier :
> My name is Burkhard Meier and I wrote the "Python GUI Programming Cookbook"
> published by Packt.
>
> It is available on Amazon and PacktPub.com.

Welcome!

> Maybe I can become more involved in the Python community as a Python
> developer on Windows .

You can use the Developer Guide to start:
https://docs.python.org/devguide/

See also the Python menthors to get help on a dedicated and private
mailing list:
http://pythonmentors.com/

Sadly yes, we have many open issues specific to Windows. I'm trying to
sometimes give time to fix some of them, but I'm less interested than
in open source operating systems ;-)

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] My first post here ~ do you need more Python core developers on Windows?

[Nick Coghlan]

On 18 April 2016 at 15:23, Burkhard Meier  wrote:

> Maybe I can become more involved in the Python community as a Python
> developer on Windows .
>

Welcome! We definitely still have a marked skew towards Linux and *nix
programmers in general relative to the global software development
population, so participation from additional experienced Windows developers
is always appreciated :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()

[Nick Coghlan]

On 18 April 2016 at 07:05, Koos Zevenhoven  wrote:

> On Sun, Apr 17, 2016 at 9:14 PM, Ethan Furman  wrote:
> > On 04/17/2016 06:58 AM, Koos Zevenhoven wrote:
> >
> >> So, as a summary: With a str+bytes-polymorphic __fspath__, with the
> >> above argumentation and the rough implementation of os.fspath(...),
> >> the conclusion is that the os.fspath function should indeed be public,
> >> and that no further variations are needed.
> >
> >
> > Nice summation, thank you.  :)
> >
>
> Come on, Ethan, that summary was not for you ;)


As Chris noted though, the "Yes, that summary is accurate" from active
participants in the discussion helps assure readers that it's a good
overview :)

Given the variant you suggested, what if we defined the API semantics like
this:

# Offer the simplest possible API as the public vesion
def fspath(pathlike) -> str:
return os._raw_fspath(pathlike)

# Expose the complexity in the "private" variant
def _raw_fspath(pathlike, *, output_types = (str,)) -> (str, bytes):
# Short-circuit for instances of the output type
if isinstance(pathlike, output_types):
return pathlike
# We'd have a tidier error message here for non-path objects
result = pathlike.__fspath__()
if not isinstance(result, output_types):
raise TypeError("argument is not and does not provide an
acceptable pathname")
return result

That way, the default API would be saying unambiguously that the preferred
way of manipulating filesystem paths is as text, but the lower level
"mainly for the standard library" API would explicitly handle the 3
different scenarios (binary-input-is-a-bug, text-input-is-a-bug, and
either-binary-or-text-input-is-fine).

That way the structure of the additional parameters on _raw_fspath can be
tailored specifically to the needs of the standard library, without
worrying as much about 3rd party use cases.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com