Re: [Python-Dev] PEP 467: last round (?)

[Greg Ewing]

Ethan Furman wrote:
The problem with only having `bchr` is that it doesn't help with 
`bytearray`; the problem with not having `bchr` is who wants to write 
`bytes.fromord`?


If we called it 'bytes.fnord' (From Numeric Ordinal)
people would want to write it just for the fun factor.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Update on PEP 523 and adding a co_extra field to code objects

[Brett Cannon]

On Fri, 2 Sep 2016 at 17:37 MRAB  wrote:

> On 2016-09-02 23:45, Brett Cannon wrote:
> >
> >
> > On Fri, 2 Sep 2016 at 15:11 Chris Angelico  > > wrote:
> >
> > On Sat, Sep 3, 2016 at 7:56 AM, Brett Cannon  > > wrote:
> > > On Fri, 2 Sep 2016 at 13:31 Dino Viehland via Python-Dev
> > > > wrote:
> > >>
> > >> So it looks like both list and tuple are about within 5% of using
> > co_extra
> > >> directly.  Using a tuple instead of a list is about a wash except
> for
> > >> make_v2 where list is 1.4x slower for some reason (which I didn't
> > dig into).
> > >>
> > >> I would say that using a tuple and copying the tuple on updates
> makes
> > >> sense as we don't expect these to change very often and we don't
> > expect
> > >> collisions to happen very often.
> > >
> > >
> > > So would making co_extra a PyTupleObject instead of PyObject
> alleviate
> > > people's worry of a collision problem? You're going to have to
> > hold the GIL
> > > anyway to interact with the tuple so there won't be any race
> > condition in
> > > replacing the tuple when it's grown (or initially set).
> > >
> >
> > I'm not following how this solves the collision problem. If you have
> a
> > tuple, how do the two (or more) users of it know which index they're
> > using? They'd need to keep track separately for each object, or else
> > inefficiently search the tuple for an object of appropriate type
> every
> > time. What am I missing here?
> >
> >
> > You're not missing anything, you just have to pay for the search cost,
> > otherwise we're back to square one here of not worrying about the case
> > of multiple users. I don't see how you can have multiple users of a
> > single struct field and yet not have to do some search of some data
> > structure to find the relevant object you care about. We've tried maps
> > and dicts and they were too slow, and we proposed not worrying about
> > multiple users but people didn't like the idea of either not caring or
> > relying on some implicit practice that  evolved around the co_extra
> > field. Using a tuple seems to be the best option we can come up with
> > short of developing a linked list which isn't that much better than a
> > tuple if you're simply storing PyObjects. So either we're sticking with
> > the lack of coordination as outlined in the PEP because you don't
> > imagine people using a combination of Pyjion, vmprof, and/or some
> > debugger simultaneously, or you do and we have to just eat the
> > performance degradation.
> >
> Could the users register themselves first? They could then be told what
> index to use.
>

But that requires they register before any tuple is created, else they run
the risk of seeing a tuple that was created before they registered. To
cover that issue you then have to check the length at which point it's no
more expensive than just iterating through a tuple (especially in the
common case of a tuple of length 1).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What should a good type checker do? (was: Please reject or postpone PEP 526)

[Greg Ewing]

Chris Angelico wrote:

Forcing people to write 1.0 just to be compatible with 1.5 will cause
a lot of annoyance.


Indeed, this would be unacceptable IMO.

The checker could have a type 'smallint' that it
considers promotable to float. But that wouldn't
avoid the problem entirely, because e.g. adding
two smallints doesn't necessarily give a smallint.

Seems to me the practical thing is just to always
allow ints to be promoted to floats. It's possible
for runtime errors to result, but this is Python,
so we're used to those.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Update on PEP 523 and adding a co_extra field to code objects

[MRAB]

On 2016-09-02 23:45, Brett Cannon wrote:



On Fri, 2 Sep 2016 at 15:11 Chris Angelico > wrote:

On Sat, Sep 3, 2016 at 7:56 AM, Brett Cannon > wrote:
> On Fri, 2 Sep 2016 at 13:31 Dino Viehland via Python-Dev
> > wrote:
>>
>> So it looks like both list and tuple are about within 5% of using
co_extra
>> directly.  Using a tuple instead of a list is about a wash except for
>> make_v2 where list is 1.4x slower for some reason (which I didn't
dig into).
>>
>> I would say that using a tuple and copying the tuple on updates makes
>> sense as we don't expect these to change very often and we don't
expect
>> collisions to happen very often.
>
>
> So would making co_extra a PyTupleObject instead of PyObject alleviate
> people's worry of a collision problem? You're going to have to
hold the GIL
> anyway to interact with the tuple so there won't be any race
condition in
> replacing the tuple when it's grown (or initially set).
>

I'm not following how this solves the collision problem. If you have a
tuple, how do the two (or more) users of it know which index they're
using? They'd need to keep track separately for each object, or else
inefficiently search the tuple for an object of appropriate type every
time. What am I missing here?


You're not missing anything, you just have to pay for the search cost,
otherwise we're back to square one here of not worrying about the case
of multiple users. I don't see how you can have multiple users of a
single struct field and yet not have to do some search of some data
structure to find the relevant object you care about. We've tried maps
and dicts and they were too slow, and we proposed not worrying about
multiple users but people didn't like the idea of either not caring or
relying on some implicit practice that  evolved around the co_extra
field. Using a tuple seems to be the best option we can come up with
short of developing a linked list which isn't that much better than a
tuple if you're simply storing PyObjects. So either we're sticking with
the lack of coordination as outlined in the PEP because you don't
imagine people using a combination of Pyjion, vmprof, and/or some
debugger simultaneously, or you do and we have to just eat the
performance degradation.

Could the users register themselves first? They could then be told what 
index to use.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: last round (?)

[Random832]

On Fri, Sep 2, 2016, at 19:44, Ethan Furman wrote:
> The problem with only having `bchr` is that it doesn't help with
> `bytearray`;

What is the use case for bytearray.fromord? Even in the rare case
someone needs it, why not bytearray(bchr(...))?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: last round (?)

[Ethan Furman]

On 09/01/2016 04:07 PM, Victor Stinner wrote:

2016-09-02 0:04 GMT+02:00 Ethan Furman:



- `fromord` to replace the mistaken purpose of the default constructor


To replace a bogus bytes(obj)? If someone writes bytes(obj) but expect
to create a byte string from an integer, why not using bchr() to fix
the code?


The problem with only having `bchr` is that it doesn't help with `bytearray`; 
the problem with not having `bchr` is who wants to write `bytes.fromord`?

So we need `bchr`, and we need `bytearray.fromord`; and since the major 
difference between `bytes` and `bytearray` is that one is mutable and one is 
not, `bytes` should also have `fromord`.

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What should a good type checker do? (was: Please reject or postpone PEP 526)

[Random832]

On Fri, Sep 2, 2016, at 18:49, Koos Zevenhoven wrote:
> Then again, (b) instead of that being working code, it might be an
> error and spam only takes float. No problem, the type checker will
> catch that.

There are very few functions that should only take float and not int.

> On Fri, Sep 2, 2016 at 9:04 PM, Steven D'Aprano 
> wrote:
> > Maybe you think that it's okay because ints and floats are somewhat
> > compatible. But suppose I wrote:
> >
> > if cond:
> > x = HTTPServer(*args)
> > else:
> > x = 1.5
> 
> It might be clear by now, but no, that's not why I wrote that. That
> was just a slightly more "realistic" example than this HTTP & 1.5 one.

The other thing is... I'd kind of want it to infer Number in the first
case.

And if I assign both a list and a generator expression to something,
that should be Iterable[whatever] and maybe even whatever gets worked
out for "proper iterable and not string or bytes or memoryview".

Certainly if you can return HTTPServer() or None it should infer
Union[HTTPServer, None], otherwise spelled Optional[HTTPServer].

Maybe what we need is a "protoize"-alike that can run through a source
file and produce a stub file with all its inferences, for manual
inspection. So if you see something nonsensical like Union[HTTPServer,
float] you can think "wait a minute, where's that coming from" and go
look at the code.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What should a good type checker do? (was: Please reject or postpone PEP 526)

[Guido van Rossum]

Won't you ll agree that this thread belongs on python-ideas?

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What should a good type checker do? (was: Please reject or postpone PEP 526)

[Chris Angelico]

On Sat, Sep 3, 2016 at 8:49 AM, Koos Zevenhoven  wrote:
> On Fri, Sep 2, 2016 at 9:04 PM, Steven D'Aprano  wrote:
>> On Fri, Sep 02, 2016 at 08:10:24PM +0300, Koos Zevenhoven wrote:
>>
>>> A good checker should be able to infer that x is a union type at the
>>> point that it's passed to spam, even without the type annotation. For
>>> example:
>>>
>>> def eggs(cond:bool):
>>> if cond:
>>> x = 1
>>> else:
>>> x = 1.5
>>> spam(x)   # a good type checker infers that x is of type Union[int, 
>>> float]
>>
>> Oh I really hope not. I wouldn't call that a *good* type checker. I
>> would call that a type checker that is overly permissive.
>
> I guess it's perfectly fine if we disagree about type checking ideals,
> and I can imagine the justification for you thinking that way. There
> can also be different type checkers, and which can have different
> modes.
>
> But assume (a) that the above function is perfectly working code, and
> spam(...) accepts Union[int, float]. Why would I want the type checker
> to complain?

I wonder if it would be different if you wrote that as a single expression:

x = 1 if cond else 1.5

x = sum([1] + [0.5] * cond)

What should type inference decide x is in these cases? Assume an
arbitrarily smart type checker that can implement your ideal; it's
equally plausible to pretend that the type checker can recognize an
if/else block (or even if/elif/else tree of arbitrary length) as a
single "assignment" operation. IMO both of these examples - and by
extension, the if/else of the original - should be assigning a Union
type. Lots of Python code assumes that smallish integers [1] are
entirely compatible with floats. Is Python 4 going to have to deal
with the int/float distinction the way Python 3 did for bytes/text, or
are they fundamentally compatible concepts? Is the "small integer"
like the "ASCII byte/character" as a kind of hybrid beast that people
treat as simultaneously two types? (Personally, I don't think it's
anything like bytes/text. But I'm open to argument.)

Forcing people to write 1.0 just to be compatible with 1.5 will cause
a lot of annoyance. I leave it to you to decide whether there's a
fundamental difference that needs to be acknowledged, or just
subtleties of representational limitations to be ignored until they
become a problem.

ChrisA

[1] And by "smallish" I mean less than 2**53. Big enough for a lot of
purposes. Bigger (by definition) than JavaScript's integers, which cap
out at 2**32.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Update on PEP 523 and adding a co_extra field to code objects

[Chris Angelico]

On Sat, Sep 3, 2016 at 8:45 AM, Brett Cannon  wrote:
>> I'm not following how this solves the collision problem. If you have a
>> tuple, how do the two (or more) users of it know which index they're
>> using? They'd need to keep track separately for each object, or else
>> inefficiently search the tuple for an object of appropriate type every
>> time. What am I missing here?
>
>
> You're not missing anything, you just have to pay for the search cost,
> otherwise we're back to square one here of not worrying about the case of
> multiple users. I don't see how you can have multiple users of a single
> struct field and yet not have to do some search of some data structure to
> find the relevant object you care about. We've tried maps and dicts and they
> were too slow, and we proposed not worrying about multiple users but people
> didn't like the idea of either not caring or relying on some implicit
> practice that  evolved around the co_extra field. Using a tuple seems to be
> the best option we can come up with short of developing a linked list which
> isn't that much better than a tuple if you're simply storing PyObjects. So
> either we're sticking with the lack of coordination as outlined in the PEP
> because you don't imagine people using a combination of Pyjion, vmprof,
> and/or some debugger simultaneously, or you do and we have to just eat the
> performance degradation.

Got it, thanks. I hope the vagaries of linear search don't mess with
profilers - a debugger isn't going to be bothered by whether it gets
first slot or second, but profiling and performance might get subtle
differences based on which thing looks at a function first. A dict
would avoid that (constant-time lookups with a pre-selected key will
be consistent), but costs a lot more.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] What should a good type checker do? (was: Please reject or postpone PEP 526)

[Koos Zevenhoven]

On Fri, Sep 2, 2016 at 9:04 PM, Steven D'Aprano  wrote:
> On Fri, Sep 02, 2016 at 08:10:24PM +0300, Koos Zevenhoven wrote:
>
>> A good checker should be able to infer that x is a union type at the
>> point that it's passed to spam, even without the type annotation. For
>> example:
>>
>> def eggs(cond:bool):
>> if cond:
>> x = 1
>> else:
>> x = 1.5
>> spam(x)   # a good type checker infers that x is of type Union[int, 
>> float]
>
> Oh I really hope not. I wouldn't call that a *good* type checker. I
> would call that a type checker that is overly permissive.

I guess it's perfectly fine if we disagree about type checking ideals,
and I can imagine the justification for you thinking that way. There
can also be different type checkers, and which can have different
modes.

But assume (a) that the above function is perfectly working code, and
spam(...) accepts Union[int, float]. Why would I want the type checker
to complain?

Then again, (b) instead of that being working code, it might be an
error and spam only takes float. No problem, the type checker will
catch that.

In case of (b), to get the behavior you want (but in my hypothetical
semantics), this could be annotated as

def eggs(cond:bool):
x : float
if cond:
x = 1  # type checker says error
else:
x = 1.5
spam(x)

So here the programmer thinks the type of x should be more constrained
than what spam(...) accepts.

Or you might have something like this

def eggs(cond:bool):
if cond:
x = 1
else:
x = 1.5
# type checker has inferred x to be Union[int, float]
x : float  # type checker finds an error
spam(x)

Here, the same error is found, but at a different location.

> Maybe you think that it's okay because ints and floats are somewhat
> compatible. But suppose I wrote:
>
> if cond:
> x = HTTPServer(*args)
> else:
> x = 1.5

It might be clear by now, but no, that's not why I wrote that. That
was just a slightly more "realistic" example than this HTTP & 1.5 one.

[...]
> Do you have a better idea for variable
> syntax?

I had one but it turned out it was worse.

-- Koos

>
>
> --
> Steve
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Update on PEP 523 and adding a co_extra field to code objects

[Brett Cannon]

On Fri, 2 Sep 2016 at 15:11 Chris Angelico  wrote:

> On Sat, Sep 3, 2016 at 7:56 AM, Brett Cannon  wrote:
> > On Fri, 2 Sep 2016 at 13:31 Dino Viehland via Python-Dev
> >  wrote:
> >>
> >> So it looks like both list and tuple are about within 5% of using
> co_extra
> >> directly.  Using a tuple instead of a list is about a wash except for
> >> make_v2 where list is 1.4x slower for some reason (which I didn't dig
> into).
> >>
> >> I would say that using a tuple and copying the tuple on updates makes
> >> sense as we don't expect these to change very often and we don't expect
> >> collisions to happen very often.
> >
> >
> > So would making co_extra a PyTupleObject instead of PyObject alleviate
> > people's worry of a collision problem? You're going to have to hold the
> GIL
> > anyway to interact with the tuple so there won't be any race condition in
> > replacing the tuple when it's grown (or initially set).
> >
>
> I'm not following how this solves the collision problem. If you have a
> tuple, how do the two (or more) users of it know which index they're
> using? They'd need to keep track separately for each object, or else
> inefficiently search the tuple for an object of appropriate type every
> time. What am I missing here?
>

You're not missing anything, you just have to pay for the search cost,
otherwise we're back to square one here of not worrying about the case of
multiple users. I don't see how you can have multiple users of a single
struct field and yet not have to do some search of some data structure to
find the relevant object you care about. We've tried maps and dicts and
they were too slow, and we proposed not worrying about multiple users but
people didn't like the idea of either not caring or relying on some
implicit practice that  evolved around the co_extra field. Using a tuple
seems to be the best option we can come up with short of developing a
linked list which isn't that much better than a tuple if you're simply
storing PyObjects. So either we're sticking with the lack of coordination
as outlined in the PEP because you don't imagine people using a combination
of Pyjion, vmprof, and/or some debugger simultaneously, or you do and we
have to just eat the performance degradation.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Updated version of PEP 526 (Syntax for Variable Annotations)

[Guido van Rossum]

We've prepared an updated version of PEP 526:
https://www.python.org/dev/peps/pep-0526/

This changes the title to "Syntax for Variable Annotations", now that
we've settled on global, class, instance, and local variables as the
things you might annotate.

There is one substantial change: where the previous version supported only

  NAME: TYPE
  TARGET: TYPE = VALUE

the new PEP removes the distinction and just allows

  TARGET: TYPE [= VALUE]

This simplifies the explanations a bit and enables type checkers to
support separating the annotation from the assignment for instance
variables in the __init__ method, e.g.

  def __init__(self):
  self.name: str
  if :
  self.name = 
  else:
  self.name = 

The other changes are all minor editing nits, or clarifications about
the scope of the PEP. The scope clarification is important: while I
really want the new syntax settled in 3.6, I have no intention to pin
down the way type checkers use this syntax, apart from the observation
that

  TARGET: TYPE = VALUE

is just meant as a cleaner way to write what you'd currently write
using PEP 484 as

  TARGET = VALUE  # type: TYPE

The PEP does *not* claim that you have to use variable annotations --
in fact we'd prefer that they were unnecessary, but the prevalence of
type comments in code we've annotated so far makes it clear that there
are plenty of uses for them, and we'd rather have a clean syntax for
them that tools can see in the AST.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Update on PEP 523 and adding a co_extra field to code objects

[Chris Angelico]

On Sat, Sep 3, 2016 at 7:56 AM, Brett Cannon  wrote:
> On Fri, 2 Sep 2016 at 13:31 Dino Viehland via Python-Dev
>  wrote:
>>
>> So it looks like both list and tuple are about within 5% of using co_extra
>> directly.  Using a tuple instead of a list is about a wash except for
>> make_v2 where list is 1.4x slower for some reason (which I didn't dig into).
>>
>> I would say that using a tuple and copying the tuple on updates makes
>> sense as we don't expect these to change very often and we don't expect
>> collisions to happen very often.
>
>
> So would making co_extra a PyTupleObject instead of PyObject alleviate
> people's worry of a collision problem? You're going to have to hold the GIL
> anyway to interact with the tuple so there won't be any race condition in
> replacing the tuple when it's grown (or initially set).
>

I'm not following how this solves the collision problem. If you have a
tuple, how do the two (or more) users of it know which index they're
using? They'd need to keep track separately for each object, or else
inefficiently search the tuple for an object of appropriate type every
time. What am I missing here?

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Update on PEP 523 and adding a co_extra field to code objects

[Brett Cannon]

On Fri, 2 Sep 2016 at 13:31 Dino Viehland via Python-Dev <
python-dev@python.org> wrote:

> So it looks like both list and tuple are about within 5% of using co_extra
> directly.  Using a tuple instead of a list is about a wash except for
> make_v2 where list is 1.4x slower for some reason (which I didn't dig into).
>
> I would say that using a tuple and copying the tuple on updates makes
> sense as we don't expect these to change very often and we don't expect
> collisions to happen very often.
>

So would making co_extra a PyTupleObject instead of PyObject alleviate
people's worry of a collision problem? You're going to have to hold the GIL
anyway to interact with the tuple so there won't be any race condition in
replacing the tuple when it's grown (or initially set).

-Brett


>
> > -Original Message-
> > From: Python-Dev [mailto:python-dev-
> > bounces+dinov=microsoft@python.org] On Behalf Of Chris Angelico
> > Sent: Tuesday, August 30, 2016 2:11 PM
> > To: python-dev 
> > Subject: Re: [Python-Dev] Update on PEP 523 and adding a co_extra field
> to
> > code objects
> >
> > On Wed, Aug 31, 2016 at 4:55 AM, Serhiy Storchaka 
> > wrote:
> > > On 30.08.16 21:20, Antoine Pitrou wrote:
> > >>
> > >> On Tue, 30 Aug 2016 18:12:01 +
> > >> Brett Cannon  wrote:
> > 
> >  Why not make it always a list?  List objects are reasonably cheap
> >  in memory and access time... (unlike dicts)
> > >>>
> > >>>
> > >>> Because I would prefer to avoid any form of unnecessary performance
> > >>> overhead for the common case.
> > >>
> > >>
> > >> But the performance overhead of iterating over a 1-element list is
> > >> small enough (it's just an array access after a pointer dereference)
> > >> that it may not be larger than the overhead of the multiple tests and
> > >> conditional branches your example shows.
> > >
> > >
> > > Iterating over a tuple is even faster. It needs one pointer
> > > dereference less.
> > >
> > > And for memory efficiency we can use just a raw array of pointers.
> >
> > Didn't all this kind of thing come up when function annotations were
> > discussed? Insane schemes like dictionaries with UUID keys and so on.
> > The decision then was YAGNI. The decision now, IMO, should be the same.
> > Keep things simple.
> >
> > ChrisA
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> >
> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fmail.pyt
> > hon.org%2fmailman%2flistinfo%2fpython-
> > dev=01%7c01%7cdinov%40microsoft.com%7c9d750b06b2134a2145c70
> > 8d3d11a4ab0%7c72f988bf86f141af91ab2d7cd011db47%7c1=szub1gs
> > DW2rdns3IQGV68J3tCqWiNcjqG77xYIfoORc%3d
> > Unsubscribe:
> >
> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fmail.pyt
> > hon.org%2fmailman%2foptions%2fpython-
> > dev%2fdinov%2540microsoft.com=01%7c01%7cdinov%40microsoft.co
> > m%7c9d750b06b2134a2145c708d3d11a4ab0%7c72f988bf86f141af91ab2d7c
> > d011db47%7c1=TEzMSyJLmAe2BVZGPugXAh6bga2xN1WQw3bR0z0b
> > %2fLg%3d
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Update on PEP 523 and adding a co_extra field to code objects

[Dino Viehland via Python-Dev]

So it looks like both list and tuple are about within 5% of using co_extra 
directly.  Using a tuple instead of a list is about a wash except for make_v2 
where list is 1.4x slower for some reason (which I didn't dig into).

I would say that using a tuple and copying the tuple on updates makes sense as 
we don't expect these to change very often and we don't expect collisions to 
happen very often.

> -Original Message-
> From: Python-Dev [mailto:python-dev-
> bounces+dinov=microsoft@python.org] On Behalf Of Chris Angelico
> Sent: Tuesday, August 30, 2016 2:11 PM
> To: python-dev 
> Subject: Re: [Python-Dev] Update on PEP 523 and adding a co_extra field to
> code objects
> 
> On Wed, Aug 31, 2016 at 4:55 AM, Serhiy Storchaka 
> wrote:
> > On 30.08.16 21:20, Antoine Pitrou wrote:
> >>
> >> On Tue, 30 Aug 2016 18:12:01 +
> >> Brett Cannon  wrote:
> 
>  Why not make it always a list?  List objects are reasonably cheap
>  in memory and access time... (unlike dicts)
> >>>
> >>>
> >>> Because I would prefer to avoid any form of unnecessary performance
> >>> overhead for the common case.
> >>
> >>
> >> But the performance overhead of iterating over a 1-element list is
> >> small enough (it's just an array access after a pointer dereference)
> >> that it may not be larger than the overhead of the multiple tests and
> >> conditional branches your example shows.
> >
> >
> > Iterating over a tuple is even faster. It needs one pointer
> > dereference less.
> >
> > And for memory efficiency we can use just a raw array of pointers.
> 
> Didn't all this kind of thing come up when function annotations were
> discussed? Insane schemes like dictionaries with UUID keys and so on.
> The decision then was YAGNI. The decision now, IMO, should be the same.
> Keep things simple.
> 
> ChrisA
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fmail.pyt
> hon.org%2fmailman%2flistinfo%2fpython-
> dev=01%7c01%7cdinov%40microsoft.com%7c9d750b06b2134a2145c70
> 8d3d11a4ab0%7c72f988bf86f141af91ab2d7cd011db47%7c1=szub1gs
> DW2rdns3IQGV68J3tCqWiNcjqG77xYIfoORc%3d
> Unsubscribe:
> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fmail.pyt
> hon.org%2fmailman%2foptions%2fpython-
> dev%2fdinov%2540microsoft.com=01%7c01%7cdinov%40microsoft.co
> m%7c9d750b06b2134a2145c708d3d11a4ab0%7c72f988bf86f141af91ab2d7c
> d011db47%7c1=TEzMSyJLmAe2BVZGPugXAh6bga2xN1WQw3bR0z0b
> %2fLg%3d
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Guido van Rossum]

On Fri, Sep 2, 2016 at 10:47 AM, Steve Dower  wrote:
> "I'm not seeing what distinction you think you are making here. What
> distinction do you see between:
>
> x: int = func(value)
>
> and
>
> x = func(value)  # type: int"
>
> Not sure whether I agree with Mark on this particular point, but the
> difference I see here is that the first describes what types x may ever
> contain, while the latter describes what type of being assigned to x right
> here. So one is a variable annotation while the other is an expression
> annotation.

But that's not what type comments mean! They don't annotate the
expression. They annotate the variable. The text in PEP 484 that
introduces them is clear about this (it never mentions expressions,
only variables).

> Personally, I prefer expression annotations over variable annotations, as
> there are many other languages I'd prefer if variable have fixed types (e.g.
> C++, where I actually enjoy doing horrible things with implicit casting ;)
> ).
>
> Variable annotations appear to be inherently restrictive, so either we need
> serious clarification as to why they are not, or they actually are and we
> ought to be more sure that it's the direction we want the language to go.

At runtime the variable annotations are ignored. And a type checker
will only ask for them when it cannot infer the type. So I think we'll
be fine.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Update on PEP 523 and adding a co_extra field to code objects

[Dino Viehland via Python-Dev]

So I ran the tests with both a list and a tuple.  They were about 5% slower on 
a handful of benchmarks, and then the difference between the tuple and list 
again had a few benchmarks that were around 5% slower.  There was one benchmark 
where the tuple one significantly for some reason (mako_v2) which was 1.4x 
slower.  It seems to me we should go with the tuple just because the common 
case will be having a single object and it'll be even less common to have these 
changing very frequently.

-Original Message-
From: Python-Dev [mailto:python-dev-bounces+dinov=microsoft@python.org] On 
Behalf Of Chris Angelico
Sent: Tuesday, August 30, 2016 2:11 PM
To: python-dev 
Subject: Re: [Python-Dev] Update on PEP 523 and adding a co_extra field to code 
objects

On Wed, Aug 31, 2016 at 4:55 AM, Serhiy Storchaka  wrote:
> On 30.08.16 21:20, Antoine Pitrou wrote:
>>
>> On Tue, 30 Aug 2016 18:12:01 +
>> Brett Cannon  wrote:

 Why not make it always a list?  List objects are reasonably cheap 
 in memory and access time... (unlike dicts)
>>>
>>>
>>> Because I would prefer to avoid any form of unnecessary performance 
>>> overhead for the common case.
>>
>>
>> But the performance overhead of iterating over a 1-element list is 
>> small enough (it's just an array access after a pointer dereference) 
>> that it may not be larger than the overhead of the multiple tests and 
>> conditional branches your example shows.
>
>
> Iterating over a tuple is even faster. It needs one pointer 
> dereference less.
>
> And for memory efficiency we can use just a raw array of pointers.

Didn't all this kind of thing come up when function annotations were discussed? 
Insane schemes like dictionaries with UUID keys and so on.
The decision then was YAGNI. The decision now, IMO, should be the same. Keep 
things simple.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fmail.python.org%2fmailman%2flistinfo%2fpython-dev=01%7c01%7cdinov%40microsoft.com%7c9d750b06b2134a2145c708d3d11a4ab0%7c72f988bf86f141af91ab2d7cd011db47%7c1=szub1gsDW2rdns3IQGV68J3tCqWiNcjqG77xYIfoORc%3d
Unsubscribe: 
https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fmail.python.org%2fmailman%2foptions%2fpython-dev%2fdinov%2540microsoft.com=01%7c01%7cdinov%40microsoft.com%7c9d750b06b2134a2145c708d3d11a4ab0%7c72f988bf86f141af91ab2d7cd011db47%7c1=TEzMSyJLmAe2BVZGPugXAh6bga2xN1WQw3bR0z0b%2fLg%3d
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Need help in debugging the python core

[Sajjanshetty, Amresh]

Yes

Thanks and Regards,
Amresh

From: Burkhard Meier 
Date: Saturday, September 3, 2016 at 12:12 AM
To: Amresh Sajjanshetty 
Cc: "python-dev@python.org" 
Subject: Re: [Python-Dev] Need help in debugging the python core


You are using bash?
On Sep 2, 2016 8:56 AM, "Sajjanshetty, Amresh" 
> wrote:
Dear All,

I’m using asyncio and paramiko to multiplex different channels into a single 
SSH connection. Things were working fine till recently but suddenly started 
seeing that python getting crashed whenever I tried to write to the channel. I 
have very limited knowledge on how python interpreter works, so I’m finding 
difficulty in understanding the stack trace. Can you please help in 
understanding the below backtarce.

bash-4.2$ gdb /usr/software/bin/python3.4.3 core.60015
Traceback (most recent call last):
  File "", line 70, in 
  File "", line 67, in GdbSetPythonDirectory
  File "/usr/software/share/gdb/python/gdb/__init__.py", line 19, in 
import _gdb
ImportError: No module named _gdb
GNU gdb (GDB) 7.5
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/software/bin/python3.4.3...done.

warning: core file may not match specified executable file.
[New LWP 60015]
[New LWP 60018]
[New LWP 60019]
[New LWP 60020]
[New LWP 60021]
[New LWP 60022]
[New LWP 60023]
[New LWP 60024]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/software/lib/libthread_db.so.1".
Core was generated by `/usr/software/bin/python3.4.3 
/x/eng/bbrtp/users/amresh/sshproxy_3896926_160824'.
Program terminated with signal 11, Segmentation fault.
#0  _PyObject_Malloc (ctx=0x0, nbytes=52) at Objects/obmalloc.c:1159
1159Objects/obmalloc.c: No such file or directory.
(gdb) bt
#0  _PyObject_Malloc (ctx=0x0, nbytes=52) at Objects/obmalloc.c:1159
#1  0x7ff2e511474a in PyUnicode_New (maxchar=, size=3) at 
Objects/unicodeobject.c:1093
#2  PyUnicode_New (size=3, maxchar=) at 
Objects/unicodeobject.c:1033
#3  0x7ff2e5139da2 in _PyUnicodeWriter_PrepareInternal 
(writer=writer@entry=0x7fff3d5c8640, length=, maxchar=, maxchar@entry=127) at Objects/unicodeobject.c:13327
#4  0x7ff2e513f38b in PyUnicode_DecodeUTF8Stateful 
(s=s@entry=0x7ff2e3572f78 
"tcp\245reuse\001\253socket_type\244pull\251transport\246zeromq", 
size=size@entry=3,
errors=errors@entry=0x7ff2dee5dd70 "strict", consumed=consumed@entry=0x0) 
at Objects/unicodeobject.c:4757
#5  0x7ff2e5140690 in PyUnicode_Decode (s=0x7ff2e3572f78 
"tcp\245reuse\001\253socket_type\244pull\251transport\246zeromq", size=3, 
encoding=0x7ff2dee5df28 "utf-8", errors=0x7ff2dee5dd70 "strict")
at Objects/unicodeobject.c:3012
#6  0x7ff2de49bfdf in unpack_callback_raw (o=, l=3, 
p=0x7ff2e3572f78 
"tcp\245reuse\001\253socket_type\244pull\251transport\246zeromq", 
u=0x7fff3d5c8840, b=)
at msgpack/unpack.h:229
#7  unpack_execute (ctx=ctx@entry=0x7fff3d5c8840, data=0x7ff2e3572ec0 
"\205\245_auth\300\245_call\246expect\243_i", , 
len=, off=off@entry=0x7fff3d5c8820)
at msgpack/unpack_template.h:312
#8  0x7ff2de49fe3d in __pyx_pf_7msgpack_9_unpacker_2unpackb 
(__pyx_v_packed=__pyx_v_packed@entry=0x7ff2e3572ea0, 
__pyx_v_object_hook=__pyx_v_object_hook@entry=0x7ff2e54934b0 <_Py_NoneStruct>,
__pyx_v_list_hook=__pyx_v_list_hook@entry=0x7ff2e54934b0 <_Py_NoneStruct>, 
__pyx_v_use_list=1, __pyx_v_encoding=0x7ff2dee5df08, 
__pyx_v_unicode_errors=0x7ff2dee5dd50,
__pyx_v_object_pairs_hook=0x7ff2e54934b0 <_Py_NoneStruct>, 
__pyx_v_ext_hook=0x13db2d8, 
__pyx_v_max_str_len=__pyx_v_max_str_len@entry=2147483647,
__pyx_v_max_bin_len=__pyx_v_max_bin_len@entry=2147483647, 
__pyx_v_max_array_len=2147483647, 
__pyx_v_max_map_len=2147483647, 
__pyx_v_max_ext_len=__pyx_v_max_ext_len@entry=2147483647,
__pyx_self=) at msgpack/_unpacker.pyx:139
#9  0x7ff2de4a1395 in __pyx_pw_7msgpack_9_unpacker_3unpackb 
(__pyx_self=, __pyx_args=, __pyx_kwds=) at msgpack/_unpacker.pyx:102
#10 0x7ff2e5174ed3 in do_call (nk=, na=, 
pp_stack=0x7fff3d5d2b80, func=0x7ff2df20ddc8) at Python/ceval.c:4463
#11 call_function (oparg=, pp_stack=0x7fff3d5d2b80) at 
Python/ceval.c:4264
#12 PyEval_EvalFrameEx (f=f@entry=0x7ff2def02208, throwflag=throwflag@entry=0) 
at Python/ceval.c:2838
#13 0x7ff2e5175f45 in PyEval_EvalCodeEx (_co=, 
globals=, locals=locals@entry=0x0, args=, 
argcount=argcount@entry=1, kws=0x7ff2deefec30, kwcount=0, defs=0x0,
defcount=0, kwdefs=0x0, closure=0x0) at Python/ceval.c:3588
#14 

Re: [Python-Dev] Need help in debugging the python core

[Burkhard Meier]

You are using bash?
On Sep 2, 2016 8:56 AM, "Sajjanshetty, Amresh" <
amresh.sajjanshe...@netapp.com> wrote:

> Dear All,
>
>
>
> I’m using asyncio and paramiko to multiplex different channels into a
> single SSH connection. Things were working fine till recently but suddenly
> started seeing that python getting crashed whenever I tried to write to the
> channel. I have very limited knowledge on how python interpreter works, so
> I’m finding difficulty in understanding the stack trace. Can you please
> help in understanding the below backtarce.
>
>
>
> bash-4.2$ gdb /usr/software/bin/python3.4.3 core.60015
>
> Traceback (most recent call last):
>
>   File "", line 70, in 
>
>   File "", line 67, in GdbSetPythonDirectory
>
>   File "/usr/software/share/gdb/python/gdb/__init__.py", line 19, in
> 
>
> import _gdb
>
> ImportError: No module named _gdb
>
> GNU gdb (GDB) 7.5
>
> Copyright (C) 2012 Free Software Foundation, Inc.
>
> License GPLv3+: GNU GPL version 3 or later  html>
>
> This is free software: you are free to change and redistribute it.
>
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>
> and "show warranty" for details.
>
> This GDB was configured as "x86_64-unknown-linux-gnu".
>
> For bug reporting instructions, please see:
>
> ...
>
> Reading symbols from /usr/software/bin/python3.4.3...done.
>
>
>
> warning: core file may not match specified executable file.
>
> [New LWP 60015]
>
> [New LWP 60018]
>
> [New LWP 60019]
>
> [New LWP 60020]
>
> [New LWP 60021]
>
> [New LWP 60022]
>
> [New LWP 60023]
>
> [New LWP 60024]
>
> [Thread debugging using libthread_db enabled]
>
> Using host libthread_db library "/usr/software/lib/libthread_db.so.1".
>
> Core was generated by `/usr/software/bin/python3.4.3
> /x/eng/bbrtp/users/amresh/sshproxy_3896926_160824'.
>
> Program terminated with signal 11, Segmentation fault.
>
> #0  _PyObject_Malloc (ctx=0x0, nbytes=52) at Objects/obmalloc.c:1159
>
> 1159Objects/obmalloc.c: No such file or directory.
>
> (gdb) bt
>
> #0  _PyObject_Malloc (ctx=0x0, nbytes=52) at Objects/obmalloc.c:1159
>
> #1  0x7ff2e511474a in PyUnicode_New (maxchar=, size=3)
> at Objects/unicodeobject.c:1093
>
> #2  PyUnicode_New (size=3, maxchar=) at
> Objects/unicodeobject.c:1033
>
> #3  0x7ff2e5139da2 in _PyUnicodeWriter_PrepareInternal
> (writer=writer@entry=0x7fff3d5c8640, length=,
> maxchar=, maxchar@entry=127) at
> Objects/unicodeobject.c:13327
>
> #4  0x7ff2e513f38b in PyUnicode_DecodeUTF8Stateful 
> (s=s@entry=0x7ff2e3572f78
> "tcp\245reuse\001\253socket_type\244pull\251transport\246zeromq",
> size=size@entry=3,
>
> errors=errors@entry=0x7ff2dee5dd70 "strict", consumed=consumed@entry=0x0)
> at Objects/unicodeobject.c:4757
>
> #5  0x7ff2e5140690 in PyUnicode_Decode (s=0x7ff2e3572f78
> "tcp\245reuse\001\253socket_type\244pull\251transport\246zeromq", size=3,
> encoding=0x7ff2dee5df28 "utf-8", errors=0x7ff2dee5dd70 "strict")
>
> at Objects/unicodeobject.c:3012
>
> #6  0x7ff2de49bfdf in unpack_callback_raw (o=, l=3,
> p=0x7ff2e3572f78 
> "tcp\245reuse\001\253socket_type\244pull\251transport\246zeromq",
> u=0x7fff3d5c8840, b=)
>
> at msgpack/unpack.h:229
>
> #7  unpack_execute (ctx=ctx@entry=0x7fff3d5c8840,
> data=0x7ff2e3572ec0 "\205\245_auth\300\245_call\246expect\243_i",
> , len=, off=off@entry
> =0x7fff3d5c8820)
>
> at msgpack/unpack_template.h:312
>
> #8  0x7ff2de49fe3d in __pyx_pf_7msgpack_9_unpacker_2unpackb
> (__pyx_v_packed=__pyx_v_packed@entry=0x7ff2e3572ea0,
> __pyx_v_object_hook=__pyx_v_object_hook@entry=0x7ff2e54934b0
> <_Py_NoneStruct>,
>
> __pyx_v_list_hook=__pyx_v_list_hook@entry=0x7ff2e54934b0
> <_Py_NoneStruct>, __pyx_v_use_list=1, __pyx_v_encoding=0x7ff2dee5df08,
> __pyx_v_unicode_errors=0x7ff2dee5dd50,
>
> __pyx_v_object_pairs_hook=0x7ff2e54934b0 <_Py_NoneStruct>,
> __pyx_v_ext_hook=0x13db2d8, __pyx_v_max_str_len=__pyx_v_max_str_len@entry=
> 2147483647,
>
> __pyx_v_max_bin_len=__pyx_v_max_bin_len@entry=2147483647,
> __pyx_v_max_array_len=2147483647, __pyx_v_max_map_len=2147483647,
> __pyx_v_max_ext_len=__pyx_v_max_ext_len@entry=2147483647,
>
> __pyx_self=) at msgpack/_unpacker.pyx:139
>
> #9  0x7ff2de4a1395 in __pyx_pw_7msgpack_9_unpacker_3unpackb
> (__pyx_self=, __pyx_args=,
> __pyx_kwds=) at msgpack/_unpacker.pyx:102
>
> #10 0x7ff2e5174ed3 in do_call (nk=, na=,
> pp_stack=0x7fff3d5d2b80, func=0x7ff2df20ddc8) at Python/ceval.c:4463
>
> #11 call_function (oparg=, pp_stack=0x7fff3d5d2b80) at
> Python/ceval.c:4264
>
> #12 PyEval_EvalFrameEx (f=f@entry=0x7ff2def02208,
> throwflag=throwflag@entry=0) at Python/ceval.c:2838
>
> #13 0x7ff2e5175f45 in PyEval_EvalCodeEx (_co=,
> globals=, locals=locals@entry=0x0, args=,
> argcount=argcount@entry=1, kws=0x7ff2deefec30, kwcount=0, defs=0x0,
>
> defcount=0, kwdefs=0x0, closure=0x0) at Python/ceval.c:3588
>
> #14 0x7ff2e51734da in fast_function (nk=, na=1,

Re: [Python-Dev] Need help in debugging the python core

[Sajjanshetty, Amresh]

Surprisingly I’m not seeing the core dump/crash after adding 
‘faulthandler.enable()’ . Would it catch the signal and ignore by default?

Thanks and Regards,
Amresh

From: Burkhard Meier 
Date: Friday, September 2, 2016 at 11:19 PM
To: Victor Stinner 
Cc: Amresh Sajjanshetty , 
"python-dev@python.org" 
Subject: Re: [Python-Dev] Need help in debugging the python core

How could I help?

Burkhard

On Fri, Sep 2, 2016 at 10:47 AM, Victor Stinner 
> wrote:
Oh, I forgot to mention that it would help to get the Python traceback
on the crash. Try faulthandler: add faulthandler.enable() at the
beginning of your program.
https://docs.python.org/dev/library/faulthandler.html

Maybe I should write once tools to debug such bug :-)

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/burkhardameier%40gmail.com

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Stefan Krah]

[Replying to  Steve Dower]
On Sat, Sep 03, 2016 at 04:19:13AM +1000, Steven D'Aprano wrote:
> On Fri, Sep 02, 2016 at 10:47:41AM -0700, Steve Dower wrote:
> > "I'm not seeing what distinction you think you are making here. What 
> > distinction do you see between:
> > 
> > x: int = func(value)
> > 
> > and
> > 
> > x = func(value)  #type: int"
> > 
> > Not sure whether I agree with Mark on this particular point, but the 
> > difference I see here is that the first describes what types x may 
> > ever contain, while the latter describes what type of being assigned 
> > to x right here. So one is a variable annotation while the other is an 
> > expression annotation.

I see it differently, but I'm quite used to OCaml:

# let f () =
let x : int = 10 in
let x : float = 320.0 in
  x;;
Warning 26: unused variable x.
val f : unit -> float = 
# f();;
- : float = 320.


Like in Python, in OCaml variables can be rebound and indeed have different
types with different explicit type constraints.


Expressions can also be annotated, but require parentheses (silly example):

# let x = (10 * 20 : int);;
val x : int = 200



So I'm quite happy with the proposed syntax in the PEP, perhaps the
parenthesized expression annotations could also be added.  But these
are only very rarely needed.


Stefan Krah



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Steven D'Aprano]

On Fri, Sep 02, 2016 at 10:47:41AM -0700, Steve Dower wrote:
> "I'm not seeing what distinction you think you are making here. What 
> distinction do you see between:
> 
> x: int = func(value)
> 
> and
> 
> x = func(value)  #type: int"
> 
> Not sure whether I agree with Mark on this particular point, but the 
> difference I see here is that the first describes what types x may 
> ever contain, while the latter describes what type of being assigned 
> to x right here. So one is a variable annotation while the other is an 
> expression annotation.

Ultimately Python is a dynamically typed language, and that's not 
changing. This means types are fundamentally associated with *values*, 
not *variables* (names). But in practice, you can go a long way by 
pretending that it is the variable that carries the type. That's the 
point of the static type checker: if you see that x holds an int here, 
then assume (unless told differently) that x should always be an int. 
Because in practice, most exceptions to that are due to bugs, or at 
least sloppy code.

Of course, it is up to the type checker to decide how strict it wants to 
be, whether to treat violations as a warning or a error, whether to 
offer the user a flag to set the behaviour, etc. None of this is 
relevant to the PEP. The PEP only specifies the syntax, leaving 
enforcement or non-enforcement to the checker, and it says:

PEP 484 introduced type hints, a.k.a. type annotations. While its 
main focus was function annotations, it also introduced the notion
of type comments to annotate VARIABLES [emphasis added]

not expressions. And:

This PEP aims at adding syntax to Python for annotating the types
of variables and attributes, instead of expressing them through
comments

which to me obviously implies that the two ways (type comment, and 
variable type hint) are intended to be absolutely identical in 
semantics, at least as far as the type-checker is concerned.

(They will have different semantics at runtime: the comment is just a 
comment, while the type hint will set an __annotations__ mapping.)

But perhaps the PEP needs to make it explicit that they are to be 
treated exactly the same.


-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Steven D'Aprano]

On Fri, Sep 02, 2016 at 08:10:24PM +0300, Koos Zevenhoven wrote:

> A good checker should be able to infer that x is a union type at the
> point that it's passed to spam, even without the type annotation. For
> example:
> 
> def eggs(cond:bool):
> if cond:
> x = 1
> else:
> x = 1.5
> spam(x)   # a good type checker infers that x is of type Union[int, float]

Oh I really hope not. I wouldn't call that a *good* type checker. I 
would call that a type checker that is overly permissive.

Maybe you think that it's okay because ints and floats are somewhat 
compatible. But suppose I wrote:

if cond:
x = HTTPServer(*args)
else:
x = 1.5

Would you want the checker to infer Union[HTTPServer, float]? I 
wouldn't. I would want the checker to complain that the two branches of 
the `if` result in different types for x. If I really mean it, then I 
can give a type-hint.

In any case, this PEP isn't about specifying when to declare variable 
types, it is for picking syntax. Do you have a better idea for variable 
syntax?



-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 526 ready for review: Syntax for Variable and Attribute Annotations

[Ivan Levkivskyi]

On 2 September 2016 at 18:17, Guido van Rossum  wrote:
> On Fri, Sep 2, 2016 at 6:43 AM, Ivan Levkivskyi 
wrote:
> > On 2 September 2016 at 04:38, Nick Coghlan  wrote:
> >> However, a standalone Ellipsis doesn't currently have a meaning as a
> >> type annotation (it's only meaningful when subscripting Tuple and
> >> Callable), so a spelling like this might work:
> >>
> >> NAME: ...
> >>
> >> That spelling could then also be used in function definitions to say
> >> "infer the return type from the return statements rather than assuming
> >> Any"
> >
> > Interesting idea.
> > This is somehow similar to one of the existing use of Ellipsis: in
numpy it
> > infers how many dimensions needs to have the full slice, it is like
saying
> > "You know what I mean". So I am +1 on this solution.
>
> I like it too, but I think it's better to leave any firm promises
> about the *semantics* of variable annotations out of the PEP. I just
> spoke to someone who noted that the PEP is likely to evoke an outsize
> emotional response. (Similar to what happened with PEP 484.)
>
> Pinning down the semantics is not why I am pushing for PEP 526 -- I
> only want to pin down the *syntax* to the point where we won't have to
> change it again for many versions, since it's much harder to change
> the syntax than it is to change the behavior of type checkers (which
> have fewer backwards compatibility constraints, a faster release
> cycle, and narrower user bases than core Python itself).

This is a good point. I totally agree.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 467: last round (?)

[Koos Zevenhoven]

Some quick comments below, a few more later:

On Thu, Sep 1, 2016 at 10:36 PM, Ethan Furman  wrote:
> One more iteration. PEPs repo not updated yet. Changes are renaming of
> methods to be ``fromsize()`` and ``fromord()``, and moving ``memoryview``
to
> an Open Questions section.
>
>
> PEP: 467
> Title: Minor API improvements for binary sequences
> Version: $Revision$
> Last-Modified: $Date$
> Author: Nick Coghlan , Ethan Furman <
et...@stoneleaf.us>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 2014-03-30
> Python-Version: 3.6
> Post-History: 2014-03-30 2014-08-15 2014-08-16 2016-06-07 2016-09-01
>
>
> Abstract
> 
>
> During the initial development of the Python 3 language specification, the
> core ``bytes`` type for arbitrary binary data started as the mutable type
> that is now referred to as ``bytearray``. Other aspects of operating in
> the binary domain in Python have also evolved over the course of the
Python
> 3 series.
>
> This PEP proposes five small adjustments to the APIs of the ``bytes`` and
> ``bytearray`` types to make it easier to operate entirely in the binary
> domain:
>
> * Deprecate passing single integer values to ``bytes`` and ``bytearray``
> * Add ``bytes.fromsize`` and ``bytearray.fromsize`` alternative
constructors
> * Add ``bytes.fromord`` and ``bytearray.fromord`` alternative constructors
> * Add ``bytes.getbyte`` and ``bytearray.getbyte`` byte retrieval methods
> * Add ``bytes.iterbytes`` and ``bytearray.iterbytes`` alternative
iterators

I wonder if from_something with an underscore is more consistent (according
to a quick search perhaps yes).

What about bytes.getchar and iterchars? A 'byte' in python 3 seems to be an
integer. (I would still like a .chars property that gives a sequence view
with __getitem__ and __len__ so that the getchar and iterchars methods are
not needed)

chrb seems to be more in line with some bytes versions in for instance os
than bchr.

Do we really need chrb? Better to introduce from_int or from_ord also in
str and recommend that over chr?

-- Koos (mobile)

>
> Proposals
> =
>
> Deprecation of current "zero-initialised sequence" behaviour without
removal
>

>
> Currently, the ``bytes`` and ``bytearray`` constructors accept an integer
> argument and interpret it as meaning to create a zero-initialised sequence
> of the given size::
>
> >>> bytes(3)
> b'\x00\x00\x00'
> >>> bytearray(3)
> bytearray(b'\x00\x00\x00')
>
> This PEP proposes to deprecate that behaviour in Python 3.6, but to leave
> it in place for at least as long as Python 2.7 is supported, possibly
> indefinitely.
>
> No other changes are proposed to the existing constructors.
>
>
> Addition of explicit "count and byte initialised sequence" constructors
> ---
>
> To replace the deprecated behaviour, this PEP proposes the addition of an
> explicit ``fromsize`` alternative constructor as a class method on both
> ``bytes`` and ``bytearray`` whose first argument is the count, and whose
> second argument is the fill byte to use (defaults to ``\x00``)::
>
> >>> bytes.fromsize(3)
> b'\x00\x00\x00'
> >>> bytearray.fromsize(3)
> bytearray(b'\x00\x00\x00')
> >>> bytes.fromsize(5, b'\x0a')
> b'\x0a\x0a\x0a\x0a\x0a'
> >>> bytearray.fromsize(5, b'\x0a')
> bytearray(b'\x0a\x0a\x0a\x0a\x0a')
>
> ``fromsize`` will behave just as the current constructors behave when
passed
> a single
> integer, while allowing for non-zero fill values when needed.
>
>
> Addition of "bchr" function and explicit "single byte" constructors
> ---
>
> As binary counterparts to the text ``chr`` function, this PEP proposes
> the addition of a ``bchr`` function and an explicit ``fromord``
alternative
> constructor as a class method on both ``bytes`` and ``bytearray``::
>
> >>> bchr(ord("A"))
> b'A'
> >>> bchr(ord(b"A"))
> b'A'
> >>> bytes.fromord(65)
> b'A'
> >>> bytearray.fromord(65)
> bytearray(b'A')
>
> These methods will only accept integers in the range 0 to 255
(inclusive)::
>
> >>> bytes.fromord(512)
> Traceback (most recent call last):
> File "", line 1, in 
> ValueError: integer must be in range(0, 256)
>
> >>> bytes.fromord(1.0)
> Traceback (most recent call last):
> File "", line 1, in 
> TypeError: 'float' object cannot be interpreted as an integer
>
> While this does create some duplication, there are valid reasons for it::
>
> * the ``bchr`` builtin is to recreate the ord/chr/unichr trio from Python
> 2 under a different naming scheme
> * the class method is mainly for the ``bytearray.fromord`` case, with
> ``bytes.fromord`` added for consistency
>
> The documentation of the ``ord`` builtin will be updated to explicitly
note
> that ``bchr`` is the primary inverse operation for binary data, while
> ``chr``
> is the inverse 

Re: [Python-Dev] Need help in debugging the python core

[Burkhard Meier]

How could I help?

Burkhard

On Fri, Sep 2, 2016 at 10:47 AM, Victor Stinner 
wrote:

> Oh, I forgot to mention that it would help to get the Python traceback
> on the crash. Try faulthandler: add faulthandler.enable() at the
> beginning of your program.
> https://docs.python.org/dev/library/faulthandler.html
>
> Maybe I should write once tools to debug such bug :-)
>
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> burkhardameier%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Ivan Levkivskyi]

On 2 September 2016 at 17:59, Guido van Rossum  wrote:
> On Fri, Sep 2, 2016 at 6:47 AM, Mark Shannon  wrote:
> > Quoting from the PEP:
> > ```
> > a: int
> > a: str # Static type checker will warn about this.
> > ```
> > In other words, it is illegal for a checker to split up the variable,
even
> > though it is straightforward to do so.
>
> One of my co-authors has gone too far here. The intent is not to
legislate what should happen in this case but to leave it to the checker.
In mypy, the equivalent syntax using type comments is currently indeed
rejected, but we're considering a change here (
https://github.com/python/mypy/issues/1174). The PEP 526 syntax will not
make a difference here.

If I remember correctly, I added this example. At that time the intention
was to propose to "loosen" the behaviour of type checkers (note it is a
warning, not an error like in mypy). But now I agree with Guido that we
should be even more liberal. We could left this to type checker to decide
what to do (they could even have options like -Werror or -Wignore).

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Steve Dower]

"I'm not seeing what distinction you think you are making here. What 
distinction do you see between:

x: int = func(value)

and

x = func(value)  #type: int"

Not sure whether I agree with Mark on this particular point, but the difference 
I see here is that the first describes what types x may ever contain, while the 
latter describes what type of being assigned to x right here. So one is a 
variable annotation while the other is an expression annotation.

Personally, I prefer expression annotations over variable annotations, as there 
are many other languages I'd prefer if variable have fixed types (e.g. C++, 
where I actually enjoy doing horrible things with implicit casting ;) ).

Variable annotations appear to be inherently restrictive, so either we need 
serious clarification as to why they are not, or they actually are and we ought 
to be more sure that it's the direction we want the language to go.

Cheers,
Steve

Top-posted from my Windows Phone

-Original Message-
From: "Steven D'Aprano" 
Sent: ‎9/‎2/‎2016 9:43
To: "python-dev@python.org" 
Subject: Re: [Python-Dev] Please reject or postpone PEP 526

Hi Mark,

I'm going to trim your post drastically, down to the bare essentials, in 
order to keep this already long post down to a manageable size.


On Fri, Sep 02, 2016 at 02:47:00PM +0100, Mark Shannon wrote:

[...]
> With type comments, this is intuitively correct and should type check:
> def eggs(cond:bool):
> if cond:
> x = None
> else:
> x = [] # type: List[int]
> spam(x)  # Here we can infer the type of x

It isn't correct. You've declared something to be a list of ints, but 
assigned a value None to it! How is that not an error?

The error is more obvious if you swap the order of assignments:

if cond:
x = [] # type: List[int]
else:
x = None


MyPy currently requires the experimental --strict-optional flag to 
detect this error:

[steve@ando ~]$ mypy --strict-optional test.py
test.py: note: In function "eggs":
test.py:10: error: Incompatible types in assignment (expression has type 
None, variable has type List[int])


Changing that from comment syntax to (proposed) Python syntax will not 
change that. There is no semantic difference to the type checker 
between 

x = []  #type: List[int]

and

x: List[int] = []

and any type checker will have to treat them identically.


> With PEP 526 we loose the ability to infer types.

On re-reading the PEP, I have just noticed that nowhere does it 
explicitly state that checkers are still expected to perform type 
inference. However, this is the companion to PEP 484, which states:

Type comments

No first-class syntax support for explicitly marking variables
as being of a specific type is added by this PEP. TO HELP WITH 
TYPE INFERENCE IN COMPLEX CASES, a comment of the following
format may be used: [...]

(Emphasis added.) So the intention is that, regardless of whether you 
use a type annotation using a comment or the proposed syntax, that is 
intended to *help* the checker perform inference, not to replace it.

Perhaps this PEP should include an explicit note to that end.



[...]
> So we need to use a more complex type
> def eggs(cond:bool):
> x: Optional[List[int]]
> if cond:
> x = None # Now legal
> else:
> x: = []
> spam(x)
> 
> I don't think this improves readability.

Maybe not, but it sure improves *correctness*.

A human reader might be able to logically deduce that x = None and 
x = [] are both valid, given that spam() takes either a list or None, 
but I'm not sure if that level of intelligence is currently possible in 
type inference. (Maybe it is, and MyPy simply doesn't support it yet.) 
So it may be that this is a case where you do have to apply an explicit 
type hint, using *either* a type comment or this new proposed syntax:

x: Optional[List[int]]
if cond:
x = None
else:
x = []

should be precisely the same as:

if cond:
x = None #type: Optional[List[int]]
else:
x = []


> Quoting from the PEP:
> ```
> a: int
> a: str # Static type checker will warn about this.
> ```
> In other words, it is illegal for a checker to split up the variable, 
> even though it is straightforward to do so.

No, it is a *warning*, not an error.

Remember, the Python interpreter itself won't care. The type checker is 
optional and not part of the interpreter. You can still write code like:

a = 1
do_something(a)
a = "string"
do_another(a)

and Python will be happy. But if you do run a type checker, it should 
warn you that you're changing types, as that suggests the possibility of 
a type error. (And it also goes against a lot of people's style 
guidelines.)


> We should be free to add extra variables, whenever we choose, for 
> clarity. For example,
> total = foo() - bar()
> should not be treated differently from:
> revenue = 

Re: [Python-Dev] Need help in debugging the python core

[Victor Stinner]

Oh, I forgot to mention that it would help to get the Python traceback
on the crash. Try faulthandler: add faulthandler.enable() at the
beginning of your program.
https://docs.python.org/dev/library/faulthandler.html

Maybe I should write once tools to debug such bug :-)

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Need help in debugging the python core

[Victor Stinner]

2016-09-02 8:49 GMT+02:00 Sajjanshetty, Amresh :
> I’m using asyncio and paramiko to multiplex different channels into a single
> SSH connection.

Hum, asyncio found bugs in CPython. Please try with a more recent
version of CPython than 3.4.3 :-/

> Program terminated with signal 11, Segmentation fault.
>
> #0  _PyObject_Malloc (ctx=0x0, nbytes=52) at Objects/obmalloc.c:1159

Hum, a crash on a memory allocation is usually a buffer overflow.

Please retry with Python 3.6 using PYTHONMALLOC=debug:
https://docs.python.org/dev/using/cmdline.html#envvar-PYTHONMALLOC

Calling regulary gc.collect() may help PYTHONMALLOC=debug to detect
buffer overflows earlier.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

I agree about some concerns and disagree about several. I see most use
for class/instance attribute annotations, both for type checking and
for other uses. I'm least sure about their syntax and about
annotations in functions or at module level in the proposed form.

Below some comments:

On Fri, Sep 2, 2016 at 4:47 PM, Mark Shannon  wrote:
> Hi everyone,
>
> I think we should reject, or at least postpone PEP 526.
>
> PEP 526 represents a major change to the language, however there are, I
> believe, a number of technical flaws with the PEP.

[...]
>
> In many cases it makes it more effort than type comments
> 
>
> Type hints should be as easy to use as possible, and that means pushing as
> much work as possible onto the checker, and not burdening the programmer.
>
> Attaching type hints to variables, rather than expressions, reduces the
> potential for inference. This makes it harder for programmer, but easier for
> the checker, which is the wrong way around.

The more you infer types, the less you check them. It's up to the
programmers to choose the amount of annotation.

> For example,, given a function:
> def spam(x: Optional[List[int]])->None: ...
>
> With type comments, this is intuitively correct and should type check:
> def eggs(cond:bool):
> if cond:
> x = None
> else:
> x = [] # type: List[int]
> spam(x)  # Here we can infer the type of x
>
> With PEP 526 we loose the ability to infer types.
> def eggs(cond:bool):
> if cond:
> x = None # Not legal due to type declaration below
> else:
>x: List[int] = []
> spam(x)

I'm also a little worried about not being able to reannotate a name.

>
> So we need to use a more complex type
> def eggs(cond:bool):
> x: Optional[List[int]]
> if cond:
> x = None # Now legal
> else:
> x: = []
> spam(x)
>

A good checker should be able to infer that x is a union type at the
point that it's passed to spam, even without the type annotation. For
example:

def eggs(cond:bool):
if cond:
x = 1
else:
x = 1.5
spam(x)   # a good type checker infers that x is of type Union[int, float]

Or with annotations:

def eggs(cond:bool):
if cond:
x : int = foo() # foo may not have a return type hint
else:
x : float = bar() # bar may not have a return type hint
spam(x)   # a good type checker infers that x is of type Union[int, float]


[...]
> It limits the use of variables
> ==
>
> In Python a name (variable) is just a binding that refers to an object.
> A name only exists in a meaningful sense once an object has been assigned to
> it. Any attempt to use that name, without an object bound to it, will result
> in a NameError.
>

IIUC, that would still be the case after PEP 526.

[...]
>
> We should be free to add extra variables, whenever we choose, for clarity.
> For example,
> total = foo() - bar()
> should not be treated differently from:
> revenue = foo()
> tax = bar()
> total = revenue - tax
>
> If types are inferred, there is no problem.
> However, if they must be declared, then the use of meaningfully named
> variables is discouraged.
>

Who says they *must* be declared?

[...]
> It is premature
> ===
>
> There are still plenty of issues to iron out w.r.t. PEP 484 types. I don't
> think we should be adding more syntax, until we have a *precise* idea of
> what is required.
>
> PEP 484 states:
> "If type hinting proves useful in general, a syntax for typing variables may
> be provided in a future Python version."
> Has it proved useful in general? I don't think it has. Maybe it will in
> future, but it hasn't yet.
>

Yeah, I hope someone has enough experience to know whether this is the
right thing for Python as a whole.

> It seems confused about class attributes and instance attributes
> 
>
> The PEP also includes a section of how to define class attributes and
> instance attributes. It seems that everything needs to be defined in the
> class scope, even it is not an attribute of the class, but of its instances.
> This seems confusing, both to human reader and machine analyser.

I don't see the problem here, isn't that how it's usually done in
strongly typed languages? And methods are defined in the class scope
too (well yes, they do also exist in the class namespace, but
anyway...).

But I agree in the sense that the proposed syntax is far from explicit
about these being instance attributes by default.

> Example from PEP 526:
>
> class Starship:
>
> captain: str = 'Picard'
> damage: int
> stats: ClassVar[Dict[str, int]] = {}
>
> def __init__(self, damage: int, captain: str = None):
> self.damage = damage
> if captain:
> self.captain = captain  # Else keep the default
>
> With type hints as they currently 

Re: [Python-Dev] Please reject or postpone PEP 526

[Sven R. Kunze]

Hi Mark,

I agree with you about postponing.

Not so much because of the issues you mentioned. Those all seem 
resolvable to me and mostly concerns type checkers, linters and coding 
styles not Python itself. However, I also don't like the rushing through 
as if this beta were the only chance to get it into Python.


Python haven't had variable annotations for decades until now and Python 
will still be there in 10 decades or so. Thus, 2 years more to wait and 
to hammer out the details does not seem much compared to the entire 
lifetime of this language.


The PEP also remains silent about when to use annotations. Compared to 
recent additions like f-strings or async, it's completely clear when to 
use it. However for variable annotations it's not (all variables? most 
used variables? least used variables?). So, I also agree with you that 
improving type checkers is the better way than adding all static type 
annotations all over the place. Python is dynamic and also types should 
be as dynamic and redundant-free as possible. Thus, some guidance would 
be great here.


Cheers,
Sven


On 02.09.2016 15:47, Mark Shannon wrote:

Hi everyone,

I think we should reject, or at least postpone PEP 526.

PEP 526 represents a major change to the language, however there are, 
I believe, a number of technical flaws with the PEP.


It is probable that with significant revisions it can be a worthwhile 
addition to the language, but that cannot happen in time for 3.6 beta 
1 (in 11 days).


PEP 526 claims to be an extension of PEP 484, but I don't think that 
is entirely correct.
PEP 484 was primarily about defining the semantics of pre-existing 
syntax. PEP 526 is about adding new syntax.
Historically the bar for adding new syntax has been set very high. I 
don't think that PEP 526, in its current form, reaches that bar.


Below is a list of the issues I have with the PEP as it stands.

In many cases it makes it more effort than type comments


Type hints should be as easy to use as possible, and that means 
pushing as much work as possible onto the checker, and not burdening 
the programmer.


Attaching type hints to variables, rather than expressions, reduces 
the potential for inference. This makes it harder for programmer, but 
easier for the checker, which is the wrong way around.


For example,, given a function:
def spam(x: Optional[List[int]])->None: ...

With type comments, this is intuitively correct and should type check:
def eggs(cond:bool):
if cond:
x = None
else:
x = [] # type: List[int]
spam(x)  # Here we can infer the type of x

With PEP 526 we loose the ability to infer types.
def eggs(cond:bool):
if cond:
x = None # Not legal due to type declaration below
else:
   x: List[int] = []
spam(x)

So we need to use a more complex type
def eggs(cond:bool):
x: Optional[List[int]]
if cond:
x = None # Now legal
else:
x: = []
spam(x)

I don't think this improves readability.
Whether this is an acceptable change is debatable, but it does need 
some debate.


It limits the use of variables
==

In Python a name (variable) is just a binding that refers to an object.
A name only exists in a meaningful sense once an object has been 
assigned to it. Any attempt to use that name, without an object bound 
to it, will result in a NameError.


PEP 526 makes variables more than just bindings, as any rebinding must 
conform to the given type. This looses us some of the dynamism for 
which we all love Python.


Quoting from the PEP:
```
a: int
a: str # Static type checker will warn about this.
```
In other words, it is illegal for a checker to split up the variable, 
even though it is straightforward to do so.


However, without the type declarations,
```
a = 1
a = "Hi"
```
is just fine. Useless, but fine.

We should be free to add extra variables, whenever we choose, for 
clarity. For example,

total = foo() - bar()
should not be treated differently from:
revenue = foo()
tax = bar()
total = revenue - tax

If types are inferred, there is no problem.
However, if they must be declared, then the use of meaningfully named 
variables is discouraged.


[A note about type-inference:
Type inference is not a universal panacea, but it can make life a lot 
easier for programmers in statically type languages.
Languages like C# use local type inference extensively and it means 
that many variables often do not need their type declared. We should 
take care not to limit the ability of checkers to infer values and 
types and make programmers' lives easier.
Within a function, type inference is near perfect, failing only 
occasionally for some generic types.
One place where type inference definitely breaks down is across calls, 
which is why PEP 484 is necessary.

]

It is premature
===

There are still plenty of issues to iron out w.r.t. PEP 484 

Re: [Python-Dev] Please reject or postpone PEP 526

[Steven D'Aprano]

Hi Mark,

I'm going to trim your post drastically, down to the bare essentials, in 
order to keep this already long post down to a manageable size.


On Fri, Sep 02, 2016 at 02:47:00PM +0100, Mark Shannon wrote:

[...]
> With type comments, this is intuitively correct and should type check:
> def eggs(cond:bool):
> if cond:
> x = None
> else:
> x = [] # type: List[int]
> spam(x)  # Here we can infer the type of x

It isn't correct. You've declared something to be a list of ints, but 
assigned a value None to it! How is that not an error?

The error is more obvious if you swap the order of assignments:

if cond:
x = [] # type: List[int]
else:
x = None


MyPy currently requires the experimental --strict-optional flag to 
detect this error:

[steve@ando ~]$ mypy --strict-optional test.py
test.py: note: In function "eggs":
test.py:10: error: Incompatible types in assignment (expression has type 
None, variable has type List[int])


Changing that from comment syntax to (proposed) Python syntax will not 
change that. There is no semantic difference to the type checker 
between 

x = []  #type: List[int]

and

x: List[int] = []

and any type checker will have to treat them identically.


> With PEP 526 we loose the ability to infer types.

On re-reading the PEP, I have just noticed that nowhere does it 
explicitly state that checkers are still expected to perform type 
inference. However, this is the companion to PEP 484, which states:

Type comments

No first-class syntax support for explicitly marking variables
as being of a specific type is added by this PEP. TO HELP WITH 
TYPE INFERENCE IN COMPLEX CASES, a comment of the following
format may be used: [...]

(Emphasis added.) So the intention is that, regardless of whether you 
use a type annotation using a comment or the proposed syntax, that is 
intended to *help* the checker perform inference, not to replace it.

Perhaps this PEP should include an explicit note to that end.



[...]
> So we need to use a more complex type
> def eggs(cond:bool):
> x: Optional[List[int]]
> if cond:
> x = None # Now legal
> else:
> x: = []
> spam(x)
> 
> I don't think this improves readability.

Maybe not, but it sure improves *correctness*.

A human reader might be able to logically deduce that x = None and 
x = [] are both valid, given that spam() takes either a list or None, 
but I'm not sure if that level of intelligence is currently possible in 
type inference. (Maybe it is, and MyPy simply doesn't support it yet.) 
So it may be that this is a case where you do have to apply an explicit 
type hint, using *either* a type comment or this new proposed syntax:

x: Optional[List[int]]
if cond:
x = None
else:
x = []

should be precisely the same as:

if cond:
x = None #type: Optional[List[int]]
else:
x = []


> Quoting from the PEP:
> ```
> a: int
> a: str # Static type checker will warn about this.
> ```
> In other words, it is illegal for a checker to split up the variable, 
> even though it is straightforward to do so.

No, it is a *warning*, not an error.

Remember, the Python interpreter itself won't care. The type checker is 
optional and not part of the interpreter. You can still write code like:

a = 1
do_something(a)
a = "string"
do_another(a)

and Python will be happy. But if you do run a type checker, it should 
warn you that you're changing types, as that suggests the possibility of 
a type error. (And it also goes against a lot of people's style 
guidelines.)


> We should be free to add extra variables, whenever we choose, for 
> clarity. For example,
> total = foo() - bar()
> should not be treated differently from:
> revenue = foo()
> tax = bar()
> total = revenue - tax
> 
> If types are inferred, there is no problem.
> However, if they must be declared, then the use of meaningfully named 
> variables is discouraged.

Was there something in the PEP that lead you to believe that they "must" 
be declared? Under "Non-goals", the PEP states in bold text:

"the authors have no desire to ever make type hints mandatory"

so I'm not sure why you think that types must be declared.

Perhaps the PEP should make it more obvious that type hints on variables 
are *in addition to* and not a substitute for type inference.


> PEP 484 states:
> "If type hinting proves useful in general, a syntax for typing variables 
> may be provided in a future Python version."
> Has it proved useful in general? I don't think it has.

According to the PEP, it has proved useful in typeshed.


> It seems confused about class attributes and instance attributes
> 
> 
> The PEP also includes a section of how to define class attributes and 
> instance attributes. It seems that everything needs to be defined in the 
> class scope, even it is not an attribute of 

Re: [Python-Dev] PEP 526 ready for review: Syntax for Variable and Attribute Annotations

[Guido van Rossum]

On Fri, Sep 2, 2016 at 6:43 AM, Ivan Levkivskyi  wrote:
> On 2 September 2016 at 04:38, Nick Coghlan  wrote:
>> However, a standalone Ellipsis doesn't currently have a meaning as a
>> type annotation (it's only meaningful when subscripting Tuple and
>> Callable), so a spelling like this might work:
>>
>> NAME: ...
>>
>> That spelling could then also be used in function definitions to say
>> "infer the return type from the return statements rather than assuming
>> Any"
>
> Interesting idea.
> This is somehow similar to one of the existing use of Ellipsis: in numpy it
> infers how many dimensions needs to have the full slice, it is like saying
> "You know what I mean". So I am +1 on this solution.

I like it too, but I think it's better to leave any firm promises
about the *semantics* of variable annotations out of the PEP. I just
spoke to someone who noted that the PEP is likely to evoke an outsize
emotional response. (Similar to what happened with PEP 484.)

Pinning down the semantics is not why I am pushing for PEP 526 -- I
only want to pin down the *syntax* to the point where we won't have to
change it again for many versions, since it's much harder to change
the syntax than it is to change the behavior of type checkers (which
have fewer backwards compatibility constraints, a faster release
cycle, and narrower user bases than core Python itself).

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Summary of Python tracker Issues

[Python tracker]

ACTIVITY SUMMARY (2016-08-26 - 2016-09-02)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open5630 (+24)
  closed 34067 (+49)
  total  39697 (+73)

Open issues with patches: 2441 


Issues opened (52)
==

#27868: Unconditionally state when a build succeeds
http://bugs.python.org/issue27868  opened by brett.cannon

#27869: test failures on Bash on Windows
http://bugs.python.org/issue27869  opened by brett.cannon

#27872: Update os/os.path docs to mention path-like object support
http://bugs.python.org/issue27872  opened by brett.cannon

#27873: multiprocessing.pool.Pool.map should take more than one iterab
http://bugs.python.org/issue27873  opened by Jason Yu

#27874: inconsistent sys.path behavior when using PythonXX.zip
http://bugs.python.org/issue27874  opened by Joseph.Shen

#27875: Syslogs /usr/sbin/foo as /foo instead of as foo
http://bugs.python.org/issue27875  opened by canvon

#27876: Add SSLContext.set_version_range(minver, maxver=None)
http://bugs.python.org/issue27876  opened by christian.heimes

#27877: Add recipe for "valueless" Enums to docs
http://bugs.python.org/issue27877  opened by John Hagen

#27879: add os.syncfs()
http://bugs.python.org/issue27879  opened by mmarkk

#27880: cPickle fails on large objects (still - 2011 and counting)
http://bugs.python.org/issue27880  opened by rob...@smithpierce.net

#27881: Fix possible bugs when setting sqlite3.Connection.isolation_le
http://bugs.python.org/issue27881  opened by xiang.zhang

#27883: sqlite-3.14.1 for Python_3.6.0b1 ?
http://bugs.python.org/issue27883  opened by Big Stone

#27884: during 'make install', pre-existing site-packages residents ar
http://bugs.python.org/issue27884  opened by MattDMo

#27886: Docs: the difference between rename and replace is not obvious
http://bugs.python.org/issue27886  opened by asvetlov

#27889: ctypes interfers with signal handling
http://bugs.python.org/issue27889  opened by Andre Merzky

#27890: platform.release() incorrect in Python 3.5.2 on Windows 2008Se
http://bugs.python.org/issue27890  opened by James Domingo

#27892: Idlelib: document or move delayed imports
http://bugs.python.org/issue27892  opened by terry.reedy

#27896: Allow passing sphinx options to Doc/Makefile
http://bugs.python.org/issue27896  opened by sizeof

#27897: Avoid possible crash in pysqlite_connection_create_collation
http://bugs.python.org/issue27897  opened by xiang.zhang

#27898: regexp performance degradation between 2.7.6 and 2.7.12
http://bugs.python.org/issue27898  opened by steve.newcomb

#27900: ctypes fails to find ncurses via ncursesw on Arch Linux
http://bugs.python.org/issue27900  opened by blueyed

#27901: inspect.ismethod returns different results on the same basic c
http://bugs.python.org/issue27901  opened by anthony-flury

#27902: pstats.Stats: strip_dirs() method cannot handle file paths fro
http://bugs.python.org/issue27902  opened by Jaroslav

#27903: Avoid ResourceWarnings from platform._dist_try_harder
http://bugs.python.org/issue27903  opened by scop

#27905: Add documentation for typing.Type
http://bugs.python.org/issue27905  opened by michael0x2a

#27906: Socket accept exhaustion during high TCP traffic
http://bugs.python.org/issue27906  opened by kevinconway

#27908: del _limbo[self] KeyError
http://bugs.python.org/issue27908  opened by Dima.Tisnek

#27910: Doc/library/traceback.rst — references to tuples should be r
http://bugs.python.org/issue27910  opened by torsava

#27911: Unnecessary error checks in exec_builtin_or_dynamic
http://bugs.python.org/issue27911  opened by xiang.zhang

#27914: Incorrect comment in PyModule_ExcDef
http://bugs.python.org/issue27914  opened by xiang.zhang

#27915: Use 'ascii' instead of 'us-ascii' to bypass lookup machinery
http://bugs.python.org/issue27915  opened by scop

#27916: Use time.monotonic instead of time.time where appropriate
http://bugs.python.org/issue27916  opened by scop

#27918: Running test suites without gui but still having windows flash
http://bugs.python.org/issue27918  opened by xiang.zhang

#27919: Deprecate and remove extra_path distribution kwarg
http://bugs.python.org/issue27919  opened by jason.coombs

#27920: Embedding python in a shared library fails to import the Pytho
http://bugs.python.org/issue27920  opened by suzaku

#27921: f-strings: do not allow backslashes
http://bugs.python.org/issue27921  opened by eric.smith

#27923: PEP 467 -- Minor API improvements for binary sequences
http://bugs.python.org/issue27923  opened by elias

#27925: Nicer interface to convert hashlib digests to int
http://bugs.python.org/issue27925  opened by steven.daprano

#27926: ctypes is too slow to convert a Python list to a C array
http://bugs.python.org/issue27926  opened by Tom Cornebize

#27927: argparse: default propagation of formatter_class from Argument
http://bugs.python.org/issue27927  opened by 

Re: [Python-Dev] Please reject or postpone PEP 526

[Guido van Rossum]

On Fri, Sep 2, 2016 at 6:47 AM, Mark Shannon  wrote:
> Hi everyone,
>
> I think we should reject, or at least postpone PEP 526.
>
> PEP 526 represents a major change to the language, however there are, I
> believe, a number of technical flaws with the PEP.
>
> It is probable that with significant revisions it can be a worthwhile
> addition to the language, but that cannot happen in time for 3.6 beta 1
(in
> 11 days).
>
> PEP 526 claims to be an extension of PEP 484, but I don't think that is
> entirely correct.
> PEP 484 was primarily about defining the semantics of pre-existing syntax.
> PEP 526 is about adding new syntax.
> Historically the bar for adding new syntax has been set very high. I don't
> think that PEP 526, in its current form, reaches that bar.
>
> Below is a list of the issues I have with the PEP as it stands.
>
> In many cases it makes it more effort than type comments
> 
>
> Type hints should be as easy to use as possible, and that means pushing as
> much work as possible onto the checker, and not burdening the programmer.
>
> Attaching type hints to variables, rather than expressions, reduces the
> potential for inference. This makes it harder for programmer, but easier
for
> the checker, which is the wrong way around.
>
> For example,, given a function:
> def spam(x: Optional[List[int]])->None: ...
>
> With type comments, this is intuitively correct and should type check:
> def eggs(cond:bool):
> if cond:
> x = None
> else:
> x = [] # type: List[int]
> spam(x) # Here we can infer the type of x
>
> With PEP 526 we loose the ability to infer types.
> def eggs(cond:bool):
> if cond:
> x = None # Not legal due to type declaration below
> else:
> x: List[int] = []
> spam(x)
>
> So we need to use a more complex type
> def eggs(cond:bool):
> x: Optional[List[int]]
> if cond:
> x = None # Now legal
> else:
> x: = []
> spam(x)
>
> I don't think this improves readability.
> Whether this is an acceptable change is debatable, but it does need some
> debate.

It looks like you're misinterpreting the intent of the PEP. It does not
mean legislate the behavior of the type checker in this way. In mypy, the
first example is already rejected because it wants the annotation on the
first occurrence. The plan is for mypy not to change its behavior -- the
old form

TARGET = VALUE # type: TYPE

will be treated the same way as the new form

TARGET: TYPE = VALUE

(If you have a beef with what this means in mypy you should probably take
it up with mypy, not with PEP 526.)

> It limits the use of variables
> ==
>
> In Python a name (variable) is just a binding that refers to an object.
> A name only exists in a meaningful sense once an object has been assigned
to
> it. Any attempt to use that name, without an object bound to it, will
result
> in a NameError.

(Or UnboundLocalError, if the compiler knows there is an assignment to the
name anywhere in the same (function) scope.)

> PEP 526 makes variables more than just bindings, as any rebinding must
> conform to the given type. This looses us some of the dynamism for which
we
> all love Python.

Thanks for catching this; that's not the intent.

> Quoting from the PEP:
> ```
> a: int
> a: str # Static type checker will warn about this.
> ```
> In other words, it is illegal for a checker to split up the variable, even
> though it is straightforward to do so.

One of my co-authors has gone too far here. The intent is not to legislate
what should happen in this case but to leave it to the checker. In mypy,
the equivalent syntax using type comments is currently indeed rejected, but
we're considering a change here (https://github.com/python/mypy/issues/1174).
The PEP 526 syntax will not make a difference here.

> However, without the type declarations,
> ```
> a = 1
> a = "Hi"
> ```
> is just fine. Useless, but fine.

And believe me, I want to keep it this way. I will amend the example and
clarify the intent in the text.

> We should be free to add extra variables, whenever we choose, for clarity.
> For example,
> total = foo() - bar()
> should not be treated differently from:
> revenue = foo()
> tax = bar()
> total = revenue - tax
>
> If types are inferred, there is no problem.
> However, if they must be declared, then the use of meaningfully named
> variables is discouraged.

There is no mandate to declare variables! I actually see the main use of
variable annotations in class bodies where it can crate a lot of clarity
around which instance variables exist and what they mean.

> [A note about type-inference:
> Type inference is not a universal panacea, but it can make life a lot
easier
> for programmers in statically type languages.
> Languages like C# use local type inference extensively and it means that
> many variables often do not need their type declared. We should take care
> not to limit the ability of checkers to infer values and types and make
> programmers' 

[Python-Dev] Need help in debugging the python core

[Sajjanshetty, Amresh]

Dear All,

I’m using asyncio and paramiko to multiplex different channels into a single 
SSH connection. Things were working fine till recently but suddenly started 
seeing that python getting crashed whenever I tried to write to the channel. I 
have very limited knowledge on how python interpreter works, so I’m finding 
difficulty in understanding the stack trace. Can you please help in 
understanding the below backtarce.

bash-4.2$ gdb /usr/software/bin/python3.4.3 core.60015
Traceback (most recent call last):
  File "", line 70, in 
  File "", line 67, in GdbSetPythonDirectory
  File "/usr/software/share/gdb/python/gdb/__init__.py", line 19, in 
import _gdb
ImportError: No module named _gdb
GNU gdb (GDB) 7.5
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/software/bin/python3.4.3...done.

warning: core file may not match specified executable file.
[New LWP 60015]
[New LWP 60018]
[New LWP 60019]
[New LWP 60020]
[New LWP 60021]
[New LWP 60022]
[New LWP 60023]
[New LWP 60024]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/software/lib/libthread_db.so.1".
Core was generated by `/usr/software/bin/python3.4.3 
/x/eng/bbrtp/users/amresh/sshproxy_3896926_160824'.
Program terminated with signal 11, Segmentation fault.
#0  _PyObject_Malloc (ctx=0x0, nbytes=52) at Objects/obmalloc.c:1159
1159Objects/obmalloc.c: No such file or directory.
(gdb) bt
#0  _PyObject_Malloc (ctx=0x0, nbytes=52) at Objects/obmalloc.c:1159
#1  0x7ff2e511474a in PyUnicode_New (maxchar=, size=3) at 
Objects/unicodeobject.c:1093
#2  PyUnicode_New (size=3, maxchar=) at 
Objects/unicodeobject.c:1033
#3  0x7ff2e5139da2 in _PyUnicodeWriter_PrepareInternal 
(writer=writer@entry=0x7fff3d5c8640, length=, maxchar=, maxchar@entry=127) at Objects/unicodeobject.c:13327
#4  0x7ff2e513f38b in PyUnicode_DecodeUTF8Stateful 
(s=s@entry=0x7ff2e3572f78 
"tcp\245reuse\001\253socket_type\244pull\251transport\246zeromq", 
size=size@entry=3,
errors=errors@entry=0x7ff2dee5dd70 "strict", consumed=consumed@entry=0x0) 
at Objects/unicodeobject.c:4757
#5  0x7ff2e5140690 in PyUnicode_Decode (s=0x7ff2e3572f78 
"tcp\245reuse\001\253socket_type\244pull\251transport\246zeromq", size=3, 
encoding=0x7ff2dee5df28 "utf-8", errors=0x7ff2dee5dd70 "strict")
at Objects/unicodeobject.c:3012
#6  0x7ff2de49bfdf in unpack_callback_raw (o=, l=3, 
p=0x7ff2e3572f78 
"tcp\245reuse\001\253socket_type\244pull\251transport\246zeromq", 
u=0x7fff3d5c8840, b=)
at msgpack/unpack.h:229
#7  unpack_execute (ctx=ctx@entry=0x7fff3d5c8840, data=0x7ff2e3572ec0 
"\205\245_auth\300\245_call\246expect\243_i", , 
len=, off=off@entry=0x7fff3d5c8820)
at msgpack/unpack_template.h:312
#8  0x7ff2de49fe3d in __pyx_pf_7msgpack_9_unpacker_2unpackb 
(__pyx_v_packed=__pyx_v_packed@entry=0x7ff2e3572ea0, 
__pyx_v_object_hook=__pyx_v_object_hook@entry=0x7ff2e54934b0 <_Py_NoneStruct>,
__pyx_v_list_hook=__pyx_v_list_hook@entry=0x7ff2e54934b0 <_Py_NoneStruct>, 
__pyx_v_use_list=1, __pyx_v_encoding=0x7ff2dee5df08, 
__pyx_v_unicode_errors=0x7ff2dee5dd50,
__pyx_v_object_pairs_hook=0x7ff2e54934b0 <_Py_NoneStruct>, 
__pyx_v_ext_hook=0x13db2d8, 
__pyx_v_max_str_len=__pyx_v_max_str_len@entry=2147483647,
__pyx_v_max_bin_len=__pyx_v_max_bin_len@entry=2147483647, 
__pyx_v_max_array_len=2147483647, __pyx_v_max_map_len=2147483647, 
__pyx_v_max_ext_len=__pyx_v_max_ext_len@entry=2147483647,
__pyx_self=) at msgpack/_unpacker.pyx:139
#9  0x7ff2de4a1395 in __pyx_pw_7msgpack_9_unpacker_3unpackb 
(__pyx_self=, __pyx_args=, __pyx_kwds=) at msgpack/_unpacker.pyx:102
#10 0x7ff2e5174ed3 in do_call (nk=, na=, 
pp_stack=0x7fff3d5d2b80, func=0x7ff2df20ddc8) at Python/ceval.c:4463
#11 call_function (oparg=, pp_stack=0x7fff3d5d2b80) at 
Python/ceval.c:4264
#12 PyEval_EvalFrameEx (f=f@entry=0x7ff2def02208, throwflag=throwflag@entry=0) 
at Python/ceval.c:2838
#13 0x7ff2e5175f45 in PyEval_EvalCodeEx (_co=, 
globals=, locals=locals@entry=0x0, args=, 
argcount=argcount@entry=1, kws=0x7ff2deefec30, kwcount=0, defs=0x0,
defcount=0, kwdefs=0x0, closure=0x0) at Python/ceval.c:3588
#14 0x7ff2e51734da in fast_function (nk=, na=1, n=, pp_stack=0x7fff3d5d2e10, func=0x7ff2dee9b7b8) at Python/ceval.c:4344
#15 call_function (oparg=, pp_stack=0x7fff3d5d2e10) at 
Python/ceval.c:4262
#16 PyEval_EvalFrameEx (f=f@entry=0x7ff2deefea98, throwflag=throwflag@entry=0) 
at Python/ceval.c:2838
#17 0x7ff2e5175f45 in PyEval_EvalCodeEx (_co=, 
globals=, locals=locals@entry=0x0, args=, 
argcount=argcount@entry=1, kws=0x14566c8, kwcount=0,

Re: [Python-Dev] Please reject or postpone PEP 526

[Ryan Gonzalez]

On Sep 2, 2016 8:51 AM, "Mark Shannon"  wrote:
>
> Hi everyone,
>
> I think we should reject, or at least postpone PEP 526.
>
> PEP 526 represents a major change to the language, however there are, I
believe, a number of technical flaws with the PEP.
>
> It is probable that with significant revisions it can be a worthwhile
addition to the language, but that cannot happen in time for 3.6 beta 1 (in
11 days).
>
> PEP 526 claims to be an extension of PEP 484, but I don't think that is
entirely correct.
> PEP 484 was primarily about defining the semantics of pre-existing
syntax. PEP 526 is about adding new syntax.
> Historically the bar for adding new syntax has been set very high. I
don't think that PEP 526, in its current form, reaches that bar.
>
> Below is a list of the issues I have with the PEP as it stands.
>
> In many cases it makes it more effort than type comments
> 
>
> Type hints should be as easy to use as possible, and that means pushing
as much work as possible onto the checker, and not burdening the programmer.
>
> Attaching type hints to variables, rather than expressions, reduces the
potential for inference. This makes it harder for programmer, but easier
for the checker, which is the wrong way around.
>
> For example,, given a function:
> def spam(x: Optional[List[int]])->None: ...
>
> With type comments, this is intuitively correct and should type check:
> def eggs(cond:bool):
> if cond:
> x = None
> else:
> x = [] # type: List[int]
> spam(x)  # Here we can infer the type of x
>
> With PEP 526 we loose the ability to infer types.
> def eggs(cond:bool):
> if cond:
> x = None # Not legal due to type declaration below
> else:
>x: List[int] = []
> spam(x)
>
> So we need to use a more complex type
> def eggs(cond:bool):
> x: Optional[List[int]]
> if cond:
> x = None # Now legal
> else:
> x: = []
> spam(x)
>
> I don't think this improves readability.
> Whether this is an acceptable change is debatable, but it does need some
debate.
>
> It limits the use of variables
> ==
>
> In Python a name (variable) is just a binding that refers to an object.
> A name only exists in a meaningful sense once an object has been assigned
to it. Any attempt to use that name, without an object bound to it, will
result in a NameError.
>
> PEP 526 makes variables more than just bindings, as any rebinding must
conform to the given type. This looses us some of the dynamism for which we
all love Python.
>
> Quoting from the PEP:
> ```
> a: int
> a: str # Static type checker will warn about this.
> ```
> In other words, it is illegal for a checker to split up the variable,
even though it is straightforward to do so.
>
> However, without the type declarations,
> ```
> a = 1
> a = "Hi"
> ```
> is just fine. Useless, but fine.
>

But isn't that the same way with type comments? Except uglier?

> We should be free to add extra variables, whenever we choose, for
clarity. For example,
> total = foo() - bar()
> should not be treated differently from:
> revenue = foo()
> tax = bar()
> total = revenue - tax
>
> If types are inferred, there is no problem.
> However, if they must be declared, then the use of meaningfully named
variables is discouraged.
>
> [A note about type-inference:
> Type inference is not a universal panacea, but it can make life a lot
easier for programmers in statically type languages.
> Languages like C# use local type inference extensively and it means that
many variables often do not need their type declared. We should take care
not to limit the ability of checkers to infer values and types and make
programmers' lives easier.
> Within a function, type inference is near perfect, failing only
occasionally for some generic types.
> One place where type inference definitely breaks down is across calls,
which is why PEP 484 is necessary.
> ]
>
> It is premature
> ===
>
> There are still plenty of issues to iron out w.r.t. PEP 484 types. I
don't think we should be adding more syntax, until we have a *precise* idea
of what is required.
>
> PEP 484 states:
> "If type hinting proves useful in general, a syntax for typing variables
may be provided in a future Python version."
> Has it proved useful in general? I don't think it has. Maybe it will in
future, but it hasn't yet.
>
> It seems confused about class attributes and instance attributes
> 
>
> The PEP also includes a section of how to define class attributes and
instance attributes. It seems that everything needs to be defined in the
class scope, even it is not an attribute of the class, but of its
instances. This seems confusing, both to human reader and machine analyser.
>
> Example from PEP 526:
>
> class Starship:
>
> captain: str = 'Picard'
> damage: int
> stats: 

[Python-Dev] Please reject or postpone PEP 526

[Mark Shannon]

Hi everyone,

I think we should reject, or at least postpone PEP 526.

PEP 526 represents a major change to the language, however there are, I 
believe, a number of technical flaws with the PEP.


It is probable that with significant revisions it can be a worthwhile 
addition to the language, but that cannot happen in time for 3.6 beta 1 
(in 11 days).


PEP 526 claims to be an extension of PEP 484, but I don't think that is 
entirely correct.
PEP 484 was primarily about defining the semantics of pre-existing 
syntax. PEP 526 is about adding new syntax.
Historically the bar for adding new syntax has been set very high. I 
don't think that PEP 526, in its current form, reaches that bar.


Below is a list of the issues I have with the PEP as it stands.

In many cases it makes it more effort than type comments


Type hints should be as easy to use as possible, and that means pushing 
as much work as possible onto the checker, and not burdening the programmer.


Attaching type hints to variables, rather than expressions, reduces the 
potential for inference. This makes it harder for programmer, but easier 
for the checker, which is the wrong way around.


For example,, given a function:
def spam(x: Optional[List[int]])->None: ...

With type comments, this is intuitively correct and should type check:
def eggs(cond:bool):
if cond:
x = None
else:
x = [] # type: List[int]
spam(x)  # Here we can infer the type of x

With PEP 526 we loose the ability to infer types.
def eggs(cond:bool):
if cond:
x = None # Not legal due to type declaration below
else:
   x: List[int] = []
spam(x)

So we need to use a more complex type
def eggs(cond:bool):
x: Optional[List[int]]
if cond:
x = None # Now legal
else:
x: = []
spam(x)

I don't think this improves readability.
Whether this is an acceptable change is debatable, but it does need some 
debate.


It limits the use of variables
==

In Python a name (variable) is just a binding that refers to an object.
A name only exists in a meaningful sense once an object has been 
assigned to it. Any attempt to use that name, without an object bound to 
it, will result in a NameError.


PEP 526 makes variables more than just bindings, as any rebinding must 
conform to the given type. This looses us some of the dynamism for which 
we all love Python.


Quoting from the PEP:
```
a: int
a: str # Static type checker will warn about this.
```
In other words, it is illegal for a checker to split up the variable, 
even though it is straightforward to do so.


However, without the type declarations,
```
a = 1
a = "Hi"
```
is just fine. Useless, but fine.

We should be free to add extra variables, whenever we choose, for 
clarity. For example,

total = foo() - bar()
should not be treated differently from:
revenue = foo()
tax = bar()
total = revenue - tax

If types are inferred, there is no problem.
However, if they must be declared, then the use of meaningfully named 
variables is discouraged.


[A note about type-inference:
Type inference is not a universal panacea, but it can make life a lot 
easier for programmers in statically type languages.
Languages like C# use local type inference extensively and it means that 
many variables often do not need their type declared. We should take 
care not to limit the ability of checkers to infer values and types and 
make programmers' lives easier.
Within a function, type inference is near perfect, failing only 
occasionally for some generic types.
One place where type inference definitely breaks down is across calls, 
which is why PEP 484 is necessary.

]

It is premature
===

There are still plenty of issues to iron out w.r.t. PEP 484 types. I 
don't think we should be adding more syntax, until we have a *precise* 
idea of what is required.


PEP 484 states:
"If type hinting proves useful in general, a syntax for typing variables 
may be provided in a future Python version."
Has it proved useful in general? I don't think it has. Maybe it will in 
future, but it hasn't yet.


It seems confused about class attributes and instance attributes


The PEP also includes a section of how to define class attributes and 
instance attributes. It seems that everything needs to be defined in the 
class scope, even it is not an attribute of the class, but of its 
instances. This seems confusing, both to human reader and machine analyser.


Example from PEP 526:

class Starship:

captain: str = 'Picard'
damage: int
stats: ClassVar[Dict[str, int]] = {}

def __init__(self, damage: int, captain: str = None):
self.damage = damage
if captain:
self.captain = captain  # Else keep the default

With type hints as they currently exist, the same code is shorter and
doesn't 

Re: [Python-Dev] PEP 526 ready for review: Syntax for Variable and Attribute Annotations

[Ivan Levkivskyi]

On 1 September 2016 at 22:37, Guido van Rossum  wrote:
> On Thu, Sep 1, 2016 at 9:30 AM, Ivan Levkivskyi 
wrote:
> > There is a convention for function annotations in PEP 484 that a missing
> > annotation is equivalent to Any, so that I like your first option more.
>
> But Steven wasn't proposing it to mean Any, he was proposing it to
> mean "type checker should infer". Where I presume the inference should
> be done based on the assignment in __init__ only.

Sorry for misunderstanding.

On 2 September 2016 at 04:38, Nick Coghlan  wrote:
> However, a standalone Ellipsis doesn't currently have a meaning as a
> type annotation (it's only meaningful when subscripting Tuple and
> Callable), so a spelling like this might work:
>
> NAME: ...
>
> That spelling could then also be used in function definitions to say
> "infer the return type from the return statements rather than assuming
> Any"

Interesting idea.
This is somehow similar to one of the existing use of Ellipsis: in numpy it
infers how many dimensions needs to have the full slice, it is like saying
"You know what I mean". So I am +1 on this solution.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[Paul Moore]

On 2 September 2016 at 03:35, Steve Dower  wrote:
> I'd need to test to be sure, but writing an incomplete code point should
> just truncate to before that point. It may currently raise OSError if that
> truncated to zero length, as I believe that's not currently distinguished
> from an error. What behavior would you propose?

For "correct" behaviour, you should retain the unwritten bytes, and
write them as part of the next call (essentially making the API
stateful, in the same way that incremental codecs work). I'm pretty
sure that this could cause actual problems, for example I think invoke
(https://github.com/pyinvoke/invoke) gets byte streams from
subprocesses and dumps them direct to stdout in blocks (so could
easily end up splitting multibyte sequences). It''s arguable that it
should be decoding the bytes from the subprocess and then re-encoding
them, but that gets us into "guess the encoding used by the
subprocess" territory.

The problem is that we're not going to simply drop some bad data in
the common case - it's not so much the dropping of the start of an
incomplete code point that bothers me, as the encoding error you hit
at the start of the *next* block of data you send. So people will get
random, unexplained, encoding errors.

I don't see an easy answer here other than a stateful API.

> Reads of less than four bytes fail instantly, as in the worst case we need
> four bytes to represent one Unicode character. This is an unfortunate
> reality of trying to limit it to one system call - you'll never get a full
> buffer from a single read, as there is no simple mapping between
> length-as-utf8 and length-as-utf16 for an arbitrary string.

And here - "read a single byte" is a not uncommon way of getting some
data. Once again see invoke:

https://github.com/pyinvoke/invoke/blob/master/invoke/platform.py#L147

used at

https://github.com/pyinvoke/invoke/blob/master/invoke/runners.py#L548

I'm not saying that there's an easy answer here, but this *will* break
code. And actually, it's in violation of the documentation: see
https://docs.python.org/3/library/io.html#io.RawIOBase.read

"""
read(size=-1)

Read up to size bytes from the object and return them. As a
convenience, if size is unspecified or -1, readall() is called.
Otherwise, only one system call is ever made. Fewer than size bytes
may be returned if the operating system call returns fewer than size
bytes.

If 0 bytes are returned, and size was not 0, this indicates end of
file. If the object is in non-blocking mode and no bytes are
available, None is returned.
"""

You're not allowed to return 0 bytes if the requested size was not 0,
and you're not at EOF.

Having said all this, I'm strongly +1 on the idea of this PEP, it
would be fantastic to resolve the above issues and get this in.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 525, third round, better finalization

[Nathaniel Smith]

On Thu, Sep 1, 2016 at 3:34 PM, Yury Selivanov  wrote:
> Hi,
>
> I've spent quite a while thinking and experimenting with PEP 525 trying to
> figure out how to make asynchronous generators (AG) finalization reliable.
> I've tried to replace the callback for GCed with a callback to intercept
> first iteration of AGs.  Turns out it's very hard to work with weak-refs and
> make asyncio event loop to reliably track and shutdown all open AGs.
>
> My new approach is to replace the "sys.set_asyncgen_finalizer(finalizer)"
> function with "sys.set_asyncgen_hooks(firstiter=None, finalizer=None)".

1) Can/should these hooks be used by other types besides async
generators? (e.g., async iterators that are not async generators?)
What would that look like?

2) In the asyncio design it's legal for an event loop to be stopped
and then started again. Currently (I guess for this reason?) asyncio
event loops do not forcefully clean up resources associated with them
on shutdown. For example, if I open a StreamReader, loop.stop() and
loop.close() will not automatically close it for me. When, concretely,
are you imagining that asyncio will run these finalizers?

3) Should the cleanup code in the generator be able to distinguish
between "this iterator has left scope" versus "the event loop is being
violently shut down"?

4) More fundamentally -- this revision is definitely an improvement,
but it doesn't really address the main concern I have. Let me see if I
can restate it more clearly.

Let's define 3 levels of cleanup handling:

  Level 0: resources (e.g. file descriptors) cannot be reliably cleaned up.

  Level 1: resources are cleaned up reliably, but at an unpredictable time.

  Level 2: resources are cleaned up both reliably and promptly.

In Python 3.5, unless you're very anal about writing cumbersome 'async
with' blocks around every single 'async for', resources owned by aysnc
iterators land at level 0. (Because the only cleanup method available
is __del__, and __del__ cannot make async calls, so if you need async
calls to do clean up then you're just doomed.)

I think at the revised draft does a good job of moving async
generators from level 0 to level 1 -- the finalizer hook gives a way
to effectively call back into the event loop from __del__, and the
shutdown hook gives us a way to guarantee that the cleanup happens
while the event loop is still running.

But... IIUC, it's now generally agreed that for Python code, level 1
is simply *not good enough*. (Or to be a little more precise, it's
good enough for the case where the resource being cleaned up is
memory, because the garbage collector knows when memory is short, but
it's not good enough for resources like file descriptors.) The classic
example of this is code like:

 # used to be good, now considered poor style:
 def get_file_contents(path):
  handle = open(path)
  return handle.read()

This works OK on CPython because the reference-counting gc will call
handle.__del__() at the end of the scope (so on CPython it's at level
2), but it famously causes huge problems when porting to PyPy with
it's much faster and more sophisticated gc that only runs when
triggered by memory pressure. (Or for "PyPy" you can substitute
"Jython", "IronPython", whatever.) Technically this code doesn't
actually "leak" file descriptors on PyPy, because handle.__del__()
will get called *eventually* (this code is at level 1, not level 0),
but by the time "eventually" arrives your server process has probably
run out of file descriptors and crashed. Level 1 isn't good enough. So
now we have all learned to instead write

 # good modern Python style:
 def get_file_contents(path):
  with open(path) as handle:
  return handle.read()

and we have fancy tools like the ResourceWarning machinery to help us
catch these bugs.

Here's the analogous example for async generators. This is a useful,
realistic async generator, that lets us incrementally read from a TCP
connection that streams newline-separated JSON documents:

  async def read_json_lines_from_server(host, port):
  async for line in asyncio.open_connection(host, port)[0]:
  yield json.loads(line)

You would expect to use this like:

  async for data in read_json_lines_from_server(host, port):
  ...

BUT, with the current PEP 525 proposal, trying to use this generator
in this way is exactly analogous to the open(path).read() case: on
CPython it will work fine -- the generator object will leave scope at
the end of the 'async for' loop, cleanup methods will be called, etc.
But on PyPy, the weakref callback will not be triggered until some
arbitrary time later, you will "leak" file descriptors, and your
server will crash. For correct operation, you have to replace the
simple 'async for' loop with this lovely construct:

  async with aclosing(read_json_lines_from_server(host, port)) as ait:
  async for data in ait:
  ...

Of course, you only have to do this on loops whose 

Re: [Python-Dev] PEP 529: Change Windows filesystem encoding to UTF-8

[Paul Moore]

On 1 September 2016 at 23:31, Steve Dower  wrote:
[...]
> As a result, POSIX-focused library authors prefer to use bytes to represent
> paths.

A minor point, but in my experience, a lot of POSIX-focused authors
are happy to move to a better text/bytes separation, so I'd soften
this to "some POSIX-focused library authors...".

Other than that minor point, this looks great - +1 from me.
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com