Re: [Python-Dev] Please reject or postpone PEP 526

[Guido van Rossum]

On Tue, Sep 6, 2016 at 9:00 AM, Nick Coghlan  wrote:
> On 6 September 2016 at 14:04, Guido van Rossum  wrote:
>> I'm sorry, but we're not going to invent new syntax this late in the
>> game. The syntax proposed by the PEP has been on my mind ever since
>> PEP 484 with very minor variations; I first proposed it seriously on
>> python-ideas over a month ago, we've been debating the details since
>> then, and it's got a solid implementation based on those debates by
>> Ivan Levkivskyi. In contrast, it looks like you just made the "assert
>> x: T" syntax up last night in response to the worries expressed by
>> Mark Shannon, and "assert" sounds a lot like a run-time constraint to
>> me.
>
> That's a fair description, but the notation also helped me a lot in
> articulating the concepts I was concerned about without having to put
> dummy annotated functions everywhere :)

Thanks Nick! It seems your writings has helped some others (e.g.
Ethan) understand PEP 526.

>> Instead, I encourage you to participate in the writing of a separate
>> PEP explaining how type checkers are expected to work (since PEP 526
>> doesn't specify that). Ivan is also interested in such a PEP and we
>> hope Mark will also lend us his expertise.
>
> Aye, I'd be happy to help with that - I think everything proposed can
> be described in terms of existing PEP 484 primitives and the
> descriptor protocol, so the requirements on typecheckers would just be
> for them to be self-consistent, rather than defining fundamentally new
> behaviours.

Beware that there are by now some major type checkers that already
claim conformance to PEP 484 in various ways: mypy, pytype, PyCharm,
and probably Semmle.com where Mark works has one too. Each one has
some specialty and each one is a work in progress, but a PEP shouldn't
start out by declaring the approach used by any existing checker
unlawful.

As an example, mypy doesn't yet support Optional by default: it
recognizes the syntax but it doesn't distinguish between e.g. int and
Optional[int]. (It will do the right thing when you pass the
`--strict-optional` flag, but there are still some issues with that
before we can make it the default behavior.)

As another example: mypy understands isinstance() checks so that e.g.
the following works:

def foo(x: Union[int, str]) -> str:
if isinstance(x, str):
return x
return str(x)

I don't think you can find anything in PEP 484 that says this should
work; but without it mypy would be much less useful. (The example here
is silly, but such code appears in real life frequently.)

One final thought: this is not the first time that Python has used
syntax that looks like another language but gives it a different
meaning. In fact, apart from `if`, almost everything in Python works
differently than it works in C++ or Java. So I don't worry much about
that.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Nick Coghlan]

On 6 September 2016 at 02:51, Ian Foote  wrote:
> On 05/09/16 14:46, Nick Coghlan wrote:
>> That's not what the PEP proposes for uninitialised variables though:
>> it proposes processing them *before* a series of assignment
>> statements, which *only makes sense* if you plan to use them to
>> constrain those assignments in some way.
>>
>> If you wanted to write something like that under a type assertion
>> spelling, then you could enlist the aid of the "all" builtin:
>>
>> assert all(x) : List[T] # All local assignments to "x" must abide
>> by this constraint
>> if case1:
>> x = ...
>> elif case2:
>> x = ...
>> else:
>> x = ...
>>
>
> Would the `assert all(x)` be executed at runtime as well or would this
> be syntax only for type checkers? I think this particular spelling at
> least is potentially confusing.

Only for typecheckers, same as the plans for function level bare
annotations. Otherwise it wouldn't work, since you'd be calling
"all()" on a non-iterable :)

Guido doesn't like the syntax though, so the only place it would ever
appear is explanatory notes describing the purpose of the new syntax,
and hence can be replaced by something like:

# After all future assignments to x, check that x conforms to T

Cheers,
Nick.

P.S. Or, if you're particularly fond of mathematical notation, and we
take type categories as sets:

# ∀x: x ∈ T

That would be a singularly unhelpful explanatory comment for the vast
majority of folks, though :)

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Nick Coghlan]

On 6 September 2016 at 14:04, Guido van Rossum  wrote:
> I'm sorry, but we're not going to invent new syntax this late in the
> game. The syntax proposed by the PEP has been on my mind ever since
> PEP 484 with very minor variations; I first proposed it seriously on
> python-ideas over a month ago, we've been debating the details since
> then, and it's got a solid implementation based on those debates by
> Ivan Levkivskyi. In contrast, it looks like you just made the "assert
> x: T" syntax up last night in response to the worries expressed by
> Mark Shannon, and "assert" sounds a lot like a run-time constraint to
> me.

That's a fair description, but the notation also helped me a lot in
articulating the concepts I was concerned about without having to put
dummy annotated functions everywhere :)

> Instead, I encourage you to participate in the writing of a separate
> PEP explaining how type checkers are expected to work (since PEP 526
> doesn't specify that). Ivan is also interested in such a PEP and we
> hope Mark will also lend us his expertise.

Aye, I'd be happy to help with that - I think everything proposed can
be described in terms of existing PEP 484 primitives and the
descriptor protocol, so the requirements on typecheckers would just be
for them to be self-consistent, rather than defining fundamentally new
behaviours.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Ian Foote]

On 05/09/16 14:46, Nick Coghlan wrote:
> That's not what the PEP proposes for uninitialised variables though:
> it proposes processing them *before* a series of assignment
> statements, which *only makes sense* if you plan to use them to
> constrain those assignments in some way.
> 
> If you wanted to write something like that under a type assertion
> spelling, then you could enlist the aid of the "all" builtin:
> 
> assert all(x) : List[T] # All local assignments to "x" must abide
> by this constraint
> if case1:
> x = ...
> elif case2:
> x = ...
> else:
> x = ...
> 

Would the `assert all(x)` be executed at runtime as well or would this
be syntax only for type checkers? I think this particular spelling at
least is potentially confusing.

Regards,
Ian F



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Guido van Rossum]

I'm sorry, but we're not going to invent new syntax this late in the
game. The syntax proposed by the PEP has been on my mind ever since
PEP 484 with very minor variations; I first proposed it seriously on
python-ideas over a month ago, we've been debating the details since
then, and it's got a solid implementation based on those debates by
Ivan Levkivskyi. In contrast, it looks like you just made the "assert
x: T" syntax up last night in response to the worries expressed by
Mark Shannon, and "assert" sounds a lot like a run-time constraint to
me.

Instead, I encourage you to participate in the writing of a separate
PEP explaining how type checkers are expected to work (since PEP 526
doesn't specify that). Ivan is also interested in such a PEP and we
hope Mark will also lend us his expertise.

On Mon, Sep 5, 2016 at 8:46 PM, Nick Coghlan  wrote:
> On 6 September 2016 at 04:15, Guido van Rossum  wrote:
>> On Mon, Sep 5, 2016 at 10:24 AM, Ethan Furman  wrote:
>>> On 09/05/2016 06:46 AM, Nick Coghlan wrote:
>>>
>>> [an easy to understand explanation for those of us who aren't type-inferring
>>> gurus]
>>>
>>> Thanks, Nick.  I think I finally have a grip on what Mark was talking about,
>>> and about how these things should work.
>>>
>>> Much appreciated!
>>
>> There must be some misunderstanding. The message from Nick with that
>> timestamp 
>> (https://mail.python.org/pipermail/python-dev/2016-September/146200.html)
>> hinges on an incorrect understanding of the intention of annotations
>> without value (e.g. `x: Optional[int]`), leading to a -1 on the PEP.
>
> Short version of below: after sleeping on it, I'd be OK with the PEP
> again if it just *added* the explicit type assertions, such that the
> shorthand notation could be described in those terms.
>
> Specifically, "x: T = expr" would be syntactic sugar for:
>
> x = expr
> assert x: T
>
> While the bare "x: T" would be syntactic sugar for:
>
> assert all(x): T
>
> which in turn would imply that all future bindings of that assignment
> target should be accompanied by a type assertion (and typecheckers may
> differ in how they define "all future bindings").
>
> Even if everyone always writes the short forms, the explicit
> assertions become a useful aid in explaining what those short forms
> mean.
>
> The main exploratory question pushed back to the typechecking
> community to answer by 3.7 would then be to resolve precisely what
> "assert all(TARGET): ANNOTATION" means for different kinds of target
> and for different scopes (e.g. constraining nonlocal name rebindings
> in closures, constraining attribute rebinding in modules, classes, and
> instances).
>
>> I can't tell if this is an honest misunderstanding or a strawman, but
>> I want to set the intention straight.
>
> I'm pretty sure I understand your intentions (and broadly agree with
> them), I just also agree with Mark that people are going to need some
> pretty strong hints that these are not Java/C/C++/C# style type
> declarations, and am suggesting a different way of getting there by
> being more prescriptive about your intended semantics.
>
> Specifically:
>
> * for 3.6, push everything into a new form of assert statement and
> define those assertions as syntactic sugar for PEP 484 constructs
> * for 3.7 (and provisionally in 3.6), consider blessing some of those
> assertions with the bare annotation syntax
>
> Folks are already comfortable with the notion of assertions not
> necessarily being executed at runtime, and they're also comfortable
> with them as a way of doing embedded correctness testing inline with
> the code.
>
>> First of all, the PEP does not require the type checker to interpret
>> anything in a particular way; it intentionally shies away from
>> prescribing semantics (other than the runtime semantics of updating
>> __annotations__ or verifying that the target appears assignable).
>
> Unfortunately, the ordering problem the PEP introduces means it pushes
> very heavily in a particular direction, such that I think we're going
> to be better off if you actually specify draft semantics in the PEP
> (in terms of existing PEP 484 annotations), rather than leaving it
> completely open to interpretation. It's still provisional so you can
> change your mind later, but the notion of describing a not yet bound
> name is novel enough that I think more guidance (even if it's
> provisional) is needed here than was needed in the case of function
> annotations.
>
> (I realise you already understand most of the background I go through
> below - I'm spelling out my reasoning so you can hopefully figure out
> where I'm diverging from your point of view)
>
> If we look at PEP 484, all uses of annotations exist between two
> pieces of code: one that produces a value, and one that binds the
> value to a reference.
>
> As such, they act as type assertions:
>
> - on parameters, they assert "I am expecting this argument to be of 

Re: [Python-Dev] Please reject or postpone PEP 526

[Nick Coghlan]

On 6 September 2016 at 04:15, Guido van Rossum  wrote:
> On Mon, Sep 5, 2016 at 10:24 AM, Ethan Furman  wrote:
>> On 09/05/2016 06:46 AM, Nick Coghlan wrote:
>>
>> [an easy to understand explanation for those of us who aren't type-inferring
>> gurus]
>>
>> Thanks, Nick.  I think I finally have a grip on what Mark was talking about,
>> and about how these things should work.
>>
>> Much appreciated!
>
> There must be some misunderstanding. The message from Nick with that
> timestamp 
> (https://mail.python.org/pipermail/python-dev/2016-September/146200.html)
> hinges on an incorrect understanding of the intention of annotations
> without value (e.g. `x: Optional[int]`), leading to a -1 on the PEP.

Short version of below: after sleeping on it, I'd be OK with the PEP
again if it just *added* the explicit type assertions, such that the
shorthand notation could be described in those terms.

Specifically, "x: T = expr" would be syntactic sugar for:

x = expr
assert x: T

While the bare "x: T" would be syntactic sugar for:

assert all(x): T

which in turn would imply that all future bindings of that assignment
target should be accompanied by a type assertion (and typecheckers may
differ in how they define "all future bindings").

Even if everyone always writes the short forms, the explicit
assertions become a useful aid in explaining what those short forms
mean.

The main exploratory question pushed back to the typechecking
community to answer by 3.7 would then be to resolve precisely what
"assert all(TARGET): ANNOTATION" means for different kinds of target
and for different scopes (e.g. constraining nonlocal name rebindings
in closures, constraining attribute rebinding in modules, classes, and
instances).

> I can't tell if this is an honest misunderstanding or a strawman, but
> I want to set the intention straight.

I'm pretty sure I understand your intentions (and broadly agree with
them), I just also agree with Mark that people are going to need some
pretty strong hints that these are not Java/C/C++/C# style type
declarations, and am suggesting a different way of getting there by
being more prescriptive about your intended semantics.

Specifically:

* for 3.6, push everything into a new form of assert statement and
define those assertions as syntactic sugar for PEP 484 constructs
* for 3.7 (and provisionally in 3.6), consider blessing some of those
assertions with the bare annotation syntax

Folks are already comfortable with the notion of assertions not
necessarily being executed at runtime, and they're also comfortable
with them as a way of doing embedded correctness testing inline with
the code.

> First of all, the PEP does not require the type checker to interpret
> anything in a particular way; it intentionally shies away from
> prescribing semantics (other than the runtime semantics of updating
> __annotations__ or verifying that the target appears assignable).

Unfortunately, the ordering problem the PEP introduces means it pushes
very heavily in a particular direction, such that I think we're going
to be better off if you actually specify draft semantics in the PEP
(in terms of existing PEP 484 annotations), rather than leaving it
completely open to interpretation. It's still provisional so you can
change your mind later, but the notion of describing a not yet bound
name is novel enough that I think more guidance (even if it's
provisional) is needed here than was needed in the case of function
annotations.

(I realise you already understand most of the background I go through
below - I'm spelling out my reasoning so you can hopefully figure out
where I'm diverging from your point of view)

If we look at PEP 484, all uses of annotations exist between two
pieces of code: one that produces a value, and one that binds the
value to a reference.

As such, they act as type assertions:

- on parameters, they assert "I am expecting this argument to be of this type"
- on assignments, they assert "I an expecting this initialiser to be
of this type"

Typecheckers can then use those assertions in two ways: as a
constraint on the value producer, and as a more precise hint if type
inference either isn't possible (e.g. function parameters,
initialisation to None), or gives an overly broad answer (e.g empty
containers)

The "x: T = expr" syntax is entirely conformant with that system - all
it does is change the spelling of the existing type hint comments.

Allowing "assert x: T" would permit that existing kind of type
assertion to be inserted at arbitrary points in the code without
otherwise affecting control flow or type inference, as if you had
written:

 # PEP 484
 def is_T(arg: T) -> None:
 pass

is_T(x)

Or:

# PEP 526
x: T = x

By contrast, bare annotations on new assignment targets without an
initialiser can't be interpreted that way, as there is no *preceding
value to constrain*.

That inability to interpret them in the same sense 

Re: [Python-Dev] Please reject or postpone PEP 526

[Ivan Levkivskyi]

On 5 September 2016 at 20:15, Guido van Rossum  wrote:

> There are actually at least two separate cases: if x is a local
> variable, the intention of `x: ` is quite different from when x
> occurs in a class.
>

If I understand you correctly this also matches my mental model.
In local scope

x: ann = value

acts like a filter allowing only something compatible to be assigned
at this point (and/or casting to a more precise type).
While in class or module it is a part of an "API specification" for that
class/module.


> I am at a loss how to modify the PEP to avoid this misunderstanding,
> since it appears it is entirely in the reader's mind. The PEP is not a
> tutorial but a spec for the implementation, ...
>

I was thinking about changing terminology to name annotations, but that will
not solve problem. The PEP mentions a separate document (guidelines) that
will be published.
I think a real solution will be to make a separate PEP that will explain in
details
what is preferred meaning of types and what people and machines could do
with types.

Is anyone interested in going in this direction? I would like to especially
invite Mark,
you have a lot of experience with types inference that would be very helpful
(also it seems to me that you are concerned about this).

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Terry Reedy]

On 9/5/2016 11:34 AM, Guido van Rossum wrote:

On Mon, Sep 5, 2016 at 7:19 AM, Mark Shannon  wrote:



Indeed, we shouldn't panic. We should take our time, review this carefully
and make sure that the version of typehints that lands in 3.7 is one that we
most of us are happy with and all of us can at least tolerate.


Right, we want the best possible version to land in 3.7. And in order
to make that possible, I have to accept it *provisionally* for 3.6 and


Until now, the 'provisional' part has not been clear to me, and 
presumably others who have written as if acceptance meant 'baked in 
stone'.  We have had provisional modules, but not, that I can think of, 
syntax that remains provisional past the x.y.0 release.



Ivan's implementation will go into 3.6b1. We will then have until 3.7
to experiment with it and tweak it as necessary.


New syntax is usually implemented within python itself, and can be fully 
experimented with during alpha and beta releases.  In this case, the 
effective implementation will be in 3rd party checkers and 
experimentation will take longer.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Guido van Rossum]

On Mon, Sep 5, 2016 at 10:24 AM, Ethan Furman  wrote:
> On 09/05/2016 06:46 AM, Nick Coghlan wrote:
>
> [an easy to understand explanation for those of us who aren't type-inferring
> gurus]
>
> Thanks, Nick.  I think I finally have a grip on what Mark was talking about,
> and about how these things should work.
>
> Much appreciated!

There must be some misunderstanding. The message from Nick with that
timestamp 
(https://mail.python.org/pipermail/python-dev/2016-September/146200.html)
hinges on an incorrect understanding of the intention of annotations
without value (e.g. `x: Optional[int]`), leading to a -1 on the PEP.

I can't tell if this is an honest misunderstanding or a strawman, but
I want to set the intention straight.

First of all, the PEP does not require the type checker to interpret
anything in a particular way; it intentionally shies away from
prescribing semantics (other than the runtime semantics of updating
__annotations__ or verifying that the target appears assignable).

But there appears considerable fear about what expectations the PEP
has of a reasonable type checker. In response to this I'll try to
sketch how I think this should be implemented in mypy.

There are actually at least two separate cases: if x is a local
variable, the intention of `x: ` is quite different from when x
occurs in a class.

- When found in a class, all *uses* (which may appear in modules far
away from the definition) must be considered to conform to the stated
type -- as must all assignments to it, but I believe that's never been
in doubt. There are just too many edge cases to consider to make
stricter assumptions (e.g. threading, exceptions, signals), so that
even after seeing `self.x = 42; use(self.x)` the call to use() cannot
assume that self.x is still 42.

- But when found inside a function referring to a local variable, mypy
should treat the annotation as a restriction on assignment, and use
its own inference engine to type-check *uses* of that variable. So
that in this example (after Mark's):

def bar() -> Optional[int]: ...

def foo() -> int:
x: Optional[int]
x = bar()
if x is None:
return -1
return x

there should not be an error on `return x` because mypy is smart
enough to know it cannot be None at that point.

I am at a loss how to modify the PEP to avoid this misunderstanding,
since it appears it is entirely in the reader's mind. The PEP is not a
tutorial but a spec for the implementation, and as a spec it is quite
clear that it leaves the type-checking semantics up to individual type
checkers. And I think that is the right thing to do -- in practice
there are many other ways to write the above example, and mypy will
understand some of them, but not others, while other type checkers may
understand a different subset of examples. I can't possibly prescribe
how type checkers should behave in each case -- I can't even tell
which cases are important to distinguish.

So writing down "the type checker should not report an error in the
following case" in the PEP is not going to be helpful for anyone (in
contrast, I think discussing examples on a mailing list *is* useful).
Like a linter, a type checker has limited intelligence, and it will be
a quality of implementation issue as to how useful a type checker will
be in practice. But that's not the topic of PEP 526.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Ethan Furman]

On 09/05/2016 06:46 AM, Nick Coghlan wrote:

[an easy to understand explanation for those of us who aren't type-inferring 
gurus]

Thanks, Nick.  I think I finally have a grip on what Mark was talking about, 
and about how these things should work.

Much appreciated!

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Chris Angelico]

On Tue, Sep 6, 2016 at 2:17 AM, Steven D'Aprano  wrote:
>> > The type comment systax is required for Python 2 and backwards-
>> > compatibility. That's a given.
>>
>> Sure, but all type checkers will not have to care about Python 2.
>
> They will have to care about type comments until such time as they are
> ready to abandon all versions of Python older than 3.6.

More specifically, until *the code they check* can abandon all <3.6.
If the checker itself depends on new features (say, an improved AST
parser that retains inline comments for subsequent evaluation), you
could say "You must have Python 3.6 or better to use this checker",
but the application itself would still be able to run on older
versions. That's another reason not to delay this PEP until 3.7, as
it'd push _everything_ another 18 months (or more) into the future.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Steven D'Aprano]

On Mon, Sep 05, 2016 at 04:40:08PM +0300, Koos Zevenhoven wrote:

> On Mon, Sep 5, 2016 at 3:10 PM, Steven D'Aprano  wrote:
> >
> > [Koos Zevenhoven]
> >> >> How is it going to help that these are equivalent within one checker,
> >> >> if the meaning may differ across checkers?
> >
> > Before I can give an answer to your [Koos'] question, I have to
> > understand what you see as the problem here.
> 
> The problem was that suggested restrictive addition into PEP 526 with
> no proper justification, especially since the PEP was not supposed to
> restrict the semantics of type checking. 

What "suggested restrictive addition into PEP 526" are you referring to?

Please be specific.


> I was asking how it would
> help to add that restriction. Very simple. Maybe some people got
> confused because I did want to *discuss* best practices for type
> checking elsewhere.

I still can't answer your question, because I don't understand what 
restriction you are talking about. Unless you mean the restriction that 
variable annotations are to mean the same thing whether they are written 
as `x:T = v` or `x = v #type: T`. I don't see this as a restriction.



> > The type comment systax is required for Python 2 and backwards-
> > compatibility. That's a given.
> 
> Sure, but all type checkers will not have to care about Python 2.

They will have to care about type comments until such time as they are 
ready to abandon all versions of Python older than 3.6.

And even then, there will probably be code still written with type 
comments until Python 4000.


-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Guido van Rossum]

On Mon, Sep 5, 2016 at 7:19 AM, Mark Shannon  wrote:
> On 04/09/16 21:16, Guido van Rossum wrote:
>>
>> Everybody please stop panicking. PEP 526 does not make a stand on the
>> behavior of type checkers (other than deferring to PEP 484). If you
>> want to start a discussion about constraining type checkers please do
>> it over at python-ideas. There is no rush as type checkers are not
>> affected by the feature freeze.
>>
>
> Indeed, we shouldn't panic. We should take our time, review this carefully
> and make sure that the version of typehints that lands in 3.7 is one that we
> most of us are happy with and all of us can at least tolerate.

Right, we want the best possible version to land in 3.7. And in order
to make that possible, I have to accept it *provisionally* for 3.6 and
Ivan's implementation will go into 3.6b1. We will then have until 3.7
to experiment with it and tweak it as necessary.

Maybe ClassVar will turn out to be pointless. Maybe we'll decide that
we want to have a syntax for quickly annotating several variables with
the same type (x, y, z: T). Maybe we'll change the rules for how or
when __annotations__ is updated. Maybe we'll change slightly whether
we'll allow annotating complex assignment targets like x[f()].

But without starting the experiment now we won't be able to evaluate
any of those things. Waiting until 3.7 is just going to cause the
exact same discussions that are going on now 18 months from now.

Regarding how type checkers should use the new syntax, PEP 526 itself
give barely more guidance than PEP 3107, except that we now have PEP
484 to tell us what types ought to look like, *if* you want to use an
external type checker.

I hope that you and others will help write another PEP
(informational?) to guide type checkers and their users. Given my own
experience at Dropbox (much of it vicariously through the eyes of the
many Dropbox engineers annotating their own code) I am *very*
reluctant to try and specify the behavior of a type checker formally
myself. As anyone who has used mypy on a sizeable project knows, there
are a lot more details to sort out than how to handle branches that
assign different values to the same variable.

For people who want to read about what it is like to use mypy
seriously, I can recommend the series of three blog posts by Daniel
Moisset starting here:
http://www.machinalis.com/blog/a-day-with-mypy-part-1/

If you want to see a large open source code base that's annotated for
mypy (with 97% coverage), I recommend looking at Zulip:
https://github.com/zulip/zulip
Try digging through the history and looking for commits mentioning
mypy; a Google Summer of Code student did most of the work over the
summer. (The syntax used is the Python-2-compatible version, but
that's hardly relevant -- the important things to observe include how
they use types and how they had to change their code to pacify mypy.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

On Mon, Sep 5, 2016 at 5:02 PM, Nick Coghlan  wrote:
> On 5 September 2016 at 23:46, Nick Coghlan  wrote:
>> Under such "parameter annotation like" semantics, uninitialised
>> variable annotations would only make sense as a new form of
>> post-initialisation assertion,

Why not discuss this in the python-ideas thread where I quote myself
from last Friday regarding the notion of annotations as assertions?

>> and perhaps as some form of
>> Eiffel-style class invariant documentation syntax.

I hope this is simpler than it sounds :-)

> Thinking further about the latter half of that comment, I realised
> that the PEP 484 equivalence I'd like to see for variable annotations
> in a class body is how they would relate to a property definition
> using the existing PEP 484 syntax.
>
> For example, consider:
>
> class AnnotatedProperty:
>
> @property
> def x(self) -> int:
> ...
>
> @x.setter
> def x(self, value: int) -> None:
> ...
>
> @x.deleter
> def x(self) -> None:
> ...
>
> It would be rather surprising if that typechecked differently from:
>
> class AnnotatedVariable:
>
> x: int
>

How about just using the latter way? That's much clearer. I doubt this
needs a change in the PEP.

> For ClassVar, you'd similarly want:
>
>
> class AnnotatedClassVariable:
>
> x: ClassVar[int]
>
> to typecheck like "x" was declared as an annotated property on the metaclass.
>

Sure, there are many things that one may consider equivalent. I doubt
you'll be able to list them all in a way that everyone agrees on. And
I hope you don't take this as a challenge -- I'm in the don't-panic
camp :).


-- Koos


> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Mark Shannon]

On 04/09/16 21:16, Guido van Rossum wrote:

Everybody please stop panicking. PEP 526 does not make a stand on the
behavior of type checkers (other than deferring to PEP 484). If you
want to start a discussion about constraining type checkers please do
it over at python-ideas. There is no rush as type checkers are not
affected by the feature freeze.



Indeed, we shouldn't panic. We should take our time, review this 
carefully and make sure that the version of typehints that lands in 3.7 
is one that we most of us are happy with and all of us can at least 
tolerate.


Cheers,
Mark.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Nick Coghlan]

On 5 September 2016 at 23:46, Nick Coghlan  wrote:
> Under such "parameter annotation like" semantics, uninitialised
> variable annotations would only make sense as a new form of
> post-initialisation assertion, and perhaps as some form of
> Eiffel-style class invariant documentation syntax.

Thinking further about the latter half of that comment, I realised
that the PEP 484 equivalence I'd like to see for variable annotations
in a class body is how they would relate to a property definition
using the existing PEP 484 syntax.

For example, consider:

class AnnotatedProperty:

@property
def x(self) -> int:
...

@x.setter
def x(self, value: int) -> None:
...

@x.deleter
def x(self) -> None:
...

It would be rather surprising if that typechecked differently from:

class AnnotatedVariable:

x: int

For ClassVar, you'd similarly want:


class AnnotatedClassVariable:

x: ClassVar[int]

to typecheck like "x" was declared as an annotated property on the metaclass.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

Sorry, I don't have time to read emails of this length now, and
perhaps I'm interpreting your emails more literally than you write
them, anyway.

If PEP 484 introduces unnecessary restrictions at this point, that's a
separate issue. I see no need to copy those into PEP 526. I'll be
posting my own remaining concerns regarding PEP 526 when I find the
time.

-- Koos


On Mon, Sep 5, 2016 at 4:46 PM, Nick Coghlan  wrote:
> On 5 September 2016 at 21:46, Koos Zevenhoven  wrote:
>> The thing I'm promoting here is to not add anything to PEP 526 that
>> says what a type checker is supposed to do with type annotations.
>
> PEP 526 says it doesn't intend to expand the scope of typechecking
> semantics beyond what PEP 484 already supports. For that to be true,
> it needs to be able to define expected equivalencies between the
> existing semantics of PEP 484 and the new syntax in PEP 526.
>
> If those equivalencies can't be defined, then Mark's concerns are
> valid, and the PEP either needs to be deferred as inadvertently
> introducing new semantics while intending to only introduce new
> syntax, or else the intended semantics need to be spelled out as they
> were in PEP 484 so folks can judge the proposal accurately, rather
> than attempting to judge it based on an invalid premise.
>
> For initialised variables, the equivalence between the two PEPs is
> straightforward: "x: T = expr" is equivalent to "x = expr # type: T"
>
> If PEP 526 always required an initialiser, and didn't introduce
> ClassVar, there'd be no controversy, and we'd already be done.
>
> However, the question of "Does this new syntax necessarily imply the
> introduction of new semantics?" gets a lot murkier for uninitialised
> variables.
>
> A strict "no new semantics beyond PEP 484" interpretation would mean
> that these need to be interpreted the same way as parameter
> annotations: as a type hint on the outcome of the code executed up to
> that point, rather than as a type constraint on assignment statements
> in the code *following* that point.
>
> Consider:
>
> def simple_appender(base: List[T], value: T) -> None:
> base.append(value)
>
> This will typecheck fine - lists have append methods, and the value
> appended conforms to what our list expects.
>
> The parameter annotations mainly act as constraints on how this
> function is *called*, with the following all being problematic:
>
> simple_appender([1, 2, 3], "hello") # Container/value type mismatch
> simple_appender([1, 2, 3], None) # Value is not optional
> simple_appender((1, 2, 3), 4) # A tuple is not a list
>
> However, because of the way name binding in Python works, the
> annotations in *no way* constrain assignments inside the function
> body:
>
> def not_so_simple_appender(base: List[T], value: T) -> None:
> other_ref = base
> base = value
> other_ref.append(base)
>
> From a dynamic typechecking perspective, that's just as valid as the
> original implementation, since the "List[T]" type of "other_ref" is
> inferred from the original type of "base" before it gets rebound to
> value and has its new type inferred as "T".
>
> This freedom to rebind an annotated name without a typechecker
> complaining is what Mark is referring to when he says that PEP 484
> attaches annotations to expressions rather than types.
>
> Under such "parameter annotation like" semantics, uninitialised
> variable annotations would only make sense as a new form of
> post-initialisation assertion, and perhaps as some form of
> Eiffel-style class invariant documentation syntax.
>
> The usage to help ensure code correctness in multi-branch
> initialisation cases would then look something like this:
>
>if case1:
> x = ...
> elif case2:
> x = ...
> else:
> x = ...
> assert x : List[T] # If we get to here without x being List[T],
> something's wrong
>
> The interpreter could then optimise type assertions out entirely at
> function level (even in __debug__ mode), and turn them into
> annotations at module and class level (with typecheckers then deciding
> how to process them).
>
> That's not what the PEP proposes for uninitialised variables though:
> it proposes processing them *before* a series of assignment
> statements, which *only makes sense* if you plan to use them to
> constrain those assignments in some way.
>
> If you wanted to write something like that under a type assertion
> spelling, then you could enlist the aid of the "all" builtin:
>
> assert all(x) : List[T] # All local assignments to "x" must abide
> by this constraint
> if case1:
> x = ...
> elif case2:
> x = ...
> else:
> x = ...
>
> So I've come around to the point of view of being a solid -1 on the
> PEP as written - despite the best of intentions, it strongly
> encourages "assert all(x): List[T]" as the default interpretation of
> unitialised variable annotations, and doesn't 

Re: [Python-Dev] Please reject or postpone PEP 526

[Nick Coghlan]

On 5 September 2016 at 21:46, Koos Zevenhoven  wrote:
> The thing I'm promoting here is to not add anything to PEP 526 that
> says what a type checker is supposed to do with type annotations.

PEP 526 says it doesn't intend to expand the scope of typechecking
semantics beyond what PEP 484 already supports. For that to be true,
it needs to be able to define expected equivalencies between the
existing semantics of PEP 484 and the new syntax in PEP 526.

If those equivalencies can't be defined, then Mark's concerns are
valid, and the PEP either needs to be deferred as inadvertently
introducing new semantics while intending to only introduce new
syntax, or else the intended semantics need to be spelled out as they
were in PEP 484 so folks can judge the proposal accurately, rather
than attempting to judge it based on an invalid premise.

For initialised variables, the equivalence between the two PEPs is
straightforward: "x: T = expr" is equivalent to "x = expr # type: T"

If PEP 526 always required an initialiser, and didn't introduce
ClassVar, there'd be no controversy, and we'd already be done.

However, the question of "Does this new syntax necessarily imply the
introduction of new semantics?" gets a lot murkier for uninitialised
variables.

A strict "no new semantics beyond PEP 484" interpretation would mean
that these need to be interpreted the same way as parameter
annotations: as a type hint on the outcome of the code executed up to
that point, rather than as a type constraint on assignment statements
in the code *following* that point.

Consider:

def simple_appender(base: List[T], value: T) -> None:
base.append(value)

This will typecheck fine - lists have append methods, and the value
appended conforms to what our list expects.

The parameter annotations mainly act as constraints on how this
function is *called*, with the following all being problematic:

simple_appender([1, 2, 3], "hello") # Container/value type mismatch
simple_appender([1, 2, 3], None) # Value is not optional
simple_appender((1, 2, 3), 4) # A tuple is not a list

However, because of the way name binding in Python works, the
annotations in *no way* constrain assignments inside the function
body:

def not_so_simple_appender(base: List[T], value: T) -> None:
other_ref = base
base = value
other_ref.append(base)

>From a dynamic typechecking perspective, that's just as valid as the
original implementation, since the "List[T]" type of "other_ref" is
inferred from the original type of "base" before it gets rebound to
value and has its new type inferred as "T".

This freedom to rebind an annotated name without a typechecker
complaining is what Mark is referring to when he says that PEP 484
attaches annotations to expressions rather than types.

Under such "parameter annotation like" semantics, uninitialised
variable annotations would only make sense as a new form of
post-initialisation assertion, and perhaps as some form of
Eiffel-style class invariant documentation syntax.

The usage to help ensure code correctness in multi-branch
initialisation cases would then look something like this:

   if case1:
x = ...
elif case2:
x = ...
else:
x = ...
assert x : List[T] # If we get to here without x being List[T],
something's wrong

The interpreter could then optimise type assertions out entirely at
function level (even in __debug__ mode), and turn them into
annotations at module and class level (with typecheckers then deciding
how to process them).

That's not what the PEP proposes for uninitialised variables though:
it proposes processing them *before* a series of assignment
statements, which *only makes sense* if you plan to use them to
constrain those assignments in some way.

If you wanted to write something like that under a type assertion
spelling, then you could enlist the aid of the "all" builtin:

assert all(x) : List[T] # All local assignments to "x" must abide
by this constraint
if case1:
x = ...
elif case2:
x = ...
else:
x = ...

So I've come around to the point of view of being a solid -1 on the
PEP as written - despite the best of intentions, it strongly
encourages "assert all(x): List[T]" as the default interpretation of
unitialised variable annotations, and doesn't provide an easy way to
do arbitrary inline type assertions to statically check the
correctness of the preceding code the way we can with runtime
assertions and as would happen if the code in question was factored
out to an annotated function.

Stick the "assert" keyword in front of them though, call them type
assertions rather than type declarations, and require all() when you
want to constrain all assignments later in the function (or until the
next relevant type assertion), and I'm a solid +1.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

It looks like you are trying to make sense of this, but unfortunately
there's some added mess and copy errors regarding who said
what. I think no such errors remain in what I quote below:

On Mon, Sep 5, 2016 at 3:10 PM, Steven D'Aprano  wrote:
>
> [Koos Zevenhoven]
>> >> How is it going to help that these are equivalent within one checker,
>> >> if the meaning may differ across checkers?
>
> Before I can give an answer to your [Koos'] question, I have to
> understand what you see as the problem here.

The problem was that suggested restrictive addition into PEP 526 with
no proper justification, especially since the PEP was not supposed to
restrict the semantics of type checking. I was asking how it would
help to add that restriction. Very simple. Maybe some people got
confused because I did want to *discuss* best practices for type
checking elsewhere.

> I *think* that you are worried that two different checkers will disagree
> on what counts as a type error. That given the same chunk of code:

In the long term, I'm worried about that, but there's nothing that PEP
526 can do about it at this point.

> [Nick Coghlan]
>> > For typechecker developers, it provides absolute clarity that the
>> > semantics of the new annotations should match the behaviour of
>> > existing type comments when there's an initialiser present,
>
> [Koos]
>> I understood that, but what's the benefit?
>
> Are you asking what is the benefit of having three forms of syntax for
> the same thing?

No, still the same thing: What is the benefit of that particular
restriction, when there are no other restrictions? Better just leave
it out.

> The type comment systax is required for Python 2 and backwards-
> compatibility. That's a given.

Sure, but all type checkers will not have to care about Python 2.

> The variable annotation syntax is required because the type comment
> syntax is (according to the PEP) very much a second-best solution. See
> the PEP:
>
> https://www.python.org/dev/peps/pep-0526/#id4
>
> So this is a proposal to create a *better* syntax for something which
> already exists. The old version, using comments, cannot be deprecated or
> removed, as it is required for Python 3.5 and older.

Right.

> Once we allow
>
> x: T = value
>
> then there benefit in also allowing:
>
> x: T
> x = value
>
> since this supports some of the use cases that aren't well supported by
> type comments or one-line variable annotations. E.g. very long or deeply
> indented lines, situations where the assignment to x is inside an
> if...else branch, or any other time you wish to declare the type of the
> variable before actually setting the variable.

Sure.

> [Nick]
>> > For folks following along without necessarily keeping up with all the
>> > nuances, it makes it more explicit what Guido means when he says "PEP
>> > 526 does not make a stand on the
>> > behavior of type checkers (other than deferring to PEP 484)."
>
> [Koos]
>> What you are proposing is exactly "making a stand on the behavior of
>> type checkers", and the examples you provide below are all variations
>> of the same situation and provide no justification for a general rule.
>
> I'm sorry, I don't understand this objection. The closest I can get to
> an answer would be:
>
> A general rule is better than a large number of unconnected, arbitrary,
> special cases.

A general rule that does not solve a problem is worse than no rule.


-- Koos


>
> --
> Steve
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Steven D'Aprano]

On Mon, Sep 05, 2016 at 11:19:38AM +0300, Koos Zevenhoven wrote:
> On Mon, Sep 5, 2016 at 5:21 AM, Nick Coghlan  wrote:
[...]
> > On 5 September 2016 at 04:40, Koos Zevenhoven  wrote:
> >> On Sun, Sep 4, 2016 at 9:13 PM, Ivan Levkivskyi  
> >> wrote:
> >>> On 4 September 2016 at 19:59, Nick Coghlan  wrote:
> >> [...]


[Ivan Levkivskyi]
>  Similarly, it would be reasonable to say that these three snippets
>  should all be equivalent from a typechecking perspective:
> 
>  x = None # type: Optional[T]
> 
>  x: Optional[T] = None
> 
>  x: Optional[T]
>  x = None
[...]

[Koos Zevenhoven]
> >> How is it going to help that these are equivalent within one checker,
> >> if the meaning may differ across checkers?

Before I can give an answer to your [Koos'] question, I have to 
understand what you see as the problem here.

I *think* that you are worried that two different checkers will disagree 
on what counts as a type error. That given the same chunk of code:

x: Optional[T] = None
if x:
spam(x)
else:
   x.eggs()

two checkers will disagree as to whether or not the code is safe. Is 
that your concern? If not, can you explain in more detail what your 
concern is?


[Nick Coghlan]
> > For typechecker developers, it provides absolute clarity that the
> > semantics of the new annotations should match the behaviour of
> > existing type comments when there's an initialiser present,

[Koos]
> I understood that, but what's the benefit? 

Are you asking what is the benefit of having three forms of syntax for 
the same thing?

The type comment systax is required for Python 2 and backwards- 
compatibility. That's a given.

The variable annotation syntax is required because the type comment 
syntax is (according to the PEP) very much a second-best solution. See 
the PEP:

https://www.python.org/dev/peps/pep-0526/#id4

So this is a proposal to create a *better* syntax for something which 
already exists. The old version, using comments, cannot be deprecated or 
removed, as it is required for Python 3.5 and older.

Once we allow 

x: T = value

then there benefit in also allowing:

x: T
x = value

since this supports some of the use cases that aren't well supported by 
type comments or one-line variable annotations. E.g. very long or deeply 
indented lines, situations where the assignment to x is inside an 
if...else branch, or any other time you wish to declare the type of the 
variable before actually setting the variable.



[Koos]
> I hope there will be a type checker that breaks this "rule".

I don't understand. Do you mean that you want three different behaviours 
for these type annotations? What would they do differently?

To me, all three are clear and obvious ways of declaring the type of a 
variable. Whether I write `x: T = expr` or `x = expr  #type:T`, it 
should be clear that I intend `x` to be treated as T. What would you do 
differently?


[Nick]
> > or of a
> > parameter annotation when there's no initialiser present.

[Koos] 
> No, your suggested addition does not provide any reference to this.
> (...luckily, because that would have been worse.)

I'm sorry, I don't follow you. Are you suggesting that we should have 
the syntax `name:T = value` mean something different inside and outside 
of a function parameter list?

def func(x:T = v):
y:T = v


The first line declares x as type T with default value v; the second 
line declares y as type T with initial value v. You say this is 
"worse"... worse than what? What behaviour would you prefer to see?



[Nick]
> > For folks following along without necessarily keeping up with all the
> > nuances, it makes it more explicit what Guido means when he says "PEP
> > 526 does not make a stand on the
> > behavior of type checkers (other than deferring to PEP 484)."

[Koos] 
> What you are proposing is exactly "making a stand on the behavior of
> type checkers", and the examples you provide below are all variations
> of the same situation and provide no justification for a general rule.

I'm sorry, I don't understand this objection. The closest I can get to 
an answer would be:

A general rule is better than a large number of unconnected, arbitrary, 
special cases.

Does that help?


-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

On Mon, Sep 5, 2016 at 1:04 PM, Nick Coghlan  wrote:
> On 5 September 2016 at 18:19, Koos Zevenhoven  wrote:
>> On Mon, Sep 5, 2016 at 5:21 AM, Nick Coghlan  wrote:
>>> On 5 September 2016 at 04:40, Koos Zevenhoven  wrote:
 On Sun, Sep 4, 2016 at 9:13 PM, Ivan Levkivskyi  
 wrote:
> On 4 September 2016 at 19:59, Nick Coghlan  wrote:
 [...]
>>
>> Similarly, it would be reasonable to say that these three snippets
>> should all be equivalent from a typechecking perspective:
>>
>> x = None # type: Optional[T]
>>
>> x: Optional[T] = None
>>
>> x: Optional[T]
>> x = None
>
>
> Nice idea, explicit is better than implicit.

 How is it going to help that these are equivalent within one checker,
 if the meaning may differ across checkers?
>>>
>>> For typechecker developers, it provides absolute clarity that the
>>> semantics of the new annotations should match the behaviour of
>>> existing type comments when there's an initialiser present,
>>
>> I understood that, but what's the benefit? I hope there will be a type
>> checker that breaks this "rule".
>
> Such a typechecker means you're not writing Python anymore, you're
> writing Java/C++/C# in a language that isn't designed to be used that
> way.

I'm glad those are all the languages you accuse me of. The list could
have been a lot worse. I actually have some good memories of Java. It
felt kind of cool at that age, and it taught me many things about
undertanding the structure of large and complicated programs after I
had been programming for years in other languages, including C++. It
also taught me to value simplicity instead, so here we are.

> Fortunately, none of the current typecheckers have made that mistake,
> nor does anyone appear to be promoting this mindset outside this
> particular discussion.

The thing I'm promoting here is to not add anything to PEP 526 that
says what a type checker is supposed to do with type annotations.
Quite the opposite of Java/C++/C#, I would say.

We can, of course, speculate about the future of type checkers and the
implications of PEP 526 on it. That's what I'm trying to do on
python-ideas, speculate about the best kind of type checking
(achievable with PEP 526 annotations) [1].

--Koos


[1] https://mail.python.org/pipermail/python-ideas/2016-September/042076.html

>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Nick Coghlan]

On 5 September 2016 at 18:19, Koos Zevenhoven  wrote:
> On Mon, Sep 5, 2016 at 5:21 AM, Nick Coghlan  wrote:
>> On 5 September 2016 at 04:40, Koos Zevenhoven  wrote:
>>> On Sun, Sep 4, 2016 at 9:13 PM, Ivan Levkivskyi  
>>> wrote:
 On 4 September 2016 at 19:59, Nick Coghlan  wrote:
>>> [...]
>
> Similarly, it would be reasonable to say that these three snippets
> should all be equivalent from a typechecking perspective:
>
> x = None # type: Optional[T]
>
> x: Optional[T] = None
>
> x: Optional[T]
> x = None


 Nice idea, explicit is better than implicit.
>>>
>>> How is it going to help that these are equivalent within one checker,
>>> if the meaning may differ across checkers?
>>
>> For typechecker developers, it provides absolute clarity that the
>> semantics of the new annotations should match the behaviour of
>> existing type comments when there's an initialiser present,
>
> I understood that, but what's the benefit? I hope there will be a type
> checker that breaks this "rule".

Such a typechecker means you're not writing Python anymore, you're
writing Java/C++/C# in a language that isn't designed to be used that
way.

Fortunately, none of the current typecheckers have made that mistake,
nor does anyone appear to be promoting this mindset outside this
particular discussion.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

On Mon, Sep 5, 2016 at 5:21 AM, Nick Coghlan  wrote:
> On 5 September 2016 at 04:40, Koos Zevenhoven  wrote:
>> On Sun, Sep 4, 2016 at 9:13 PM, Ivan Levkivskyi  wrote:
>>> On 4 September 2016 at 19:59, Nick Coghlan  wrote:
>> [...]

 Similarly, it would be reasonable to say that these three snippets
 should all be equivalent from a typechecking perspective:

 x = None # type: Optional[T]

 x: Optional[T] = None

 x: Optional[T]
 x = None
>>>
>>>
>>> Nice idea, explicit is better than implicit.
>>
>> How is it going to help that these are equivalent within one checker,
>> if the meaning may differ across checkers?
>
> For typechecker developers, it provides absolute clarity that the
> semantics of the new annotations should match the behaviour of
> existing type comments when there's an initialiser present,

I understood that, but what's the benefit? I hope there will be a type
checker that breaks this "rule".

> or of a
> parameter annotation when there's no initialiser present.

No, your suggested addition does not provide any reference to this.
(...luckily, because that would have been worse.)

> For folks following along without necessarily keeping up with all the
> nuances, it makes it more explicit what Guido means when he says "PEP
> 526 does not make a stand on the
> behavior of type checkers (other than deferring to PEP 484)."

What you are proposing is exactly "making a stand on the behavior of
type checkers", and the examples you provide below are all variations
of the same situation and provide no justification for a general rule.

Here's a general rule:

The closer it gets to the end of drafting a PEP [1],
the more carefully you have to justify changes.

Justification is left as an exercise ;-).

--Koos

[1] or any document (or anything, I suppose)

> For example, the formulation of the divergent initialisation case
> where I think the preferred semantics are already implied by PEP 484
> can be looked at this way:
>
> x = None # type: Optional[List[T]]
> if arg is not None:
> x = list(arg)
> if other_arg is not None:
> x.extend(arg)
>
> It would be a strange typechecker indeed that handled that case
> differently from the new spellings made possible by PEP 526:
>
> x: Optional[List[T]] = None
> if arg is not None:
> x = list(arg)
> if other_arg is not None:
> x.extend(arg)
>
> x: Optional[List[T]]
> if arg is None:
> x = None
> else:
> x = list(arg)
> if other_arg is not None:
> x.extend(arg)
>
> x: Optional[List[T]]
> if arg is not None:
> x = list(arg)
> if other_arg is not None:
> x.extend(arg)
> else:
> x = None
>
> Or from the semantics of PEP 484 parameter annotations:
>
> def my_func(arg:Optional[List[T]], other_arg=None):
> # other_arg is implicitly Optional[Any]
> if arg is not None and other_arg is not None:
> # Here, "arg" can be assumed to be List[T]
> # while "other_arg" is Any
> arg.extend(other_arg)
>
> A self-consistent typechecker will either allow all of the above, or
> prohibit all of the above, while a typechecker that *isn't*
> self-consistent would be incredibly hard to use.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Nick Coghlan]

On 5 September 2016 at 04:40, Koos Zevenhoven  wrote:
> On Sun, Sep 4, 2016 at 9:13 PM, Ivan Levkivskyi  wrote:
>> On 4 September 2016 at 19:59, Nick Coghlan  wrote:
> [...]
>>>
>>> Similarly, it would be reasonable to say that these three snippets
>>> should all be equivalent from a typechecking perspective:
>>>
>>> x = None # type: Optional[T]
>>>
>>> x: Optional[T] = None
>>>
>>> x: Optional[T]
>>> x = None
>>
>>
>> Nice idea, explicit is better than implicit.
>
> How is it going to help that these are equivalent within one checker,
> if the meaning may differ across checkers?

For typechecker developers, it provides absolute clarity that the
semantics of the new annotations should match the behaviour of
existing type comments when there's an initialiser present, or of a
parameter annotation when there's no initialiser present.

For folks following along without necessarily keeping up with all the
nuances, it makes it more explicit what Guido means when he says "PEP
526 does not make a stand on the
behavior of type checkers (other than deferring to PEP 484)."

For example, the formulation of the divergent initialisation case
where I think the preferred semantics are already implied by PEP 484
can be looked at this way:

x = None # type: Optional[List[T]]
if arg is not None:
x = list(arg)
if other_arg is not None:
x.extend(arg)

It would be a strange typechecker indeed that handled that case
differently from the new spellings made possible by PEP 526:

x: Optional[List[T]] = None
if arg is not None:
x = list(arg)
if other_arg is not None:
x.extend(arg)

x: Optional[List[T]]
if arg is None:
x = None
else:
x = list(arg)
if other_arg is not None:
x.extend(arg)

x: Optional[List[T]]
if arg is not None:
x = list(arg)
if other_arg is not None:
x.extend(arg)
else:
x = None

Or from the semantics of PEP 484 parameter annotations:

def my_func(arg:Optional[List[T]], other_arg=None):
# other_arg is implicitly Optional[Any]
if arg is not None and other_arg is not None:
# Here, "arg" can be assumed to be List[T]
# while "other_arg" is Any
arg.extend(other_arg)

A self-consistent typechecker will either allow all of the above, or
prohibit all of the above, while a typechecker that *isn't*
self-consistent would be incredibly hard to use.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Greg Ewing]

On Sun, Sep 04, 2016 at 12:31:26PM +0100, Mark Shannon wrote:

As defined in PEP 526, I think that type 
annotations become a hindrance to type inference.


In Haskell-like languages, type annotations have no
ability to influence whether types can be inferred.
The compiler infers a type for everything, whether
you annotate or not. The annotations serve as
assertions about what the inferred types should be.
If they don't match, it means the programmer has
made a mistake somewhere.

I don't think it's possible for an annotation to
prevent the compiler from being able to infer a
type where it could have inferred one without the
annotation.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Guido van Rossum]

Everybody please stop panicking. PEP 526 does not make a stand on the
behavior of type checkers (other than deferring to PEP 484). If you
want to start a discussion about constraining type checkers please do
it over at python-ideas. There is no rush as type checkers are not
affected by the feature freeze.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

On Sun, Sep 4, 2016 at 9:13 PM, Ivan Levkivskyi  wrote:
> On 4 September 2016 at 19:59, Nick Coghlan  wrote:
[...]
>>
>> Similarly, it would be reasonable to say that these three snippets
>> should all be equivalent from a typechecking perspective:
>>
>> x = None # type: Optional[T]
>>
>> x: Optional[T] = None
>>
>> x: Optional[T]
>> x = None
>
>
> Nice idea, explicit is better than implicit.
>

How is it going to help that these are equivalent within one checker,
if the meaning may differ across checkers?

-- Koos


> --
> Ivan
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
>



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Nick Coghlan]

On 5 September 2016 at 04:13, Ivan Levkivskyi  wrote:
> On 4 September 2016 at 19:59, Nick Coghlan  wrote:
>> I suspect you'll have an easier time of it on that front if you
>> include some examples of dynamically typed code that a well-behaved
>> type-checker *must* report as correct Python code, such as:
>>
>> x: Optional[List[Any]]
>> # This is the type of "x" *after* the if statement, not *during* it
>> if arg is not None:
>> x = list(arg)
>> if other_arg is not None:
>> # A well-behaved typechecker should allow this due to
>> # the more specific initialisation in this particular branch
>> x.extend(other_arg)
>> else:
>> x = None
>
> There are very similar examples in PEP 484 (section on singletons in
> unions),
> we could just copy those or use this example,
> but I am sure Guido will not agree to word "must" (although "should" maybe
> possible :-)

"Should" would be fine by me :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Ivan Levkivskyi]

On 4 September 2016 at 19:59, Nick Coghlan  wrote:

Nick,

Thank you for good suggestions.


> I mostly agree, but the PEP still needs to address the fact that it's
> only a subset of the benefits that actually require new syntax, since
> it's that subset which provides the rationale for rejecting the use of
> a function based approach, while the rest provided the incentive to
> start looking for a way to replace the type comments.
>

I think I agree.


> I suspect you'll have an easier time of it on that front if you
> include some examples of dynamically typed code that a well-behaved
> type-checker *must* report as correct Python code, such as:
>
> x: Optional[List[Any]]
> # This is the type of "x" *after* the if statement, not *during* it
> if arg is not None:
> x = list(arg)
> if other_arg is not None:
> # A well-behaved typechecker should allow this due to
> # the more specific initialisation in this particular branch
> x.extend(other_arg)
> else:
> x = None


There are very similar examples in PEP 484 (section on singletons in
unions),
we could just copy those or use this example,
but I am sure Guido will not agree to word "must" (although "should" maybe
possible :-)


> Similarly, it would be reasonable to say that these three snippets
> should all be equivalent from a typechecking perspective:
>
> x = None # type: Optional[T]
>
> x: Optional[T] = None
>
> x: Optional[T]
> x = None
>

Nice idea, explicit is better than implicit.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Nick Coghlan]

On 5 September 2016 at 03:23, Ivan Levkivskyi  wrote:
> On 4 September 2016 at 18:43, Nick Coghlan  wrote:
>>
>> On 4 September 2016 at 21:32, Ivan Levkivskyi 
>> wrote:
>> > The first form still could be interpreted by type checkers
>> > as annotation for value (a cast to more precise type):
>> >
>> > variable = cast(annotation, value) # visually also looks similar
>>
>> I think a function based spelling needs to be discussed further, as it
>> seems to me that at least some of the goals of the PEP could be met
>> with a suitable definition of "cast" and "declare", with no syntactic
>> changes to Python. Specifically, consider:
>>
>> def cast(value, annotation):
>> return value
>>
>> def declare(annotation):
>> return object()
>
>
> Nick, If I understand you correctly, this idea is very similar to Undefined.
> It was proposed a year and half ago, when PEP 484 was discussed.

Not quite, as it deliberately doesn't create a singleton, precisely to
avoid the problems a new singleton creates - if you use declare() as
written, there's no way to a priori check for it at runtime (since
each call produces a new object), so you have to either get the
TypeError when you try to use it as whatever type it's supposed to be,
or else use a static checker to find cases where you try to use it
without initialising it properly first.

Folks can also put it behind an "if 0:" or "if typing.TYPE_CHECKING"
guard so it doesn't actually execute at runtime, and is only visible
to static analysis.

> At that time it was abandoned, it reappeared during the discussion
> of this PEP, but many people (including me) didn't like this,
> so that we decided to put it in the list of rejected ideas to this PEP.
>
> Some downsides of this approach:
>
> * People will start to expect Undefined (or whatever is returned by
> declare())
> everywhere (as in Javascript) even if we prohibit this.

Hence why I didn't use a singleton.

> * Some runtime overhead is still present: annotation gets evaluated
> at every call to cast, and many annotations involve creation of
> class objects (especially generics) that are very costly.

Hence the commentary about using an explicit guard to prevent
execution ("if 0:" in my post for the dead code elimination, although
"if typing.TYPE_CHECKING:" would be more self-explanatory).

> * Readability will be probably even worse than with comments:
> many types already have brackets and parens, so that two more form cast()
> is not good. Plus some noise of the if 0: that you mentioned, plus
> "cast" everywhere.

I mostly agree, but the PEP still needs to address the fact that it's
only a subset of the benefits that actually require new syntax, since
it's that subset which provides the rationale for rejecting the use of
a function based approach, while the rest provided the incentive to
start looking for a way to replace the type comments.

>> However, exploring this possibility still seems like a good idea to
>> me, as it should allow many of the currently thorny semantic questions
>> to be resolved, and a future syntax-only PEP for 3.7+ can just be
>> about defining syntactic sugar for semantics that can (by then)
>> already be expressed via appropriate initialisers.
>
> I think that motivation of the PEP is exactly opposite, this is why it has
> "Syntax" not "Semantics" in title. Also quoting Guido:
>
>> But I'm not in a hurry for that -- I'm only hoping to get the basic
>> syntax accepted by Python 3.6 beta 1 so that we can start using this
>> in 5 years from now rather than 7 years from now.
>
> I also think that semantics should be up to the type checkers.
> Maybe it is not a perfect comparison, but prohibiting all type semantics
> except one is like prohibiting all Python web frameworks except one.

It's the semantics that worry people though, and it's easy for folks
actively working on typecheckers to think it's just as easy for the
rest of us to make plausible assumptions about the kind of code that
well-behaved typecheckers are going to allow as it is for you. That's
not the case, which means folks get concerned, especially those
accustomed to instititutional environments where decisions about tool
use are still made by folks a long way removed from the day to day
experience of software development, rather than being delegated to the
engineering teams themselves.

I suspect you'll have an easier time of it on that front if you
include some examples of dynamically typed code that a well-behaved
type-checker *must* report as correct Python code, such as:

x: Optional[List[Any]]
# This is the type of "x" *after* the if statement, not *during* it
if arg is not None:
x = list(arg)
if other_arg is not None:
# A well-behaved typechecker should allow this due to
# the more specific initialisation in this particular branch
x.extend(other_arg)
else:
x = None

Re: [Python-Dev] Please reject or postpone PEP 526

[Ivan Levkivskyi]

On 4 September 2016 at 19:29, Nick Coghlan  wrote:

> So in this case, attempting to entirely defer specification of the
> semantics creates a significant risk of type checkers written on the
> assumption of C++ or Java style type declarations actively inhibiting
> the dynamism of Python code, suggesting that the PEP would be well
> advised to declare not only that the PEP 484 semantics are unchanged,
> but also that a typechecker that flags the example above as unsafe is
> wrong to do so.
>

I don't think that a dedicated syntax will increase the
risk more than the existing type comment syntax. Moreover,
mainstream type checkers (mypy, pytype, etc) are far
from C++ or Java, and as far as I know they are not going
to change semantics.

As I understand, the main point of Mark is that such syntax suggests
visually a variable annotation, more than a value annotation.
However, I think that the current behavior of type checkers will
have more influence on perception of people rather than a visual
appearance of annotation.

Anyway, I think it is worth adding an explicit statement to the PEP
that both interpretations are possible (maybe even add that value
semantics is inherent to Python). But I don't think that we should
*prohibit* something in the PEP.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Nick Coghlan]

On 4 September 2016 at 21:51, Steven D'Aprano  wrote:
> On Sun, Sep 04, 2016 at 12:31:26PM +0100, Mark Shannon wrote:
>
>> In other words PEP 484 specifically states that annotations are to help
>> with type inference. As defined in PEP 526, I think that type
>> annotations become a hindrance to type inference.
>
> I'm pretty sure that they don't.
>
> Have you used any languages with type inference? Any type-checkers? If
> so, can you give some actual concrete examples of how annotating a
> variable hinders type inference? It sounds like you are spreading FUD at
> the moment.

Steven, this kind of credential checking is uncalled for - Mark is
significantly more qualified than most of us to comment on this topic,
since he actually works on a shipping software quality analysis
product that does type inference on Python code (hence
https://semmle.com/semmle-analysis-now-includes-python/ ), and was
nominated as the BDFL-Delegate for PEP 484 because Guido trusted him
to critically review the proposal and keep any insurmountable problems
from getting through.

Getting accused of spreading FUD when a topic is just plain hard to
explain (due to the large expert/novice gap that needs to be bridged)
is one of the reasons python-dev and python-ideas can end up feeling
hostile to domain experts. We have the SIG system to help mitigate
that problem, but it's vastly preferable if such folks also feel their
expertise is welcomed on the main lists, rather than having it be
rejected as an inconvenient complication.

> The whole point of type annotations is that you use them to deliberately
> over-ride what the checker would infer (if it infers the wrong thing, or
> cannot infer anything). I cannot see how you conclude from this that
> type annotations will be a hindrance to type inference.

The problem arises for the "bare annotation" case, as that looks a
*lot* like traditional declarations in languages where initialisation
(which can specify a type) and assignment (which can't) are different
operations.

Consider this case:

if arg is not None:
x = list(arg)
# Type of "x" is inferred as List[Any] or something more specific here
if other_arg is not None:
# This is fine, we know "x" is a list at this point
x.extend(other_arg)
else:
x = None
# Type of "x" is inferred as type(None) here
# Type of "x" is inferred as Optional[List[Any]] from here on out

Now, consider that case with PEP 526:

x: Optional[List[Any]]
# Oops, this is the type of "x" *after* the if statement, not *during* it
if arg is not None:
x = list(arg)
if other_arg is not None:
# If we believe the type declaration here, this code will
(incorrectly) be flagged
# (as None has no "extend" method)
x.extend(other_arg)
else:
x = None

The "pre-declaration as documentation" proposal turns out to be
problematic in this case, as it misses the fact that different
branches of the if statement contribute different types to what
ultimately becomes a Union type for the rest of the code.

In order to cover the entire code block, we have to make the
pre-declaration match the state after the if statement, but then it's
overly broad for any *particular* branch.

So in this case, attempting to entirely defer specification of the
semantics creates a significant risk of type checkers written on the
assumption of C++ or Java style type declarations actively inhibiting
the dynamism of Python code, suggesting that the PEP would be well
advised to declare not only that the PEP 484 semantics are unchanged,
but also that a typechecker that flags the example above as unsafe is
wrong to do so.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Ivan Levkivskyi]

On 4 September 2016 at 18:43, Nick Coghlan  wrote:

> On 4 September 2016 at 21:32, Ivan Levkivskyi 
> wrote:
> > The first form still could be interpreted by type checkers
> > as annotation for value (a cast to more precise type):
> >
> > variable = cast(annotation, value) # visually also looks similar
>
> I think a function based spelling needs to be discussed further, as it
> seems to me that at least some of the goals of the PEP could be met
> with a suitable definition of "cast" and "declare", with no syntactic
> changes to Python. Specifically, consider:
>
> def cast(value, annotation):
> return value
>
> def declare(annotation):
> return object()
>

Nick, If I understand you correctly, this idea is very similar to Undefined.
It was proposed a year and half ago, when PEP 484 was discussed.

At that time it was abandoned, it reappeared during the discussion
of this PEP, but many people (including me) didn't like this,
so that we decided to put it in the list of rejected ideas to this PEP.

Some downsides of this approach:

* People will start to expect Undefined (or whatever is returned by
declare())
everywhere (as in Javascript) even if we prohibit this.

* Some runtime overhead is still present: annotation gets evaluated
at every call to cast, and many annotations involve creation of
class objects (especially generics) that are very costly.
Because of this overhead, such use of generics was prohibited in PEP 484:

x = Node[int]() # prohibited by PEP 484
x = Node() # type: Node[int] # this was allowed

* Readability will be probably even worse than with comments:
many types already have brackets and parens, so that two more form cast()
is not good. Plus some noise of the if 0: that you mentioned, plus
"cast" everywhere.

However, exploring this possibility still seems like a good idea to
> me, as it should allow many of the currently thorny semantic questions
> to be resolved, and a future syntax-only PEP for 3.7+ can just be
> about defining syntactic sugar for semantics that can (by then)
> already be expressed via appropriate initialisers.
>

I think that motivation of the PEP is exactly opposite, this is why it has
"Syntax" not "Semantics" in title. Also quoting Guido:

> But I'm not in a hurry for that -- I'm only hoping to get the basic
> syntax accepted by Python 3.6 beta 1 so that we can start using this
> in 5 years from now rather than 7 years from now.

I also think that semantics should be up to the type checkers.
Maybe it is not a perfect comparison, but prohibiting all type semantics
except one is like prohibiting all Python web frameworks except one.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

On Sun, Sep 4, 2016 at 7:43 PM, Nick Coghlan  wrote:
> On 4 September 2016 at 21:32, Ivan Levkivskyi  wrote:
>> The first form still could be interpreted by type checkers
>> as annotation for value (a cast to more precise type):
>>
>> variable = cast(annotation, value) # visually also looks similar
>
> I think a function based spelling needs to be discussed further, as it
> seems to me that at least some of the goals of the PEP could be met
> with a suitable definition of "cast" and "declare", with no syntactic
> changes to Python. Specifically, consider:
>
> def cast(value, annotation):
> return value
>

typing.cast already exists.

-- Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Nick Coghlan]

On 4 September 2016 at 21:32, Ivan Levkivskyi  wrote:
> The first form still could be interpreted by type checkers
> as annotation for value (a cast to more precise type):
>
> variable = cast(annotation, value) # visually also looks similar

I think a function based spelling needs to be discussed further, as it
seems to me that at least some of the goals of the PEP could be met
with a suitable definition of "cast" and "declare", with no syntactic
changes to Python. Specifically, consider:

def cast(value, annotation):
return value

def declare(annotation):
return object()

The idea here is that "cast" would be used as a hint to type checkers
to annotate an *expression*, with the runtime semantic impact being
exactly nil - the value just gets passed through unmodified.

Annotated initialisations would then look like:

from typing import cast
primes = cast([], List[int])

class Starship:
stats = cast({}, ClassVar[Dict[str, int]])

This preserves the relative annotation order of current type hints,
where the item being annotated (parameter, function declaration,
assignment statement) is on the left, and the annotation is on the
right.

In cases where the typechecker is able to infer a type for the
expression, it may complain here when there's a mismatch between the
type inference and the explicit declaration, so these would also be a
form of type assertion.

That leaves the case of declarations, where the aim is to provide a
preemptive declaration that all assignments to a particular variable
will include an implicit casting of the RHS. That would look like:

from typing import declare

captain = declare(str)

Until it left the scope, or saw a new target declaration, a
typechecker would then interpret future assignments to "captain" as if
they had been written:

captain = cast(RHS, str)

With the above definition, this would have the runtime consequence of
setting "captain" to a unique object() instance until the first
assignment took place. Both that assignment, and the runtime overhead
of evaluating the declaration, can be avoided by moving the
declaration into otherwise dead code:

if 0: captain = declare(str)

Considering the goals and problems listed in the PEP, this would be
sufficient to address many of them:

* These are real expressions, and hence will be highlighted appropriately
* declare() allows annotations of undefined variables (sort of)
* These annotations will be in the AST, just as function arguments,
rather than as custom nodes
* In situations where normal comments and type comments are used
together, it is difficult to distinguish them

For the other goals, the function-based approach may not help:

* For conditional branches, it's only arguably an improvement

if 0: my_var = declare(Logger)
if some_value:
my_var = function()
else:
my_var = another_function()

* Readability for typeshed might improve for module level and class
level declarations, but functions would likely want the leading "if
0:" noise for performance reasons

* Since the code generator remains uninvolved, this still wouldn't
allow annotation access at runtime (unless typing implemented some
sys._getframe() hacks in declare() and cast())

However, exploring this possibility still seems like a good idea to
me, as it should allow many of the currently thorny semantic questions
to be resolved, and a future syntax-only PEP for 3.7+ can just be
about defining syntactic sugar for semantics that can (by then)
already be expressed via appropriate initialisers.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

On Sun, Sep 4, 2016 at 3:43 PM, Steven D'Aprano  wrote:
[...]
> [steve@ando ~]$ cat test.py
> from typing import Optional
>
> def foo(x:Optional[int])->int:
> if x is None:
> return -1
> return x + 1
>
> def bar(x:Optional[int])->int:
> y = x  # the type of y must be inferred
> if y is None:
> return y + 1
> return len(y)
>
> [steve@ando ~]$ mypy --strict-optional test.py
> test.py: note: In function "bar":
> test.py:11: error: Unsupported operand types for + (None and "int")
> test.py:12: error: Argument 1 to "len" has incompatible type "int"; expected 
> "Sized"
>
>
> foo passes the type check; bar fails.
>

That's great. While mypy has nice features, these examples have little
to do with PEP 526 as they don't have variable annotations, not even
using comments.

For some reason, pip install --upgrade mypy fails for me at the
moment, but at least mypy version 0.4.1 does not allow this:

from typing import Callable

def foo(cond: bool, bar : Callable, baz : Callable) -> float:
if cond:
x = bar() # type: int
else:
x = baz() # type: float
return x / 2

and complains that

test.py:7: error: Name 'x' already defined". Maybe someone can confirm
this with a newer version.

Here,

def foo(cond: bool) -> float:
if cond:
x = 1
else:
x = 1.5
return x / 2

you get a different error:

test.py:5: error: Incompatible types in assignment (expression has
type "float", variable has type "int")

Maybe someone can confirm this with a newer version, but IIUC this is
still the case.

>> I want a checker to check my code and, with minimal annotations, give me
>> confidence that my code is correct
>
> Don't we all.
>

I would add *with minimal restrictions on how the code is supposed to
be written* for type checking to work. It's not at all obvious that
everyone thinks that way. Hence, the "Semantics for type checking"
thread on python-ideas.

-- Koos

>
>
> --
> Steve
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Ivan Levkivskyi]

On 4 September 2016 at 15:07, Steven D'Aprano  wrote:

> > PEP 526
> > variable: annotation = value
> >
> > Which reads very much as if the annotation refers to the variable.
>
> Since the PEP makes it clear that the two forms are to be treated the
> same, I think that whatever difference you think they have is not
> relevant. They are *defined* to mean the same thing.
>

Steve,
This has been discussed in the python/typing tracker. When you say "mean"
you are talking about semantics, but:

Me:
"""
The title of the PEP contains "Syntax", not "Semantics" because we don't
want to impose any new type semantics (apart from addition of ClassVar)
"""

Guido:
"""
I have nothing to add to what @ilevkivskyi 
said about the semantics of redefinition -- that is to be worked out
between type checkers.
(Much like PEP 484 doesn't specify how type checkers should behave -- while
it gives examples of suggested behavior, those examples are not normative,
and there are huge gray areas where the PEP doesn't give any guidance. It's
the same for PEP 526.)
"""

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Steven D'Aprano]

Referring to the alternative syntax forms:

# Proposed
x: int = func(value)

# Already accepted
x = func(value)  #type: int


On Sun, Sep 04, 2016 at 11:52:24AM +0100, Mark Shannon wrote:

> The key difference is in placement.
> PEP 484 style
> variable = value # annotation
> 
> Which reads to me as if the annotation refers to the value.

Both Guido and the PEP have stated that it doesn't refer to the value, 
but to the variable.

But what does it even mean to say that it refers to the value in the 
context of *static type-checking*? I know what it means in the context 
of dynamic type-checking, but I don't see how that has any relevance to 
a static checker.


I have seen a number of people commenting that the comment annotation 
"applies to the expression", but I don't understand what this is 
supposed to mean. How is that different from applying it to the 
variable? (That's not a rhetorical question.) Suppose I write this:

mylist = []
x = False or None or (mylist + [1])  #type: List[int]
pass  # stand-in for arbitrary code
x.append("definitely not an int")

Should the type-checker flag the call to x.append as an error? I hope we 
all agree that it should.

But it can only do that if it knows the type of the variable `x`. This 
is a *static* type-checker, it doesn't know what value x *actually* has 
at run-time because it isn't running at run-time. As far as the static 
checker is concerned, it can only flag that append as an error if it 
knows that `x` must be a list of ints.

If you distinguish the two cases:

"the expression `False or None or (mylist + [1])` is List[int]"

versus:

"the variable `x` is List[int]"

I don't even see what the first case could possible mean. But whatever 
it means, if it is different from the second case, then the type-checker 
is pretty limited in what it can do.



> PEP 526
> variable: annotation = value
> 
> Which reads very much as if the annotation refers to the variable.

Since the PEP makes it clear that the two forms are to be treated the 
same, I think that whatever difference you think they have is not 
relevant. They are *defined* to mean the same thing.




-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Steven D'Aprano]

On Sun, Sep 04, 2016 at 12:30:18PM +0100, Mark Shannon wrote:

> It would be a real shame if PEP 526 mandates against checkers doing as 
> good as job as possible. Forcing all uses of a variable to have the same 
> type is a major and, IMO crippling, limitation.

This is approaching FUD.

Guido has already stated that the section of the PEP which implied that 
*any* change of type of a variable would be a warning (not an error) is 
too strong:

https://mail.python.org/pipermail/python-dev/2016-September/146064.html

and indeed the relevant section of the PEP has already been changed:

Duplicate type annotations will be ignored. However, static type 
checkers may issue a warning for annotations of the same variable 
by a different type:

a: int
a: str  # Static type checker may or may not warn about this.


This PEP does not mandate any behaviour for type-checkers. It describes 
the syntax for type annotations in Python code. What type-checkers do 
with that information is up to them.



> E.g.
> def foo(x:Optional[int])->int:
> if x is None:
>return -1
> return x + 1
> 
> If the type of the *variable* 'x' is Optional[int] then 'return x + 1' 
> doesn't type check.

That makes no sense. Why wouldn't it type check?

It may be that some simple-minded type-checkers are incapable of 
checking that code because it is too complex. If so, that's a limitation 
of that specific checker, not of type-checkers in general. MyPy already 
can type-check that code. See below.


> If the type of the *parameter* 'x' is Optional[int] 
> then a checker can readily verify the above code.

This makes even less sense, since the parameter "x" is, of course, 
precisely the same as the variable "x".

Here's MyPy in action, successfully checking code that you state can't 
be checked:

[steve@ando ~]$ cat test.py
from typing import Optional

def foo(x:Optional[int])->int:
if x is None:
return -1
return x + 1

def bar(x:Optional[int])->int:
y = x  # the type of y must be inferred
if y is None:
return y + 1
return len(y)

[steve@ando ~]$ mypy --strict-optional test.py
test.py: note: In function "bar":
test.py:11: error: Unsupported operand types for + (None and "int")
test.py:12: error: Argument 1 to "len" has incompatible type "int"; expected 
"Sized"


foo passes the type check; bar fails.


> I want a checker to check my code and, with minimal annotations, give me 
> confidence that my code is correct.

Don't we all.



-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

On Sun, Sep 4, 2016 at 1:52 PM, Mark Shannon  wrote:
[...]
>
> The key difference is in placement.
> PEP 484 style
> variable = value # annotation
>
> Which reads to me as if the annotation refers to the value.
> PEP 526
> variable: annotation = value
>
> Which reads very much as if the annotation refers to the variable.
> That is a change in terms of semantics and a change for the worse, in terms
> of expressibility.
>

You have probably noticed this already, but in the semantics which I
have now explained more precisely on python-ideas

https://mail.python.org/pipermail/python-ideas/2016-September/042076.html

an annotation like

variable: annotation = value

is a little closer to an expression annotation. I.e. it does not say
that 'variable' should *always* have the type given by 'annotation'.

-- Koos

>
> Cheers,
> Mark.
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Ivan Levkivskyi]

On 4 September 2016 at 13:30, Mark Shannon  wrote:

> It would be a real shame if PEP 526 mandates against checkers doing as
> good as job as possible. Forcing all uses of a variable to have the same
> type is a major and, IMO crippling, limitation.
>
> E.g.
> def foo(x:Optional[int])->int:
> if x is None:
>return -1
> return x + 1
>
> If the type of the *variable* 'x' is Optional[int] then 'return x + 1'
> doesn't type check. If the type of the *parameter* 'x' is Optional[int]
> then a checker can readily verify the above code.
>

Mark,

First, in addition to the quote from my previous e-mail, I would like to
show another quote from PEP 526
"This PEP does not require type checkers to change their type checking
rules. It merely provides a more readable syntax to replace type comments"

Second, almost exactly your example has been added to PEP 484:

class Reason(Enum):
timeout = 1
error = 2

def process(response: Union[str, Reason] = '') -> str:
if response is Reason.timeout:
return 'TIMEOUT'
elif response is Reason.error:
return 'ERROR'
else:
# response can be only str, all other possible values exhausted
return 'PROCESSED: ' + response

I think mypy either already supports this or will support very soon (and
the same for Optional)

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Steven D'Aprano]

On Sun, Sep 04, 2016 at 12:31:26PM +0100, Mark Shannon wrote:

> In other words PEP 484 specifically states that annotations are to help 
> with type inference. As defined in PEP 526, I think that type 
> annotations become a hindrance to type inference.

I'm pretty sure that they don't.

Have you used any languages with type inference? Any type-checkers? If 
so, can you give some actual concrete examples of how annotating a 
variable hinders type inference? It sounds like you are spreading FUD at 
the moment.

The whole point of type annotations is that you use them to deliberately 
over-ride what the checker would infer (if it infers the wrong thing, or 
cannot infer anything). I cannot see how you conclude from this that 
type annotations will be a hindrance to type inference.

If you don't want to declare the type of a variable, simply DON'T 
declare the type, and let the checker infer whatever it can (which may 
be nothing, or may be the wrong type).



-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Mark Shannon]

On 04/09/16 11:57, Chris Angelico wrote:

On Sun, Sep 4, 2016 at 8:52 PM, Mark Shannon  wrote:

The key difference is in placement.
PEP 484 style
variable = value # annotation

Which reads to me as if the annotation refers to the value.
PEP 526
variable: annotation = value

Which reads very much as if the annotation refers to the variable.
That is a change in terms of semantics and a change for the worse, in terms
of expressibility.


So what you have is actually a change in *implication* (since the PEP
doesn't stipulate semantics); and the old way (the comment) implies
something contrary to the semantics of at least one of the type
checkers that uses it (MyPy).
Do we really want to make a major, irrevocable change to the language 
just because MyPy does something? MyPy is very far from complete (it 
doesn't even support Optional types yet).


Are there any current type checkers that

actually do associate that with the value? That is, to have "variable
= func() # str" indicate the same type check as "def func() -> str"?
If not, this is a strong argument in favour of the PEP, since it would
synchronize the syntax with the current best-of-breed checkers.
I believe pytype uses value, rather the variable, tracking. It is thus 
more precise. Of course, it is more of an inferencer than a checker.
We (semmle.com) do precise value tracking to infer a lot of "type" 
errors in unannotated code (as the vast majority of Python code is).


It would be a real shame if PEP 526 mandates against checkers doing as 
good as job as possible. Forcing all uses of a variable to have the same 
type is a major and, IMO crippling, limitation.


E.g.
def foo(x:Optional[int])->int:
if x is None:
   return -1
return x + 1

If the type of the *variable* 'x' is Optional[int] then 'return x + 1' 
doesn't type check. If the type of the *parameter* 'x' is Optional[int] 
then a checker can readily verify the above code.


I want a checker to check my code and, with minimal annotations, give me 
confidence that my code is correct.



Cheers,
Mark.



ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/mark%40hotpy.org


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Ivan Levkivskyi]

On 4 September 2016 at 12:52, Mark Shannon  wrote:

> The key difference is in placement.
> PEP 484 style
> variable = value # annotation
>
> Which reads to me as if the annotation refers to the value.
> PEP 526
> variable: annotation = value
>
> Which reads very much as if the annotation refers to the variable.
> That is a change in terms of semantics and a change for the worse, in
> terms of expressibility.


I still think it is better to leave the decision to type checkers.
The proposed syntax allows two forms:

variable: annotation = value

and

variable: annotation

The first form still could be interpreted by type checkers
as annotation for value (a cast to more precise type):

variable = cast(annotation, value) # visually also looks similar

and PEP says that annotations "are intended to help with type inference in
complex cases".
Such interpretation could be useful for function local variables
(the implementation is also optimised for such use case).
While the second form (without value)
indeed looks like annotation of variable, and is equivalent to:

__annotations__['variable'] = annotation

This form could be useful for annotating instance/class variables.
Or just for documentation (I really like this).

In addition, expression annotations are allowed

expression: annotation [= value]

This form is not used by 3rd party tools (as far as I know) but one of the
possible use cases
could be "check-points" for values:

somedict[somefunc(somevar)]: annotation # type checker could flag this if
something went wrong.

Finally, I would like to reiterate, both interpretations (annotating value
vs annotating variable)
are possible and we (OK at least me, but it looks like Guido also agree)
don't want to force 3rd party tools to use only one of those.

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Mark Shannon]

On 02/09/16 20:33, Guido van Rossum wrote:

On Fri, Sep 2, 2016 at 10:47 AM, Steve Dower  wrote:

"I'm not seeing what distinction you think you are making here. What
distinction do you see between:

x: int = func(value)

and

x = func(value)  # type: int"

Not sure whether I agree with Mark on this particular point, but the
difference I see here is that the first describes what types x may ever
contain, while the latter describes what type of being assigned to x right
here. So one is a variable annotation while the other is an expression
annotation.


But that's not what type comments mean! They don't annotate the
expression. They annotate the variable. The text in PEP 484 that
introduces them is clear about this (it never mentions expressions,
only variables).


In PEP 484, the section on type comments says:
(Quoting verbatim)
"""
No first-class syntax support for explicitly marking variables as being 
of a specific type is added by this PEP. To help with type inference in 
complex cases, a comment of the following format may be used...

"""

Some mentions of the type of a variable are made in other places in the 
PEP, but those were all added *after* I had approved the PEP.


In other words PEP 484 specifically states that annotations are to help 
with type inference. As defined in PEP 526, I think that type 
annotations become a hindrance to type inference.


Cheers,
Mark.




Personally, I prefer expression annotations over variable annotations, as
there are many other languages I'd prefer if variable have fixed types (e.g.
C++, where I actually enjoy doing horrible things with implicit casting ;)
).

Variable annotations appear to be inherently restrictive, so either we need
serious clarification as to why they are not, or they actually are and we
ought to be more sure that it's the direction we want the language to go.


At runtime the variable annotations are ignored. And a type checker
will only ask for them when it cannot infer the type. So I think we'll
be fine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Chris Angelico]

On Sun, Sep 4, 2016 at 8:52 PM, Mark Shannon  wrote:
> The key difference is in placement.
> PEP 484 style
> variable = value # annotation
>
> Which reads to me as if the annotation refers to the value.
> PEP 526
> variable: annotation = value
>
> Which reads very much as if the annotation refers to the variable.
> That is a change in terms of semantics and a change for the worse, in terms
> of expressibility.

So what you have is actually a change in *implication* (since the PEP
doesn't stipulate semantics); and the old way (the comment) implies
something contrary to the semantics of at least one of the type
checkers that uses it (MyPy). Are there any current type checkers that
actually do associate that with the value? That is, to have "variable
= func() # str" indicate the same type check as "def func() -> str"?
If not, this is a strong argument in favour of the PEP, since it would
synchronize the syntax with the current best-of-breed checkers.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Mark Shannon]

On 02/09/16 19:19, Steven D'Aprano wrote:

On Fri, Sep 02, 2016 at 10:47:41AM -0700, Steve Dower wrote:

"I'm not seeing what distinction you think you are making here. What
distinction do you see between:

x: int = func(value)

and

x = func(value)  #type: int"

Not sure whether I agree with Mark on this particular point, but the
difference I see here is that the first describes what types x may
ever contain, while the latter describes what type of being assigned
to x right here. So one is a variable annotation while the other is an
expression annotation.


Ultimately Python is a dynamically typed language, and that's not
changing. This means types are fundamentally associated with *values*,
not *variables* (names). But in practice, you can go a long way by
pretending that it is the variable that carries the type. That's the
point of the static type checker: if you see that x holds an int here,
then assume (unless told differently) that x should always be an int.
Because in practice, most exceptions to that are due to bugs, or at
least sloppy code.

Of course, it is up to the type checker to decide how strict it wants to
be, whether to treat violations as a warning or a error, whether to
offer the user a flag to set the behaviour, etc. None of this is
relevant to the PEP. The PEP only specifies the syntax, leaving
enforcement or non-enforcement to the checker, and it says:

PEP 484 introduced type hints, a.k.a. type annotations. While its
main focus was function annotations, it also introduced the notion
of type comments to annotate VARIABLES [emphasis added]


If I recall, Guido and I agreed to differ on that point. We still do, it 
seems. We did manage to agree on the syntax though.




not expressions. And:

This PEP aims at adding syntax to Python for annotating the types
of variables and attributes, instead of expressing them through
comments

which to me obviously implies that the two ways (type comment, and
variable type hint) are intended to be absolutely identical in
semantics, at least as far as the type-checker is concerned.


The key difference is in placement.
PEP 484 style
variable = value # annotation

Which reads to me as if the annotation refers to the value.
PEP 526
variable: annotation = value

Which reads very much as if the annotation refers to the variable.
That is a change in terms of semantics and a change for the worse, in 
terms of expressibility.


Cheers,
Mark.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Mark Shannon]

On 02/09/16 19:04, Steven D'Aprano wrote:

On Fri, Sep 02, 2016 at 08:10:24PM +0300, Koos Zevenhoven wrote:


A good checker should be able to infer that x is a union type at the
point that it's passed to spam, even without the type annotation. For
example:

def eggs(cond:bool):
if cond:
x = 1
else:
x = 1.5
spam(x)   # a good type checker infers that x is of type Union[int, float]


Oh I really hope not. I wouldn't call that a *good* type checker. I
would call that a type checker that is overly permissive.
Why would that be overly permissive? It infers the most precise type 
possible.




Maybe you think that it's okay because ints and floats are somewhat
compatible. But suppose I wrote:

if cond:
x = HTTPServer(*args)
else:
x = 1.5

Would you want the checker to infer Union[HTTPServer, float]? I
wouldn't. I would want the checker to complain that the two branches of
the `if` result in different types for x. If I really mean it, then I
can give a type-hint.
Yes, the checker would infer that the type of x (strictly, all uses of x 
that are  defined by these definitions) is Union[HTTPServer, float].


You example is incomplete, what do you do with x?
If you pass x to a function that takes  Union[HTTPServer, float] then 
there is no error.

If you pass it to a function that takes a number then you get an error:
"Cannot use HTTPServer (from line 2) as Number (line ...)"
as one would expect.

When it comes to checkers, people hate false positives. Flagging correct 
code as erroneous because it is bad 'style' is really unpopular.




In any case, this PEP isn't about specifying when to declare variable
types, it is for picking syntax. Do you have a better idea for variable
syntax?


No. I think that defining the type of variables, rather than expressions 
is a bad idea.


Cheers,
Mark.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Guido van Rossum]

On Fri, Sep 2, 2016 at 10:47 AM, Steve Dower  wrote:
> "I'm not seeing what distinction you think you are making here. What
> distinction do you see between:
>
> x: int = func(value)
>
> and
>
> x = func(value)  # type: int"
>
> Not sure whether I agree with Mark on this particular point, but the
> difference I see here is that the first describes what types x may ever
> contain, while the latter describes what type of being assigned to x right
> here. So one is a variable annotation while the other is an expression
> annotation.

But that's not what type comments mean! They don't annotate the
expression. They annotate the variable. The text in PEP 484 that
introduces them is clear about this (it never mentions expressions,
only variables).

> Personally, I prefer expression annotations over variable annotations, as
> there are many other languages I'd prefer if variable have fixed types (e.g.
> C++, where I actually enjoy doing horrible things with implicit casting ;)
> ).
>
> Variable annotations appear to be inherently restrictive, so either we need
> serious clarification as to why they are not, or they actually are and we
> ought to be more sure that it's the direction we want the language to go.

At runtime the variable annotations are ignored. And a type checker
will only ask for them when it cannot infer the type. So I think we'll
be fine.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Stefan Krah]

[Replying to  Steve Dower]
On Sat, Sep 03, 2016 at 04:19:13AM +1000, Steven D'Aprano wrote:
> On Fri, Sep 02, 2016 at 10:47:41AM -0700, Steve Dower wrote:
> > "I'm not seeing what distinction you think you are making here. What 
> > distinction do you see between:
> > 
> > x: int = func(value)
> > 
> > and
> > 
> > x = func(value)  #type: int"
> > 
> > Not sure whether I agree with Mark on this particular point, but the 
> > difference I see here is that the first describes what types x may 
> > ever contain, while the latter describes what type of being assigned 
> > to x right here. So one is a variable annotation while the other is an 
> > expression annotation.

I see it differently, but I'm quite used to OCaml:

# let f () =
let x : int = 10 in
let x : float = 320.0 in
  x;;
Warning 26: unused variable x.
val f : unit -> float = 
# f();;
- : float = 320.


Like in Python, in OCaml variables can be rebound and indeed have different
types with different explicit type constraints.


Expressions can also be annotated, but require parentheses (silly example):

# let x = (10 * 20 : int);;
val x : int = 200



So I'm quite happy with the proposed syntax in the PEP, perhaps the
parenthesized expression annotations could also be added.  But these
are only very rarely needed.


Stefan Krah



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Steven D'Aprano]

On Fri, Sep 02, 2016 at 10:47:41AM -0700, Steve Dower wrote:
> "I'm not seeing what distinction you think you are making here. What 
> distinction do you see between:
> 
> x: int = func(value)
> 
> and
> 
> x = func(value)  #type: int"
> 
> Not sure whether I agree with Mark on this particular point, but the 
> difference I see here is that the first describes what types x may 
> ever contain, while the latter describes what type of being assigned 
> to x right here. So one is a variable annotation while the other is an 
> expression annotation.

Ultimately Python is a dynamically typed language, and that's not 
changing. This means types are fundamentally associated with *values*, 
not *variables* (names). But in practice, you can go a long way by 
pretending that it is the variable that carries the type. That's the 
point of the static type checker: if you see that x holds an int here, 
then assume (unless told differently) that x should always be an int. 
Because in practice, most exceptions to that are due to bugs, or at 
least sloppy code.

Of course, it is up to the type checker to decide how strict it wants to 
be, whether to treat violations as a warning or a error, whether to 
offer the user a flag to set the behaviour, etc. None of this is 
relevant to the PEP. The PEP only specifies the syntax, leaving 
enforcement or non-enforcement to the checker, and it says:

PEP 484 introduced type hints, a.k.a. type annotations. While its 
main focus was function annotations, it also introduced the notion
of type comments to annotate VARIABLES [emphasis added]

not expressions. And:

This PEP aims at adding syntax to Python for annotating the types
of variables and attributes, instead of expressing them through
comments

which to me obviously implies that the two ways (type comment, and 
variable type hint) are intended to be absolutely identical in 
semantics, at least as far as the type-checker is concerned.

(They will have different semantics at runtime: the comment is just a 
comment, while the type hint will set an __annotations__ mapping.)

But perhaps the PEP needs to make it explicit that they are to be 
treated exactly the same.


-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Steven D'Aprano]

On Fri, Sep 02, 2016 at 08:10:24PM +0300, Koos Zevenhoven wrote:

> A good checker should be able to infer that x is a union type at the
> point that it's passed to spam, even without the type annotation. For
> example:
> 
> def eggs(cond:bool):
> if cond:
> x = 1
> else:
> x = 1.5
> spam(x)   # a good type checker infers that x is of type Union[int, float]

Oh I really hope not. I wouldn't call that a *good* type checker. I 
would call that a type checker that is overly permissive.

Maybe you think that it's okay because ints and floats are somewhat 
compatible. But suppose I wrote:

if cond:
x = HTTPServer(*args)
else:
x = 1.5

Would you want the checker to infer Union[HTTPServer, float]? I 
wouldn't. I would want the checker to complain that the two branches of 
the `if` result in different types for x. If I really mean it, then I 
can give a type-hint.

In any case, this PEP isn't about specifying when to declare variable 
types, it is for picking syntax. Do you have a better idea for variable 
syntax?



-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Ivan Levkivskyi]

On 2 September 2016 at 17:59, Guido van Rossum  wrote:
> On Fri, Sep 2, 2016 at 6:47 AM, Mark Shannon  wrote:
> > Quoting from the PEP:
> > ```
> > a: int
> > a: str # Static type checker will warn about this.
> > ```
> > In other words, it is illegal for a checker to split up the variable,
even
> > though it is straightforward to do so.
>
> One of my co-authors has gone too far here. The intent is not to
legislate what should happen in this case but to leave it to the checker.
In mypy, the equivalent syntax using type comments is currently indeed
rejected, but we're considering a change here (
https://github.com/python/mypy/issues/1174). The PEP 526 syntax will not
make a difference here.

If I remember correctly, I added this example. At that time the intention
was to propose to "loosen" the behaviour of type checkers (note it is a
warning, not an error like in mypy). But now I agree with Guido that we
should be even more liberal. We could left this to type checker to decide
what to do (they could even have options like -Werror or -Wignore).

--
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Please reject or postpone PEP 526

[Steve Dower]

"I'm not seeing what distinction you think you are making here. What 
distinction do you see between:

x: int = func(value)

and

x = func(value)  #type: int"

Not sure whether I agree with Mark on this particular point, but the difference 
I see here is that the first describes what types x may ever contain, while the 
latter describes what type of being assigned to x right here. So one is a 
variable annotation while the other is an expression annotation.

Personally, I prefer expression annotations over variable annotations, as there 
are many other languages I'd prefer if variable have fixed types (e.g. C++, 
where I actually enjoy doing horrible things with implicit casting ;) ).

Variable annotations appear to be inherently restrictive, so either we need 
serious clarification as to why they are not, or they actually are and we ought 
to be more sure that it's the direction we want the language to go.

Cheers,
Steve

Top-posted from my Windows Phone

-Original Message-
From: "Steven D'Aprano" <st...@pearwood.info>
Sent: ‎9/‎2/‎2016 9:43
To: "python-dev@python.org" <python-dev@python.org>
Subject: Re: [Python-Dev] Please reject or postpone PEP 526

Hi Mark,

I'm going to trim your post drastically, down to the bare essentials, in 
order to keep this already long post down to a manageable size.


On Fri, Sep 02, 2016 at 02:47:00PM +0100, Mark Shannon wrote:

[...]
> With type comments, this is intuitively correct and should type check:
> def eggs(cond:bool):
> if cond:
> x = None
> else:
> x = [] # type: List[int]
> spam(x)  # Here we can infer the type of x

It isn't correct. You've declared something to be a list of ints, but 
assigned a value None to it! How is that not an error?

The error is more obvious if you swap the order of assignments:

if cond:
x = [] # type: List[int]
else:
x = None


MyPy currently requires the experimental --strict-optional flag to 
detect this error:

[steve@ando ~]$ mypy --strict-optional test.py
test.py: note: In function "eggs":
test.py:10: error: Incompatible types in assignment (expression has type 
None, variable has type List[int])


Changing that from comment syntax to (proposed) Python syntax will not 
change that. There is no semantic difference to the type checker 
between 

x = []  #type: List[int]

and

x: List[int] = []

and any type checker will have to treat them identically.


> With PEP 526 we loose the ability to infer types.

On re-reading the PEP, I have just noticed that nowhere does it 
explicitly state that checkers are still expected to perform type 
inference. However, this is the companion to PEP 484, which states:

Type comments

No first-class syntax support for explicitly marking variables
as being of a specific type is added by this PEP. TO HELP WITH 
TYPE INFERENCE IN COMPLEX CASES, a comment of the following
format may be used: [...]

(Emphasis added.) So the intention is that, regardless of whether you 
use a type annotation using a comment or the proposed syntax, that is 
intended to *help* the checker perform inference, not to replace it.

Perhaps this PEP should include an explicit note to that end.



[...]
> So we need to use a more complex type
> def eggs(cond:bool):
> x: Optional[List[int]]
> if cond:
> x = None # Now legal
> else:
> x: = []
> spam(x)
> 
> I don't think this improves readability.

Maybe not, but it sure improves *correctness*.

A human reader might be able to logically deduce that x = None and 
x = [] are both valid, given that spam() takes either a list or None, 
but I'm not sure if that level of intelligence is currently possible in 
type inference. (Maybe it is, and MyPy simply doesn't support it yet.) 
So it may be that this is a case where you do have to apply an explicit 
type hint, using *either* a type comment or this new proposed syntax:

x: Optional[List[int]]
if cond:
x = None
else:
x = []

should be precisely the same as:

if cond:
x = None #type: Optional[List[int]]
else:
x = []


> Quoting from the PEP:
> ```
> a: int
> a: str # Static type checker will warn about this.
> ```
> In other words, it is illegal for a checker to split up the variable, 
> even though it is straightforward to do so.

No, it is a *warning*, not an error.

Remember, the Python interpreter itself won't care. The type checker is 
optional and not part of the interpreter. You can still write code like:

a = 1
do_something(a)
a = "string"
do_another(a)

and Python will be happy. But if you do run a type checker, it should 
warn you that you're changing types, as that suggests the possibility of 
a type error. (And it also goes against a lot of people's style 
guidelines.)


> We should be free to add extra var

Re: [Python-Dev] Please reject or postpone PEP 526

[Koos Zevenhoven]

I agree about some concerns and disagree about several. I see most use
for class/instance attribute annotations, both for type checking and
for other uses. I'm least sure about their syntax and about
annotations in functions or at module level in the proposed form.

Below some comments:

On Fri, Sep 2, 2016 at 4:47 PM, Mark Shannon  wrote:
> Hi everyone,
>
> I think we should reject, or at least postpone PEP 526.
>
> PEP 526 represents a major change to the language, however there are, I
> believe, a number of technical flaws with the PEP.

[...]
>
> In many cases it makes it more effort than type comments
> 
>
> Type hints should be as easy to use as possible, and that means pushing as
> much work as possible onto the checker, and not burdening the programmer.
>
> Attaching type hints to variables, rather than expressions, reduces the
> potential for inference. This makes it harder for programmer, but easier for
> the checker, which is the wrong way around.

The more you infer types, the less you check them. It's up to the
programmers to choose the amount of annotation.

> For example,, given a function:
> def spam(x: Optional[List[int]])->None: ...
>
> With type comments, this is intuitively correct and should type check:
> def eggs(cond:bool):
> if cond:
> x = None
> else:
> x = [] # type: List[int]
> spam(x)  # Here we can infer the type of x
>
> With PEP 526 we loose the ability to infer types.
> def eggs(cond:bool):
> if cond:
> x = None # Not legal due to type declaration below
> else:
>x: List[int] = []
> spam(x)

I'm also a little worried about not being able to reannotate a name.

>
> So we need to use a more complex type
> def eggs(cond:bool):
> x: Optional[List[int]]
> if cond:
> x = None # Now legal
> else:
> x: = []
> spam(x)
>

A good checker should be able to infer that x is a union type at the
point that it's passed to spam, even without the type annotation. For
example:

def eggs(cond:bool):
if cond:
x = 1
else:
x = 1.5
spam(x)   # a good type checker infers that x is of type Union[int, float]

Or with annotations:

def eggs(cond:bool):
if cond:
x : int = foo() # foo may not have a return type hint
else:
x : float = bar() # bar may not have a return type hint
spam(x)   # a good type checker infers that x is of type Union[int, float]


[...]
> It limits the use of variables
> ==
>
> In Python a name (variable) is just a binding that refers to an object.
> A name only exists in a meaningful sense once an object has been assigned to
> it. Any attempt to use that name, without an object bound to it, will result
> in a NameError.
>

IIUC, that would still be the case after PEP 526.

[...]
>
> We should be free to add extra variables, whenever we choose, for clarity.
> For example,
> total = foo() - bar()
> should not be treated differently from:
> revenue = foo()
> tax = bar()
> total = revenue - tax
>
> If types are inferred, there is no problem.
> However, if they must be declared, then the use of meaningfully named
> variables is discouraged.
>

Who says they *must* be declared?

[...]
> It is premature
> ===
>
> There are still plenty of issues to iron out w.r.t. PEP 484 types. I don't
> think we should be adding more syntax, until we have a *precise* idea of
> what is required.
>
> PEP 484 states:
> "If type hinting proves useful in general, a syntax for typing variables may
> be provided in a future Python version."
> Has it proved useful in general? I don't think it has. Maybe it will in
> future, but it hasn't yet.
>

Yeah, I hope someone has enough experience to know whether this is the
right thing for Python as a whole.

> It seems confused about class attributes and instance attributes
> 
>
> The PEP also includes a section of how to define class attributes and
> instance attributes. It seems that everything needs to be defined in the
> class scope, even it is not an attribute of the class, but of its instances.
> This seems confusing, both to human reader and machine analyser.

I don't see the problem here, isn't that how it's usually done in
strongly typed languages? And methods are defined in the class scope
too (well yes, they do also exist in the class namespace, but
anyway...).

But I agree in the sense that the proposed syntax is far from explicit
about these being instance attributes by default.

> Example from PEP 526:
>
> class Starship:
>
> captain: str = 'Picard'
> damage: int
> stats: ClassVar[Dict[str, int]] = {}
>
> def __init__(self, damage: int, captain: str = None):
> self.damage = damage
> if captain:
> self.captain = captain  # Else keep the default
>
> With type hints as they currently 

Re: [Python-Dev] Please reject or postpone PEP 526

[Sven R. Kunze]

Hi Mark,

I agree with you about postponing.

Not so much because of the issues you mentioned. Those all seem 
resolvable to me and mostly concerns type checkers, linters and coding 
styles not Python itself. However, I also don't like the rushing through 
as if this beta were the only chance to get it into Python.


Python haven't had variable annotations for decades until now and Python 
will still be there in 10 decades or so. Thus, 2 years more to wait and 
to hammer out the details does not seem much compared to the entire 
lifetime of this language.


The PEP also remains silent about when to use annotations. Compared to 
recent additions like f-strings or async, it's completely clear when to 
use it. However for variable annotations it's not (all variables? most 
used variables? least used variables?). So, I also agree with you that 
improving type checkers is the better way than adding all static type 
annotations all over the place. Python is dynamic and also types should 
be as dynamic and redundant-free as possible. Thus, some guidance would 
be great here.


Cheers,
Sven


On 02.09.2016 15:47, Mark Shannon wrote:

Hi everyone,

I think we should reject, or at least postpone PEP 526.

PEP 526 represents a major change to the language, however there are, 
I believe, a number of technical flaws with the PEP.


It is probable that with significant revisions it can be a worthwhile 
addition to the language, but that cannot happen in time for 3.6 beta 
1 (in 11 days).


PEP 526 claims to be an extension of PEP 484, but I don't think that 
is entirely correct.
PEP 484 was primarily about defining the semantics of pre-existing 
syntax. PEP 526 is about adding new syntax.
Historically the bar for adding new syntax has been set very high. I 
don't think that PEP 526, in its current form, reaches that bar.


Below is a list of the issues I have with the PEP as it stands.

In many cases it makes it more effort than type comments


Type hints should be as easy to use as possible, and that means 
pushing as much work as possible onto the checker, and not burdening 
the programmer.


Attaching type hints to variables, rather than expressions, reduces 
the potential for inference. This makes it harder for programmer, but 
easier for the checker, which is the wrong way around.


For example,, given a function:
def spam(x: Optional[List[int]])->None: ...

With type comments, this is intuitively correct and should type check:
def eggs(cond:bool):
if cond:
x = None
else:
x = [] # type: List[int]
spam(x)  # Here we can infer the type of x

With PEP 526 we loose the ability to infer types.
def eggs(cond:bool):
if cond:
x = None # Not legal due to type declaration below
else:
   x: List[int] = []
spam(x)

So we need to use a more complex type
def eggs(cond:bool):
x: Optional[List[int]]
if cond:
x = None # Now legal
else:
x: = []
spam(x)

I don't think this improves readability.
Whether this is an acceptable change is debatable, but it does need 
some debate.


It limits the use of variables
==

In Python a name (variable) is just a binding that refers to an object.
A name only exists in a meaningful sense once an object has been 
assigned to it. Any attempt to use that name, without an object bound 
to it, will result in a NameError.


PEP 526 makes variables more than just bindings, as any rebinding must 
conform to the given type. This looses us some of the dynamism for 
which we all love Python.


Quoting from the PEP:
```
a: int
a: str # Static type checker will warn about this.
```
In other words, it is illegal for a checker to split up the variable, 
even though it is straightforward to do so.


However, without the type declarations,
```
a = 1
a = "Hi"
```
is just fine. Useless, but fine.

We should be free to add extra variables, whenever we choose, for 
clarity. For example,

total = foo() - bar()
should not be treated differently from:
revenue = foo()
tax = bar()
total = revenue - tax

If types are inferred, there is no problem.
However, if they must be declared, then the use of meaningfully named 
variables is discouraged.


[A note about type-inference:
Type inference is not a universal panacea, but it can make life a lot 
easier for programmers in statically type languages.
Languages like C# use local type inference extensively and it means 
that many variables often do not need their type declared. We should 
take care not to limit the ability of checkers to infer values and 
types and make programmers' lives easier.
Within a function, type inference is near perfect, failing only 
occasionally for some generic types.
One place where type inference definitely breaks down is across calls, 
which is why PEP 484 is necessary.

]

It is premature
===

There are still plenty of issues to iron out w.r.t. PEP 484 

Re: [Python-Dev] Please reject or postpone PEP 526

[Steven D'Aprano]

Hi Mark,

I'm going to trim your post drastically, down to the bare essentials, in 
order to keep this already long post down to a manageable size.


On Fri, Sep 02, 2016 at 02:47:00PM +0100, Mark Shannon wrote:

[...]
> With type comments, this is intuitively correct and should type check:
> def eggs(cond:bool):
> if cond:
> x = None
> else:
> x = [] # type: List[int]
> spam(x)  # Here we can infer the type of x

It isn't correct. You've declared something to be a list of ints, but 
assigned a value None to it! How is that not an error?

The error is more obvious if you swap the order of assignments:

if cond:
x = [] # type: List[int]
else:
x = None


MyPy currently requires the experimental --strict-optional flag to 
detect this error:

[steve@ando ~]$ mypy --strict-optional test.py
test.py: note: In function "eggs":
test.py:10: error: Incompatible types in assignment (expression has type 
None, variable has type List[int])


Changing that from comment syntax to (proposed) Python syntax will not 
change that. There is no semantic difference to the type checker 
between 

x = []  #type: List[int]

and

x: List[int] = []

and any type checker will have to treat them identically.


> With PEP 526 we loose the ability to infer types.

On re-reading the PEP, I have just noticed that nowhere does it 
explicitly state that checkers are still expected to perform type 
inference. However, this is the companion to PEP 484, which states:

Type comments

No first-class syntax support for explicitly marking variables
as being of a specific type is added by this PEP. TO HELP WITH 
TYPE INFERENCE IN COMPLEX CASES, a comment of the following
format may be used: [...]

(Emphasis added.) So the intention is that, regardless of whether you 
use a type annotation using a comment or the proposed syntax, that is 
intended to *help* the checker perform inference, not to replace it.

Perhaps this PEP should include an explicit note to that end.



[...]
> So we need to use a more complex type
> def eggs(cond:bool):
> x: Optional[List[int]]
> if cond:
> x = None # Now legal
> else:
> x: = []
> spam(x)
> 
> I don't think this improves readability.

Maybe not, but it sure improves *correctness*.

A human reader might be able to logically deduce that x = None and 
x = [] are both valid, given that spam() takes either a list or None, 
but I'm not sure if that level of intelligence is currently possible in 
type inference. (Maybe it is, and MyPy simply doesn't support it yet.) 
So it may be that this is a case where you do have to apply an explicit 
type hint, using *either* a type comment or this new proposed syntax:

x: Optional[List[int]]
if cond:
x = None
else:
x = []

should be precisely the same as:

if cond:
x = None #type: Optional[List[int]]
else:
x = []


> Quoting from the PEP:
> ```
> a: int
> a: str # Static type checker will warn about this.
> ```
> In other words, it is illegal for a checker to split up the variable, 
> even though it is straightforward to do so.

No, it is a *warning*, not an error.

Remember, the Python interpreter itself won't care. The type checker is 
optional and not part of the interpreter. You can still write code like:

a = 1
do_something(a)
a = "string"
do_another(a)

and Python will be happy. But if you do run a type checker, it should 
warn you that you're changing types, as that suggests the possibility of 
a type error. (And it also goes against a lot of people's style 
guidelines.)


> We should be free to add extra variables, whenever we choose, for 
> clarity. For example,
> total = foo() - bar()
> should not be treated differently from:
> revenue = foo()
> tax = bar()
> total = revenue - tax
> 
> If types are inferred, there is no problem.
> However, if they must be declared, then the use of meaningfully named 
> variables is discouraged.

Was there something in the PEP that lead you to believe that they "must" 
be declared? Under "Non-goals", the PEP states in bold text:

"the authors have no desire to ever make type hints mandatory"

so I'm not sure why you think that types must be declared.

Perhaps the PEP should make it more obvious that type hints on variables 
are *in addition to* and not a substitute for type inference.


> PEP 484 states:
> "If type hinting proves useful in general, a syntax for typing variables 
> may be provided in a future Python version."
> Has it proved useful in general? I don't think it has.

According to the PEP, it has proved useful in typeshed.


> It seems confused about class attributes and instance attributes
> 
> 
> The PEP also includes a section of how to define class attributes and 
> instance attributes. It seems that everything needs to be defined in the 
> class scope, even it is not an attribute of 

Re: [Python-Dev] Please reject or postpone PEP 526

[Guido van Rossum]

On Fri, Sep 2, 2016 at 6:47 AM, Mark Shannon  wrote:
> Hi everyone,
>
> I think we should reject, or at least postpone PEP 526.
>
> PEP 526 represents a major change to the language, however there are, I
> believe, a number of technical flaws with the PEP.
>
> It is probable that with significant revisions it can be a worthwhile
> addition to the language, but that cannot happen in time for 3.6 beta 1
(in
> 11 days).
>
> PEP 526 claims to be an extension of PEP 484, but I don't think that is
> entirely correct.
> PEP 484 was primarily about defining the semantics of pre-existing syntax.
> PEP 526 is about adding new syntax.
> Historically the bar for adding new syntax has been set very high. I don't
> think that PEP 526, in its current form, reaches that bar.
>
> Below is a list of the issues I have with the PEP as it stands.
>
> In many cases it makes it more effort than type comments
> 
>
> Type hints should be as easy to use as possible, and that means pushing as
> much work as possible onto the checker, and not burdening the programmer.
>
> Attaching type hints to variables, rather than expressions, reduces the
> potential for inference. This makes it harder for programmer, but easier
for
> the checker, which is the wrong way around.
>
> For example,, given a function:
> def spam(x: Optional[List[int]])->None: ...
>
> With type comments, this is intuitively correct and should type check:
> def eggs(cond:bool):
> if cond:
> x = None
> else:
> x = [] # type: List[int]
> spam(x) # Here we can infer the type of x
>
> With PEP 526 we loose the ability to infer types.
> def eggs(cond:bool):
> if cond:
> x = None # Not legal due to type declaration below
> else:
> x: List[int] = []
> spam(x)
>
> So we need to use a more complex type
> def eggs(cond:bool):
> x: Optional[List[int]]
> if cond:
> x = None # Now legal
> else:
> x: = []
> spam(x)
>
> I don't think this improves readability.
> Whether this is an acceptable change is debatable, but it does need some
> debate.

It looks like you're misinterpreting the intent of the PEP. It does not
mean legislate the behavior of the type checker in this way. In mypy, the
first example is already rejected because it wants the annotation on the
first occurrence. The plan is for mypy not to change its behavior -- the
old form

TARGET = VALUE # type: TYPE

will be treated the same way as the new form

TARGET: TYPE = VALUE

(If you have a beef with what this means in mypy you should probably take
it up with mypy, not with PEP 526.)

> It limits the use of variables
> ==
>
> In Python a name (variable) is just a binding that refers to an object.
> A name only exists in a meaningful sense once an object has been assigned
to
> it. Any attempt to use that name, without an object bound to it, will
result
> in a NameError.

(Or UnboundLocalError, if the compiler knows there is an assignment to the
name anywhere in the same (function) scope.)

> PEP 526 makes variables more than just bindings, as any rebinding must
> conform to the given type. This looses us some of the dynamism for which
we
> all love Python.

Thanks for catching this; that's not the intent.

> Quoting from the PEP:
> ```
> a: int
> a: str # Static type checker will warn about this.
> ```
> In other words, it is illegal for a checker to split up the variable, even
> though it is straightforward to do so.

One of my co-authors has gone too far here. The intent is not to legislate
what should happen in this case but to leave it to the checker. In mypy,
the equivalent syntax using type comments is currently indeed rejected, but
we're considering a change here (https://github.com/python/mypy/issues/1174).
The PEP 526 syntax will not make a difference here.

> However, without the type declarations,
> ```
> a = 1
> a = "Hi"
> ```
> is just fine. Useless, but fine.

And believe me, I want to keep it this way. I will amend the example and
clarify the intent in the text.

> We should be free to add extra variables, whenever we choose, for clarity.
> For example,
> total = foo() - bar()
> should not be treated differently from:
> revenue = foo()
> tax = bar()
> total = revenue - tax
>
> If types are inferred, there is no problem.
> However, if they must be declared, then the use of meaningfully named
> variables is discouraged.

There is no mandate to declare variables! I actually see the main use of
variable annotations in class bodies where it can crate a lot of clarity
around which instance variables exist and what they mean.

> [A note about type-inference:
> Type inference is not a universal panacea, but it can make life a lot
easier
> for programmers in statically type languages.
> Languages like C# use local type inference extensively and it means that
> many variables often do not need their type declared. We should take care
> not to limit the ability of checkers to infer values and types and make
> programmers' 

Re: [Python-Dev] Please reject or postpone PEP 526

[Ryan Gonzalez]

On Sep 2, 2016 8:51 AM, "Mark Shannon"  wrote:
>
> Hi everyone,
>
> I think we should reject, or at least postpone PEP 526.
>
> PEP 526 represents a major change to the language, however there are, I
believe, a number of technical flaws with the PEP.
>
> It is probable that with significant revisions it can be a worthwhile
addition to the language, but that cannot happen in time for 3.6 beta 1 (in
11 days).
>
> PEP 526 claims to be an extension of PEP 484, but I don't think that is
entirely correct.
> PEP 484 was primarily about defining the semantics of pre-existing
syntax. PEP 526 is about adding new syntax.
> Historically the bar for adding new syntax has been set very high. I
don't think that PEP 526, in its current form, reaches that bar.
>
> Below is a list of the issues I have with the PEP as it stands.
>
> In many cases it makes it more effort than type comments
> 
>
> Type hints should be as easy to use as possible, and that means pushing
as much work as possible onto the checker, and not burdening the programmer.
>
> Attaching type hints to variables, rather than expressions, reduces the
potential for inference. This makes it harder for programmer, but easier
for the checker, which is the wrong way around.
>
> For example,, given a function:
> def spam(x: Optional[List[int]])->None: ...
>
> With type comments, this is intuitively correct and should type check:
> def eggs(cond:bool):
> if cond:
> x = None
> else:
> x = [] # type: List[int]
> spam(x)  # Here we can infer the type of x
>
> With PEP 526 we loose the ability to infer types.
> def eggs(cond:bool):
> if cond:
> x = None # Not legal due to type declaration below
> else:
>x: List[int] = []
> spam(x)
>
> So we need to use a more complex type
> def eggs(cond:bool):
> x: Optional[List[int]]
> if cond:
> x = None # Now legal
> else:
> x: = []
> spam(x)
>
> I don't think this improves readability.
> Whether this is an acceptable change is debatable, but it does need some
debate.
>
> It limits the use of variables
> ==
>
> In Python a name (variable) is just a binding that refers to an object.
> A name only exists in a meaningful sense once an object has been assigned
to it. Any attempt to use that name, without an object bound to it, will
result in a NameError.
>
> PEP 526 makes variables more than just bindings, as any rebinding must
conform to the given type. This looses us some of the dynamism for which we
all love Python.
>
> Quoting from the PEP:
> ```
> a: int
> a: str # Static type checker will warn about this.
> ```
> In other words, it is illegal for a checker to split up the variable,
even though it is straightforward to do so.
>
> However, without the type declarations,
> ```
> a = 1
> a = "Hi"
> ```
> is just fine. Useless, but fine.
>

But isn't that the same way with type comments? Except uglier?

> We should be free to add extra variables, whenever we choose, for
clarity. For example,
> total = foo() - bar()
> should not be treated differently from:
> revenue = foo()
> tax = bar()
> total = revenue - tax
>
> If types are inferred, there is no problem.
> However, if they must be declared, then the use of meaningfully named
variables is discouraged.
>
> [A note about type-inference:
> Type inference is not a universal panacea, but it can make life a lot
easier for programmers in statically type languages.
> Languages like C# use local type inference extensively and it means that
many variables often do not need their type declared. We should take care
not to limit the ability of checkers to infer values and types and make
programmers' lives easier.
> Within a function, type inference is near perfect, failing only
occasionally for some generic types.
> One place where type inference definitely breaks down is across calls,
which is why PEP 484 is necessary.
> ]
>
> It is premature
> ===
>
> There are still plenty of issues to iron out w.r.t. PEP 484 types. I
don't think we should be adding more syntax, until we have a *precise* idea
of what is required.
>
> PEP 484 states:
> "If type hinting proves useful in general, a syntax for typing variables
may be provided in a future Python version."
> Has it proved useful in general? I don't think it has. Maybe it will in
future, but it hasn't yet.
>
> It seems confused about class attributes and instance attributes
> 
>
> The PEP also includes a section of how to define class attributes and
instance attributes. It seems that everything needs to be defined in the
class scope, even it is not an attribute of the class, but of its
instances. This seems confusing, both to human reader and machine analyser.
>
> Example from PEP 526:
>
> class Starship:
>
> captain: str = 'Picard'
> damage: int
> stats: