Re: [Python-ideas] Fixing class scope brainstorm

2018-03-30 Thread Paul Moore 
On 30 March 2018 at 06:17, Nick Coghlan  wrote:
> The fact that deep nesting of lexical scopes within an expression is
> almost always going to be an unreadable mess in practice is one of the
> reasons I don't think there's a strong case for either backwards
> compatibility breaks *or* significant increases in the overall
> semantic complexity.
>
> Any sensible coding style is already going to say "Don't do that, it's
> too hard to read", so we're mainly caring about it at all based on our
> own senses of engineering aesthetics and a general dislike of
> implementation details leaking through as user visible semantic
> differences.

OK, cool. That I can completely agree with :-)

>> I do think the current implementation is a pretty good compromise. I'd
>> be reasonably OK with not changing anything in this area. But this
>> discussion was prompted by some of the debates around statement local
>> variables, so "not changing anything" includes, in my mind, "not
>> trying to make statement local variables work" as they interact badly
>> with the current scoping behaviour in this area.
>
> It's specifically lexical closures that lead to things getting weird,
> and that's a key rationale for PEP 572's current recommendation that
> statement locals be completely invisible to nested scopes. Trying to
> have a subscope that isn't visible to the rest of the function it's
> in, while still being visible to nested scopes defined within that
> subscope, seems to lead to sufficient conflicts in reference lifecycle
> and visibility that just using a real nested scope instead ends up
> being preferable.

OK. I can accept your point that lexical closure weirdness is the
reason for PEP 572 preferring to make statement locals ignore nested
scopes. I'm not sure I fully agree, but I can accept it. My point is
that having a construct that introduces new names but makes them
invisible to nested scopes is *also* weird enough to be a problem. As
a result, I think that PEP 572 faces a problem in that it's hit a
problem for which it doesn't have a good solution.

Whether that problem is bad enough to kill the PEP is a matter of
judgement. In my view:

* The benefits of the PEP are marginal
* We haven't yet established a suitably readable syntax (IMO, := is
probably the best suggestion so far, but it still has its problems)
* I remain unconvinced that the PEP improves readability of comprehensions
* Plus the above scoping issue

So for me, the balance is in favour of keeping the status quo. If the
scoping issue could be resolved, the remaining issues are much more
obviously matters of taste and my view on the PEP would move to "not
worth the effort" rather than against.

Paul

PS I should also add that the "just make them assignments" resolution
to PEP 572's scope issues is the only suggestion so far that *doesn't*
have scoping weirdnesses of its own (because it uses existing scoping
mechanisms). The problem with that one is that it's prone to leaking
in ways that I think people are uncomfortable with ("I think" because
I don't have time right now to review the examples given to see for
myself how compelling they are).
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fixing class scope brainstorm

2018-03-29 Thread Nick Coghlan 
On 30 March 2018 at 02:53, Paul Moore  wrote:
> On 29 March 2018 at 16:27, Nick Coghlan  wrote:
>> On 28 March 2018 at 04:47, Paul Moore  wrote:
>>> To me, that would be the ideal. I assume there are significant
>>> technical challenges, though, as otherwise I'd have thought that would
>>> have been the approach taken when Python 3 fixed the name leaking
>>> issue from Python 2.
>>
>> It isn't avoiding the names leaking that's particularly challenging
>> (there are several viable ways to do that), it's making sure that
>> inner scopes can still see them. For example:
>>
>> lazy_and_eager = [((lambda: i), (lambda i=i: i)) for i in range(3)]
>> for lazy, eager in lazy_and_eager:
>> print(lazy(), eager())
>>
>> # prints:
>> 2 0
>> 2 1
>> 2 2
>
> I'm really not sure what point you're trying to make here - are you
> saying that this is good or bad? Correct or incorrect? I don't really
> have any intuition about what's going on here, so I'd just have to
> work it out in terms of the defined scoping rules.

In this context, it's neither good nor bad, it just is :)

> And I'd then tell
> whoever wrote it to rewrite it more clearly ;-)

Aye, we get to do that in a code review, but the compiler doesn't - it
has to figure out how to make it do something at least reasonably
defensible.

> Maybe a real-life case where this was important would clarify what
> counts as intuitive here - but as it stands,I don't really care.

The fact that deep nesting of lexical scopes within an expression is
almost always going to be an unreadable mess in practice is one of the
reasons I don't think there's a strong case for either backwards
compatibility breaks *or* significant increases in the overall
semantic complexity.

Any sensible coding style is already going to say "Don't do that, it's
too hard to read", so we're mainly caring about it at all based on our
own senses of engineering aesthetics and a general dislike of
implementation details leaking through as user visible semantic
differences.

>> While it's *technically* feasible to hide "i" from the outer scope
>> without hiding it from inner scopes with a variable renaming based
>> approach, you end up having to design and document a separate lexical
>> scoping mechanism that's distinct from the one that functions already
>> use. I really didn't want to do that, so I proposed just using the
>> existing scoping mechanism instead, and while that has definitely had
>> its quirks, I'm still happy enough with it a decade later to call it a
>> successful approach.
>
> I do think the current implementation is a pretty good compromise. I'd
> be reasonably OK with not changing anything in this area. But this
> discussion was prompted by some of the debates around statement local
> variables, so "not changing anything" includes, in my mind, "not
> trying to make statement local variables work" as they interact badly
> with the current scoping behaviour in this area.

It's specifically lexical closures that lead to things getting weird,
and that's a key rationale for PEP 572's current recommendation that
statement locals be completely invisible to nested scopes. Trying to
have a subscope that isn't visible to the rest of the function it's
in, while still being visible to nested scopes defined within that
subscope, seems to lead to sufficient conflicts in reference lifecycle
and visibility that just using a real nested scope instead ends up
being preferable.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fixing class scope brainstorm

2018-03-29 Thread Paul Moore 
On 29 March 2018 at 16:27, Nick Coghlan  wrote:
> On 28 March 2018 at 04:47, Paul Moore  wrote:
>> To me, that would be the ideal. I assume there are significant
>> technical challenges, though, as otherwise I'd have thought that would
>> have been the approach taken when Python 3 fixed the name leaking
>> issue from Python 2.
>
> It isn't avoiding the names leaking that's particularly challenging
> (there are several viable ways to do that), it's making sure that
> inner scopes can still see them. For example:
>
> lazy_and_eager = [((lambda: i), (lambda i=i: i)) for i in range(3)]
> for lazy, eager in lazy_and_eager:
> print(lazy(), eager())
>
> # prints:
> 2 0
> 2 1
> 2 2

I'm really not sure what point you're trying to make here - are you
saying that this is good or bad? Correct or incorrect? I don't really
have any intuition about what's going on here, so I'd just have to
work it out in terms of the defined scoping rules. And I'd then tell
whoever wrote it to rewrite it more clearly ;-)

Maybe a real-life case where this was important would clarify what
counts as intuitive here - but as it stands,I don't really care. I
don't even care that much about compatibility. Unless someone were to
come along and demonstrate a serious breakage in their code, I think
it's perfectly OK to change the behaviour in this situation, if that's
what you're suggesting (with suitable deprecation, of course).

> While it's *technically* feasible to hide "i" from the outer scope
> without hiding it from inner scopes with a variable renaming based
> approach, you end up having to design and document a separate lexical
> scoping mechanism that's distinct from the one that functions already
> use. I really didn't want to do that, so I proposed just using the
> existing scoping mechanism instead, and while that has definitely had
> its quirks, I'm still happy enough with it a decade later to call it a
> successful approach.

I do think the current implementation is a pretty good compromise. I'd
be reasonably OK with not changing anything in this area. But this
discussion was prompted by some of the debates around statement local
variables, so "not changing anything" includes, in my mind, "not
trying to make statement local variables work" as they interact badly
with the current scoping behaviour in this area.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fixing class scope brainstorm

2018-03-29 Thread Nick Coghlan 
On 28 March 2018 at 04:47, Paul Moore  wrote:
> On 27 March 2018 at 19:43, Ethan Furman  wrote:
>> On 03/27/2018 11:12 AM, Ivan Levkivskyi wrote:
>>>
>>> On 27 March 2018 at 18:19, Guido van Rossum wrote:
>>
 Hm, so maybe we shouldn't touch lambda, but we can at least fix the scope
 issues for comprehensions and genexprs.
>>>
>>>
>>> Removing the implicit function scope in comprehensions is something I
>>> wanted for long time.
>>> It would not only "fix" the scoping, but will also fix the yield inside
>>> comprehensions.
>>
>> Can we do it without leaking names?
>
> To me, that would be the ideal. I assume there are significant
> technical challenges, though, as otherwise I'd have thought that would
> have been the approach taken when Python 3 fixed the name leaking
> issue from Python 2.

It isn't avoiding the names leaking that's particularly challenging
(there are several viable ways to do that), it's making sure that
inner scopes can still see them. For example:

lazy_and_eager = [((lambda: i), (lambda i=i: i)) for i in range(3)]
for lazy, eager in lazy_and_eager:
print(lazy(), eager())

# prints:
2 0
2 1
2 2

While it's *technically* feasible to hide "i" from the outer scope
without hiding it from inner scopes with a variable renaming based
approach, you end up having to design and document a separate lexical
scoping mechanism that's distinct from the one that functions already
use. I really didn't want to do that, so I proposed just using the
existing scoping mechanism instead, and while that has definitely had
its quirks, I'm still happy enough with it a decade later to call it a
successful approach.

Cheers,
Nick.

P.S. This is also a contributing factor to one of the aspects of the
sublocals proposal: disallowing closing over them means that a
renaming based approach *can* be used to keep them separate from
regular local variable names.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fixing class scope brainstorm

2018-03-29 Thread Nick Coghlan 
On 28 March 2018 at 03:19, Guido van Rossum  wrote:
> On Tue, Mar 27, 2018 at 6:56 AM, Nick Coghlan  wrote:
>>
>> [...] The implicit functions used in the
>> comprehension & generator expression cases are just potentially
>> simpler to handle, as we don't care about their API signatures, which
>> means we can freely pollute their APIs with eager name bindings if we
>> choose to do so. [...]
>
>
> Hm, so maybe we shouldn't touch lambda, but we can at least fix the scope
> issues for comprehensions and genexprs.

Yeah, the "immediately call the function and throw it away" aspect
makes them much easier to deal with, and they're also the case that
currently feels weird to me (since the nested function is *supposed*
to be a hidden implementation detail, but it's existence currently
leaks through here).

I don't have that niggle with lambdas, since they're visibly defining
a new nested scope, so having them behave like method definitions
doesn't feel strange.

> There may still be breakage, when the code defines a global x that is
> overridden by a class-level x, and a class-level comprehension references x
> assuming it to be the global. So we need to tread carefully even here -- but
> this case is weird already:
>
> x = 42
> class C:
> x = [1, 2, 3]
> z = [x+y for y in x]  # [43, 44, 45]

Hmm, potentially more concerning might be cases where methods are
currently ignored, but in a future Python start shadowing builtins for
class level comprehensions and genexps. We're getting to the level of
"rare case" (methods shadowing builtins) intersecting with "rare case"
(using a comprehension or genexp at class scope) there though, so it
should be feasible to handle through a regular deprecation cycle.

So I'd guess this would need a future import ("from __future__ import
implicit_scopes", perhaps? I'm struggling to come up with a good name,
since it's essentially "implicit comprehension and generator
expression nested scopes, version two"), and an associated deprecation
warning for free variable references that would start resolving
differently.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fixing class scope brainstorm

2018-03-29 Thread Brice Parent 


Hm, so maybe we shouldn't touch lambda, but we can at least fix the 
scope issues for comprehensions and genexprs.


There may still be breakage, when the code defines a global x that is 
overridden by a class-level x, and a class-level comprehension 
references x assuming it to be the global. So we need to tread 
carefully even here -- but this case is weird already:


x = 42
class C:
    x = [1, 2, 3]
    z = [x+y for y in x]  # [43, 44, 45]


Wow!
I had to try it myself!

If I had came across something like the following in a code review :
x = [1, 2]
class C:
    x = [3, 4, 5]
    z = [x for _ in x]

I would have expected C.z to equal either `[[1, 2], [1, 2]]` or `[[3, 4, 
5], [3, 4, 5], [3, 4, 5]]`, but surely not `[[1, 2], [1, 2], [1, 2]]`!


Is that intentional, or the result of other way-more-logical decisions?

- Brice
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fixing class scope brainstorm

2018-03-27 Thread Greg Ewing 

Guido van Rossum wrote:
I do notice that we probably can't easily solve this using the existing 
closure machinery ("cells") because if we were to have cells in the 
class dict, it would confuse everything else that looks in the class 
namespace.


Maybe it could be done with a new kind of cell object that
holds a reference to the class dict and an attribute name.

--
Greg
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fixing class scope brainstorm

2018-03-27 Thread Guido van Rossum 
On Tue, Mar 27, 2018 at 11:51 AM, Ivan Levkivskyi 
wrote:

> On 27 March 2018 at 19:43, Ethan Furman  wrote:
>
>> On 03/27/2018 11:12 AM, Ivan Levkivskyi wrote:
>>
>>> On 27 March 2018 at 18:19, Guido van Rossum wrote:
>>>
>>
>> Hm, so maybe we shouldn't touch lambda, but we can at least fix the scope
 issues for comprehensions and genexprs.

>>>
>>> Removing the implicit function scope in comprehensions is something I
>>> wanted for long time.
>>> It would not only "fix" the scoping, but will also fix the yield inside
>>> comprehensions.
>>>
>>
>> Can we do it without leaking names?
>>
>>
> If you mean this
>
> [i for i in range(5)]
>
> i  # NameError
>
> then yes, this is possible. Serhiy outlined the implementation few moths
> ago. The rough idea is to use automatic re-naming.
> The only problem with this is that if someone will step into debugger one
> will see a name like .0.i instead of i.
> But this can be solved in the debuggers.
>

Oh, sorry, I misread what you were talking about. You're proposing going
back to the Python 2 shared namespace. I'm not at all excited about that,
and I'm not convinced that adjusting debuggers to hide the name mangling is
effective -- it does nothing about other forms of introspection. Also
depending on how PEP 572 falls there may be assignments in there. Plus
there may be object lifetime consequences.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fixing class scope brainstorm

2018-03-27 Thread Ivan Levkivskyi 
On 27 March 2018 at 19:47, Paul Moore  wrote:

> On 27 March 2018 at 19:43, Ethan Furman  wrote:
> > On 03/27/2018 11:12 AM, Ivan Levkivskyi wrote:
> >>
> >> On 27 March 2018 at 18:19, Guido van Rossum wrote:
> >
> >>> Hm, so maybe we shouldn't touch lambda, but we can at least fix the
> scope
> >>> issues for comprehensions and genexprs.
> >>
> >>
> >> Removing the implicit function scope in comprehensions is something I
> >> wanted for long time.
> >> It would not only "fix" the scoping, but will also fix the yield inside
> >> comprehensions.
> >
> > Can we do it without leaking names?
>
> To me, that would be the ideal. I assume there are significant
> technical challenges, though, as otherwise I'd have thought that would
> have been the approach taken when Python 3 fixed the name leaking
> issue from Python 2.
>
>
Yes, this will be certainly a big PR, but if we agree to do this, I
volunteer to make the PR.

--
Ivan
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fixing class scope brainstorm

2018-03-27 Thread Ivan Levkivskyi 
On 27 March 2018 at 19:43, Ethan Furman  wrote:

> On 03/27/2018 11:12 AM, Ivan Levkivskyi wrote:
>
>> On 27 March 2018 at 18:19, Guido van Rossum wrote:
>>
>
> Hm, so maybe we shouldn't touch lambda, but we can at least fix the scope
>>> issues for comprehensions and genexprs.
>>>
>>
>> Removing the implicit function scope in comprehensions is something I
>> wanted for long time.
>> It would not only "fix" the scoping, but will also fix the yield inside
>> comprehensions.
>>
>
> Can we do it without leaking names?
>
>
If you mean this

[i for i in range(5)]

i  # NameError

then yes, this is possible. Serhiy outlined the implementation few moths
ago. The rough idea is to use automatic re-naming.
The only problem with this is that if someone will step into debugger one
will see a name like .0.i instead of i.
But this can be solved in the debuggers.

--
Ivan
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fixing class scope brainstorm

2018-03-27 Thread Paul Moore 
On 27 March 2018 at 19:43, Ethan Furman  wrote:
> On 03/27/2018 11:12 AM, Ivan Levkivskyi wrote:
>>
>> On 27 March 2018 at 18:19, Guido van Rossum wrote:
>
>>> Hm, so maybe we shouldn't touch lambda, but we can at least fix the scope
>>> issues for comprehensions and genexprs.
>>
>>
>> Removing the implicit function scope in comprehensions is something I
>> wanted for long time.
>> It would not only "fix" the scoping, but will also fix the yield inside
>> comprehensions.
>
> Can we do it without leaking names?

To me, that would be the ideal. I assume there are significant
technical challenges, though, as otherwise I'd have thought that would
have been the approach taken when Python 3 fixed the name leaking
issue from Python 2.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fixing class scope brainstorm

2018-03-27 Thread Guido van Rossum 
On Tue, Mar 27, 2018 at 11:43 AM, Ethan Furman  wrote:

> On 03/27/2018 11:12 AM, Ivan Levkivskyi wrote:
>
>> On 27 March 2018 at 18:19, Guido van Rossum wrote:
>>
>
> Hm, so maybe we shouldn't touch lambda, but we can at least fix the scope
>>> issues for comprehensions and genexprs.
>>>
>>
>> Removing the implicit function scope in comprehensions is something I
>> wanted for long time.
>> It would not only "fix" the scoping, but will also fix the yield inside
>> comprehensions.
>>
>
> Can we do it without leaking names?
>

Assuming you're concerned about leaking names out of the comprehension into
the class scope, that shouldn't be a problem. (The solution actually
involves leaking names *into* the comprehension scope, but I'm not sure
that should be called "leaking". :-)

I do notice that we probably can't easily solve this using the existing
closure machinery ("cells") because if we were to have cells in the class
dict, it would confuse everything else that looks in the class namespace.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fixing class scope brainstorm

2018-03-27 Thread Ethan Furman 

On 03/27/2018 11:12 AM, Ivan Levkivskyi wrote:

On 27 March 2018 at 18:19, Guido van Rossum wrote:



Hm, so maybe we shouldn't touch lambda, but we can at least fix the scope 
issues for comprehensions and genexprs.


Removing the implicit function scope in comprehensions is something I wanted 
for long time.
It would not only "fix" the scoping, but will also fix the yield inside 
comprehensions.


Can we do it without leaking names?

--
~Ethan~
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fixing class scope brainstorm

2018-03-27 Thread Ivan Levkivskyi 
On 27 March 2018 at 18:19, Guido van Rossum  wrote:

> On Tue, Mar 27, 2018 at 6:56 AM, Nick Coghlan  wrote:
>
>> [...] The implicit functions used in the
>> comprehension & generator expression cases are just potentially
>> simpler to handle, as we don't care about their API signatures, which
>> means we can freely pollute their APIs with eager name bindings if we
>> choose to do so. [...]
>>
>
> Hm, so maybe we shouldn't touch lambda, but we can at least fix the scope
> issues for comprehensions and genexprs.
>

Removing the implicit function scope in comprehensions is something I
wanted for long time.
It would not only "fix" the scoping, but will also fix the yield inside
comprehensions.

--
Ivan
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fixing class scope brainstorm

2018-03-27 Thread Eric Fahlgren 
On Tue, Mar 27, 2018 at 9:52 AM, Paul Moore  wrote:

> I'd actually like to see some real world use cases to get a feel for
> whether this is even worth worrying about. (I'm not saying it isn't,
> just that it's hard to get a feel for the importance based on
> artificial examples).
>

​The only reason I brought it up was because I ran into this about three
weeks ago porting some old Python 2 to 3, where there was a class level
comprehension that referenced a class variable on the lhs.  I simply
removed it by enumerating the cases by hand, no big deal, but it did take
me a while to figure out why ​that no longer worked.

My specific case looked approximately like this:

class Plugin:
plugin_dir = 'somepath'
plugin_names = [os.join(plugin_dir, name) for name in ('list', 'of',
'names')]
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fixing class scope brainstorm

2018-03-27 Thread Guido van Rossum 
On Tue, Mar 27, 2018 at 6:56 AM, Nick Coghlan  wrote:

> [...] The implicit functions used in the
> comprehension & generator expression cases are just potentially
> simpler to handle, as we don't care about their API signatures, which
> means we can freely pollute their APIs with eager name bindings if we
> choose to do so. [...]
>

Hm, so maybe we shouldn't touch lambda, but we can at least fix the scope
issues for comprehensions and genexprs.

There may still be breakage, when the code defines a global x that is
overridden by a class-level x, and a class-level comprehension references x
assuming it to be the global. So we need to tread carefully even here --
but this case is weird already:

x = 42
class C:
x = [1, 2, 3]
z = [x+y for y in x]  # [43, 44, 45]

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fixing class scope brainstorm

2018-03-27 Thread Paul Moore 
On 27 March 2018 at 16:51, Joao S. O. Bueno  wrote:
> Well, there is an idiom to "keep everything as is", and work around
> the current limitations:
>
> class A:
>def b():
>x = 1
>d = [i + x for i in range(2)]
>return locals()
>locals().update(b())
>del b
>
> Maybe if we could find a syntactic sugar for this idiom
> (with an abuse of the `with` keyword, for example), people could be happy,
> with class bodies working by default as they are now, and enabling the
> reuse of defined members by using the special syntax -
>
> class A:
>with class:
>   x = 1
>   d = [i + x for in range(2)]

I'd actually like to see some real world use cases to get a feel for
whether this is even worth worrying about. (I'm not saying it isn't,
just that it's hard to get a feel for the importance based on
artificial examples).

BTW, for an alternative workaround that avoids nested scopes altogether:

>>> import operator
>>> from functools import partial
>>> class C:
...   x = 1
...   d = list(map(partial(operator.add, x), range(2)))
...
>>> c = C()
>>> c.d
[1, 2]

No, I'm not suggesting that's clearer - but it does (at least in my
mind) make it more obvious that the problem is with comprehensions,
not with class scopes.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fixing class scope brainstorm

2018-03-27 Thread Joao S. O. Bueno 
Well, there is an idiom to "keep everything as is", and work around
the current limitations:

class A:
   def b():
   x = 1
   d = [i + x for i in range(2)]
   return locals()
   locals().update(b())
   del b

Maybe if we could find a syntactic sugar for this idiom
(with an abuse of the `with` keyword, for example), people could be happy,
with class bodies working by default as they are now, and enabling the
reuse of defined members by using the special syntax -

class A:
   with class:
  x = 1
  d = [i + x for in range(2)]





On 27 March 2018 at 12:27, Paul Moore  wrote:
> On 27 March 2018 at 15:32, Joao S. O. Bueno  wrote:
>> Yes - but that would be the intention of the code beign written as in
>> your example -
>>
>> class C:
>> x = 1
>> f = staticmethod(lambda: print(x))
>>
>> While, the classic behavior can be attained by doing:
>>
>> class C:
>>  x = 1
>>  f = classmethod(lambda cls: print(cls.x))
>>
>> And the behavior in both cases if one of no-surprises for me. For
>> coders who don't have the mechanism of class creation in their
>> mind, that could come as a surprise, but it is better than being
>> inconsistent.
>
> I wouldn't describe myself as "having the mechanism of class creation
> in my mind", but I'm not 100% sure that's a necessary criterion here.
> In an ideal world, Python's semantics is supposed to be intuitive,
> which means that understanding subtle details shouldn't be necessary
> to anticipate the behaviour of certain constructs.
>
> Looking at the following definitions:
>
> class C:
> x = 1
>
> y1 = x
> y2 = (lambda: x)()
>
> def meth1(self):
> return self.x
> meth2 = lambda self: self.x
>
> @staticmethod
> def sm1():
> return x
> sm2 = staticmethod(lambda: x)
>
> @classmethod
> def cm1(cls):
> return cls.x
> cm2 = classmethod(lambda cls: cls.x)
>
> I would expect meth1 and meth2 to be equivalent. I'd expect cm1 and
> cm2 to be equivalent. I would *not* expect sm1 to work, and I'd expect
> sm2 to fail for the same reason - namely that sm1 has no access to the
> class or an instance of it, so it should not be able to reference x.
> And so we should get NameError.
>
> These to me don't imply any sort of "understanding of how classes are
> created" - they simply need an understanding of how methods (normal,
> static and class) get access to the class/instance. They also require
> that you *don't* expect a class statement to create a scope that can
> be closed over. I didn't really think about that before I started
> analysing this code, but once I did, I realised that I've never
> expected that.
>
> So my reaction to a bare nonlocal variable reference in a method
> (whether defined in a def statement or as a lambda) would be "wait,
> what would that mean? I guess it's the global". I wouldn't even be
> looking at the x defined in the class at that point.
>
> The definition of y2 follows that rule as well - even though the fact
> that it's not defining a method, but it's "just" a bare lambda,
> slightly muddies the water.
>
> The odd one out is y1. I actually can't quite explain the logic that
> allows y1 to refer to the value of x, even though it's the most
> "natural" case. As I said, I don't think of a class as defining a new
> scope, rather I think of it as bundling together a set of statements
> that will be run to populate the class (maybe that's "having the
> mechanism of class creation in my mind"?). So I guess y1 = x is just
> normal statement sequencing.
>
> So I guess I'm saying that I don't really see a problem with the
> current behaviour here. Classes *don't* create a scope. So you don't
> close over class variables.
>
> For further emphasis of this point of view:
>
> @staticmethod
> def sm1():
> nonlocal x
> return x
>
> gives
>
> nonlocal x
> ^
> SyntaxError: no binding for nonlocal 'x' found
>
> which again I can understand as "there's no outer scope containing x".
>
> Having said all this, I guess you could say that I'm too close to the
> current behaviour to see its deficiencies. Maybe that's true. But I
> don't actually *use* any of this on a regular basis. I worked out all
> of the above based on intuition from how I understood Python's class
> model, plus a few small experiments that mostly just confirmed what I
> expected.
>
> If adding the ability to refer to a bare x in sm1/sm2 or y2 [1] means
> complicating the behaviour to the point where my mental model needs to
> involve injecting arguments into nested scopes, I'm a strong -1.If
> there's a need to "fix" comprehensions, then I'd much rather that we
> do so in a way that doesn't change the current behaviour or execution
> model. Specifically, I'd actually much *rather* that these 3 cases
> continue giving NameError.
>
> The comprehension cases:
>

Re: [Python-ideas] Fixing class scope brainstorm

2018-03-27 Thread Paul Moore 
On 27 March 2018 at 15:32, Joao S. O. Bueno  wrote:
> Yes - but that would be the intention of the code beign written as in
> your example -
>
> class C:
> x = 1
> f = staticmethod(lambda: print(x))
>
> While, the classic behavior can be attained by doing:
>
> class C:
>  x = 1
>  f = classmethod(lambda cls: print(cls.x))
>
> And the behavior in both cases if one of no-surprises for me. For
> coders who don't have the mechanism of class creation in their
> mind, that could come as a surprise, but it is better than being
> inconsistent.

I wouldn't describe myself as "having the mechanism of class creation
in my mind", but I'm not 100% sure that's a necessary criterion here.
In an ideal world, Python's semantics is supposed to be intuitive,
which means that understanding subtle details shouldn't be necessary
to anticipate the behaviour of certain constructs.

Looking at the following definitions:

class C:
x = 1

y1 = x
y2 = (lambda: x)()

def meth1(self):
return self.x
meth2 = lambda self: self.x

@staticmethod
def sm1():
return x
sm2 = staticmethod(lambda: x)

@classmethod
def cm1(cls):
return cls.x
cm2 = classmethod(lambda cls: cls.x)

I would expect meth1 and meth2 to be equivalent. I'd expect cm1 and
cm2 to be equivalent. I would *not* expect sm1 to work, and I'd expect
sm2 to fail for the same reason - namely that sm1 has no access to the
class or an instance of it, so it should not be able to reference x.
And so we should get NameError.

These to me don't imply any sort of "understanding of how classes are
created" - they simply need an understanding of how methods (normal,
static and class) get access to the class/instance. They also require
that you *don't* expect a class statement to create a scope that can
be closed over. I didn't really think about that before I started
analysing this code, but once I did, I realised that I've never
expected that.

So my reaction to a bare nonlocal variable reference in a method
(whether defined in a def statement or as a lambda) would be "wait,
what would that mean? I guess it's the global". I wouldn't even be
looking at the x defined in the class at that point.

The definition of y2 follows that rule as well - even though the fact
that it's not defining a method, but it's "just" a bare lambda,
slightly muddies the water.

The odd one out is y1. I actually can't quite explain the logic that
allows y1 to refer to the value of x, even though it's the most
"natural" case. As I said, I don't think of a class as defining a new
scope, rather I think of it as bundling together a set of statements
that will be run to populate the class (maybe that's "having the
mechanism of class creation in my mind"?). So I guess y1 = x is just
normal statement sequencing.

So I guess I'm saying that I don't really see a problem with the
current behaviour here. Classes *don't* create a scope. So you don't
close over class variables.

For further emphasis of this point of view:

@staticmethod
def sm1():
nonlocal x
return x

gives

nonlocal x
^
SyntaxError: no binding for nonlocal 'x' found

which again I can understand as "there's no outer scope containing x".

Having said all this, I guess you could say that I'm too close to the
current behaviour to see its deficiencies. Maybe that's true. But I
don't actually *use* any of this on a regular basis. I worked out all
of the above based on intuition from how I understood Python's class
model, plus a few small experiments that mostly just confirmed what I
expected.

If adding the ability to refer to a bare x in sm1/sm2 or y2 [1] means
complicating the behaviour to the point where my mental model needs to
involve injecting arguments into nested scopes, I'm a strong -1.If
there's a need to "fix" comprehensions, then I'd much rather that we
do so in a way that doesn't change the current behaviour or execution
model. Specifically, I'd actually much *rather* that these 3 cases
continue giving NameError.

The comprehension cases:

  y3 = [x+i for i in (0,1,2)] #1
  y4 = [1+i for i in (x, x+1)] #2

(#1 fails with NameError, and #2 works) strike me as odd corner cases
that are a bit odd, but are not worth disrupting the normal cases for.
And they should be viewed as oddities in how comprehensions look names
up, and not as peculiarities of class scope.

Sorry - I intended that to be an "it's all pretty simple and obvious"
comment, but it turned into a rather long post. But I do think the
basic idea remains simple.

Paul

[1] The other cases don't need to refer to a bare x, they already have
perfectly good ways of accessing the class variable x.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct:

Re: [Python-ideas] Fixing class scope brainstorm

2018-03-27 Thread Joao S. O. Bueno 
On 27 March 2018 at 10:56, Nick Coghlan  wrote:
> ... Making it so that lambdas
> can close over class attributes breaks that equivalence, and if we
> were to use the cell based approach I suggest above, the seams would
> be visible in the case where a lambda expression references an
> attribute that gets rebound after class creation:
>
> >>> C.f()
> 1
> >>> C.x = 2
> >>> C.f() # With a cell based approach, this wouldn't change
> 1
>

Yes - but that would be the intention of the code beign written as in
your example -

class C:
x = 1
f = staticmethod(lambda: print(x))

While, the classic behavior can be attained by doing:

class C:
 x = 1
 f = classmethod(lambda cls: print(cls.x))

And the behavior in both cases if one of no-surprises for me.  For
coders who don't
have the mechanism of class creation in their mind, that could come as
 a surprise, but it is
better than being inconsistent.

TL;DR: +1 for your approach - I am just saying your perceived drawback
is not one IMHO.

> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fixing class scope brainstorm

2018-03-27 Thread Nick Coghlan 
On 27 March 2018 at 01:11, Guido van Rossum  wrote:
> Honestly that sounds way too complex. In my ideal world, only functions
> defined with 'def' are methods whose __get__ auto-binds the first argument
> (unless @classmethod or @staticmethod is present), and do not get to "see"
> the class scope. This is how you define methods after all. Lambdas used at
> class scope do not get this benefit, and they see the class scope as just
> another closure; ditto for comprehensions and genexprs.
>
> I don't think that the "is the call delayed" idea is the proper way to
> distinguish the two cases here. IMO the key concept is "is it used to define
> a method". I betcha if you see a lambda here it's because there's some
> calculation going on whose result will be stored as a class variable.

Aye, I don't disagree with that. The implicit functions used in the
comprehension & generator expression cases are just potentially
simpler to handle, as we don't care about their API signatures, which
means we can freely pollute their APIs with eager name bindings if we
choose to do so.

Lambda expressions are different, since their public API matters, so
we can't mess with it freely to pass in extra name references.

What we potentially *could* do though is allow implicit additions to
their __closure__ attributes in a way that isn't permitted for regular
function definitions, such that in the following code:

class C:
x = 1
f = staticmethod(lambda: print(x))

we would do the following things differently for lambdas (vs def functions):

1. We'd pass "x" down as a candidate for nonlocal name resolution
while compiling the lambda
2. If "x" did get referenced from a lambda, we'd start allowing
non-optimised code objects to have a list of cell vars (the way
functions already do), and define a new "STORE_CLASSDEREF" opcode (as
the counterpart to LOAD_CLASSDEREF)

What STORE_CLASSDEREF would need to do is write updates both to the
current locals namespace (so the expectations of a class body are met)
*and* to the nominated cell object (so any references from lambda
expressions are updated).

The biggest downside I'd see to this is that for an incredibly long
time, we've pushed the view that "lambda expressions are just regular
functions that don't know their own name". Making it so that lambdas
can close over class attributes breaks that equivalence, and if we
were to use the cell based approach I suggest above, the seams would
be visible in the case where a lambda expression references an
attribute that gets rebound after class creation:

>>> C.f()
1
>>> C.x = 2
>>> C.f() # With a cell based approach, this wouldn't change
1

All of those potential complexities are specific to lambda functions
though - since the implicit functions created for lambda expressions
and generator expressions are called and then immediately thrown away,
they don't need a persistent cell reference to the closed over
variable name.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fixing class scope brainstorm

2018-03-26 Thread Guido van Rossum 
On Mon, Mar 26, 2018 at 6:33 AM, Nick Coghlan  wrote:

> On 26 March 2018 at 01:50, Guido van Rossum  wrote:
> > In the PEP 572 threads there's some grumbling about class scopes.
> >
> > Here's a random brainstorm. How about we change the scoping rules so that
> > only functions defined with 'def' (or inside one of those) cannot see the
> > class scope, and comprehensions and lambdas treat the class scope as an
> > outer scope so they can close over class variables?
> >
> > Not sure what should happen to nested classes, they should probably be
> > lumped in with the 'def' scopes.
>
> I think it's mainly in comprehensions that folks may find the current
> behaviour surprising, as that's the case where we define a function
> and immediately call it, so it mostly looks like regular inline code
> execution.
>
> Once explicitly delayed invocation is involved (lambdas, def
> statements), folks seem to be more comfortable with the idea "Oh, of
> course that's going to behave like a method at class scope". Generator
> expressions also mostly get a pass, since the *iteration* is delayed,
> even though the generator-iterator itself is created immediately.
>
> One possibility that has occurred to me (but I've never investigated
> to check the technical feasibility) is to see whether we might be able
> to define a notion of "implied arguments" for the implicit
> comprehension and generator expression function definitions.
>
> The gist of that idea would be:
>
> - when walking the symbol table for a comprehension or genexp, keep
> track of every variable name mentioned in that subtree which is not
> resolved in that subtree (which we already have the machinery to do,
> as it's needed for compiling functions in general)
> - treat all those names as *implied arguments* to the implicit
> function, and call it with the right references in the right positions
>
> While we'd likely still want to keep evaluation of the outermost
> iterator outside the nested scope for the sake of better genexp
> tracebacks, that would still be enough to enable cases like:
>
> class C:
> y = 1
> values = [x+y for x in range(10)]
>
> The trick would be to make it so that that comprehension gets expanded as:
>
> def _listcomp(_outermost_iter, y):
> result = []
> for x in _outermost_iter:
> result.append(x+y)
> return result
>
> _listcomp_result = _listcomp(range(10), y)
>
> One of the more attractive aspects of this possibility (assuming it
> can be made to work) is that it isn't even a special case in the
> runtime name lookup rules - it's just changing the definition of the
> implicit functions that we create for comprehensions and generator
> expressions to make sure that they can refer to *any* name that
> resolves in the namespace where they're called (even
> invisible-to-the-compiler names like those injected via __prepare__
> methods, or assignments via locals() at class scope).
>
> Cheers,
> Nick.
>
> P.S. I suspect we'd also find that this served as a performance
> optimisation, since it wouldn't only prebind class locals, it would
> prebind references to builtins and module globals as well.
>

Honestly that sounds way too complex. In my ideal world, only functions
defined with 'def' are methods whose __get__ auto-binds the first argument
(unless @classmethod or @staticmethod is present), and do not get to "see"
the class scope. This is how you define methods after all. Lambdas used at
class scope do not get this benefit, and they see the class scope as just
another closure; ditto for comprehensions and genexprs.

I don't think that the "is the call delayed" idea is the proper way to
distinguish the two cases here. IMO the key concept is "is it used to
define a method". I betcha if you see a lambda here it's because there's
some calculation going on whose result will be stored as a class variable.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fixing class scope brainstorm

2018-03-26 Thread Nick Coghlan 
On 26 March 2018 at 01:50, Guido van Rossum  wrote:
> In the PEP 572 threads there's some grumbling about class scopes.
>
> Here's a random brainstorm. How about we change the scoping rules so that
> only functions defined with 'def' (or inside one of those) cannot see the
> class scope, and comprehensions and lambdas treat the class scope as an
> outer scope so they can close over class variables?
>
> Not sure what should happen to nested classes, they should probably be
> lumped in with the 'def' scopes.

I think it's mainly in comprehensions that folks may find the current
behaviour surprising, as that's the case where we define a function
and immediately call it, so it mostly looks like regular inline code
execution.

Once explicitly delayed invocation is involved (lambdas, def
statements), folks seem to be more comfortable with the idea "Oh, of
course that's going to behave like a method at class scope". Generator
expressions also mostly get a pass, since the *iteration* is delayed,
even though the generator-iterator itself is created immediately.

One possibility that has occurred to me (but I've never investigated
to check the technical feasibility) is to see whether we might be able
to define a notion of "implied arguments" for the implicit
comprehension and generator expression function definitions.

The gist of that idea would be:

- when walking the symbol table for a comprehension or genexp, keep
track of every variable name mentioned in that subtree which is not
resolved in that subtree (which we already have the machinery to do,
as it's needed for compiling functions in general)
- treat all those names as *implied arguments* to the implicit
function, and call it with the right references in the right positions

While we'd likely still want to keep evaluation of the outermost
iterator outside the nested scope for the sake of better genexp
tracebacks, that would still be enough to enable cases like:

class C:
y = 1
values = [x+y for x in range(10)]

The trick would be to make it so that that comprehension gets expanded as:

def _listcomp(_outermost_iter, y):
result = []
for x in _outermost_iter:
result.append(x+y)
return result

_listcomp_result = _listcomp(range(10), y)

One of the more attractive aspects of this possibility (assuming it
can be made to work) is that it isn't even a special case in the
runtime name lookup rules - it's just changing the definition of the
implicit functions that we create for comprehensions and generator
expressions to make sure that they can refer to *any* name that
resolves in the namespace where they're called (even
invisible-to-the-compiler names like those injected via __prepare__
methods, or assignments via locals() at class scope).

Cheers,
Nick.

P.S. I suspect we'd also find that this served as a performance
optimisation, since it wouldn't only prebind class locals, it would
prebind references to builtins and module globals as well.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Fixing class scope brainstorm

2018-03-25 Thread Elazar 
On Sun, Mar 25, 2018 at 5:51 PM Guido van Rossum  wrote:

> In the PEP 572 threads there's some grumbling about class scopes.
>
> Here's a random brainstorm. How about we change the scoping rules so that
> only functions defined with 'def' (or inside one of those) cannot see the
> class scope, and comprehensions and lambdas treat the class scope as an
> outer scope so they can close over class variables?
>
> Not sure what should happen to nested classes, they should probably be
> lumped in with the 'def' scopes.
>
>
I would expect classes to behave like comprehensions - i.e. code that is
not delayed will see the class' scope as an enclosing scope (since it is
not namespaced yet so there's no other way to refer to it) and delayed code
(e.g. lambdas or defs) will not.

Elazar
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Fixing class scope brainstorm

2018-03-25 Thread Guido van Rossum 
In the PEP 572 threads there's some grumbling about class scopes.

Here's a random brainstorm. How about we change the scoping rules so that
only functions defined with 'def' (or inside one of those) cannot see the
class scope, and comprehensions and lambdas treat the class scope as an
outer scope so they can close over class variables?

Not sure what should happen to nested classes, they should probably be
lumped in with the 'def' scopes.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/