[Python-ideas] Re: Extension methods in Python

[Chris Angelico]

On Fri, Jun 25, 2021 at 1:54 PM Ricky Teachey  wrote:
>
> Would this feature allow me to declare str objects as not iterable in some 
> contexts?
>
> If so, +1.
>

That depends. If the proposal is to intercept every attribute lookup
(parallel to a class's __getattribute__ method), then yes, but if it's
a fallback after default behaviour fails (like __getattr__), then no.
I suspect that the latter is more likely; it's much easier to avoid
recursion problems if real attributes are tried first, plus it's
likely to impact performance a lot less.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/TIYNGARJNUV6VBYFRRDHNH7NJPSHPHZF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

[David Mertz]

I've read all the posts in this thread, and am overall at least -0.5 on the
idea. I like methods well enough, but mostly it just seems to invite
confusion versus the equivalent and existing option of importing functions.

I am happy, initially, to stipulate that "some clever technique" is
available to make accessing an extension method/attribute efficient. My
objection isn't that.

Rather, my concern is "spooky action at a distance." It becomes difficult
to know whether my object 'foo' will have the '.do_blaz()' method or not.
Not difficult like no determinate rule could exist, but difficult in the
sense that I'm looking at this one line of code in a thousand line module.
The type() of 'foo' is no longer enough information to know the answer.

That said, one crucial difference is once an extension method is "used" we
are stuck with it for the entire module. In contrast, functions can be both
namespaced and redefined. So I can do:

import alpha, beta
if alpha.do_blaz() == beta.do_blaz(): ...

I can also do this:

from alpha import do_blaz
def myfun():
from beta import do_blaz
...

We get scoping and namespaces that extension method lack.

So perhaps we could add that?

from extensions import do_blaz

with extend(list, do_blaz):
assert isinstance(foo, list)
foo.do_blaz()

This would avoid the drawbacks I perceive. On the other hand, it feels like
ain't a fair amount of complexity for negligible actual gain.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/RP6YV736DYMNDETA25IGMPO4HKGG4H3M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

[Ricky Teachey]

Would this feature allow me to declare str objects as not iterable in some
contexts?

If so, +1.

On Thu, Jun 24, 2021, 3:22 PM Chris Angelico  wrote:

> On Fri, Jun 25, 2021 at 3:31 AM Steven D'Aprano 
> wrote:
> >
> > Here's a quick and dirty proof of concept I knocked up in about 20
> > minutes, demonstrating that no deep compiler magic is needed. It's just
> > a small change to the way `object.__getattribute__` works.
> >
> > I've emulated it with my own base class, since `object` can't be
> > monkey-patched.
> >
> > The proof of concept is probably buggy and incomplete. It isn't intended
> > to be a final, polished production-ready implementation. It's not
> > implementation-agnostic: it requires the ability to inspect the call
> > stack. If you're using IronPython, this may not work.
> >
> > You will notice I didn't need to touch getattr to have it work, let
> > alone hack the interpreter to make it some sort of magical construct. It
> > all works through `__getattribute__`.
> >
> > The registration system is just the easiest thing that I could throw
> > together. There are surely better designs.
> >
> > Run A.py to see it in action.
> >
>
> Okay, so you've hidden the magic away a bit, but you have to choose
> the number [2] for your stack inspection. That means you have to be
> sure that that's the correct module, in some way. If you do *anything*
> to disrupt the exact depth of the call stack, that breaks.
>
> _hasattr = hasattr
> def hasattr(obj, attr): return _hasattr(obj, attr)
>
> Or any of the other higher level constructs. What if there's a C-level
> function in there?
>
> This is still magic. It's just that the magic has been buried slightly.
>
> ChrisA
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/RPBIHJE3SHL7S6XIS2343SXCOTW5C54A/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/Z23YL2DJQRUL5KOF6SP5FTW5EOYR3BXD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

[Chris Angelico]

On Fri, Jun 25, 2021 at 3:31 AM Steven D'Aprano  wrote:
>
> Here's a quick and dirty proof of concept I knocked up in about 20
> minutes, demonstrating that no deep compiler magic is needed. It's just
> a small change to the way `object.__getattribute__` works.
>
> I've emulated it with my own base class, since `object` can't be
> monkey-patched.
>
> The proof of concept is probably buggy and incomplete. It isn't intended
> to be a final, polished production-ready implementation. It's not
> implementation-agnostic: it requires the ability to inspect the call
> stack. If you're using IronPython, this may not work.
>
> You will notice I didn't need to touch getattr to have it work, let
> alone hack the interpreter to make it some sort of magical construct. It
> all works through `__getattribute__`.
>
> The registration system is just the easiest thing that I could throw
> together. There are surely better designs.
>
> Run A.py to see it in action.
>

Okay, so you've hidden the magic away a bit, but you have to choose
the number [2] for your stack inspection. That means you have to be
sure that that's the correct module, in some way. If you do *anything*
to disrupt the exact depth of the call stack, that breaks.

_hasattr = hasattr
def hasattr(obj, attr): return _hasattr(obj, attr)

Or any of the other higher level constructs. What if there's a C-level
function in there?

This is still magic. It's just that the magic has been buried slightly.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/RPBIHJE3SHL7S6XIS2343SXCOTW5C54A/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

[Steven D'Aprano]

Here's a quick and dirty proof of concept I knocked up in about 20 
minutes, demonstrating that no deep compiler magic is needed. It's just 
a small change to the way `object.__getattribute__` works.

I've emulated it with my own base class, since `object` can't be 
monkey-patched.

The proof of concept is probably buggy and incomplete. It isn't intended 
to be a final, polished production-ready implementation. It's not 
implementation-agnostic: it requires the ability to inspect the call 
stack. If you're using IronPython, this may not work.

You will notice I didn't need to touch getattr to have it work, let 
alone hack the interpreter to make it some sort of magical construct. It 
all works through `__getattribute__`.

The registration system is just the easiest thing that I could throw 
together. There are surely better designs.

Run A.py to see it in action.


-- 
Steve
# Extension method helpers and proof of concept.
import inspect

MODULE_REGISTRY = set()
METHOD_REGISTRY = {}

def get_execution_scope():
frm = inspect.stack()[2]
return inspect.getmodule(frm[0])

def extends(cls):
def inner(func):
METHOD_REGISTRY.setdefault(cls, set()).add(func)
return func
return inner

def using():
MODULE_REGISTRY.add(get_execution_scope())

class MyObject:
# Base class that supports extension methods.
def __getattribute__(self, name):
try:
return super().__getattribute__(name)
except AttributeError:
mod = get_execution_scope()
if mod in MODULE_REGISTRY:
xmethods = METHOD_REGISTRY.get(type(self), set())
for func in xmethods:
if func.__name__ == name:
return func.__get__(self)
raise

class Demo(MyObject):
pass

@extends(Demo)
def xmethod(self):
return "called extension method"
from extmethods import Demo

instance = Demo()
print('(Module A) has xmethod?', hasattr(instance, 'xmethod'))

import B
B.main()

print('(Module A) has xmethod?', hasattr(instance, 'xmethod'))
from extmethods import using, Demo

using()

def main():
obj = Demo()
print('(Module B) has xmethod?', hasattr(obj, 'xmethod'))
print(getattr(obj, 'xmethod')())
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/GINVBO2BIJNPSTT65DQ37YQF2PFZQYST/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

[Steven D'Aprano]

On Thu, Jun 24, 2021 at 09:19:36AM -0700, Christopher Barker wrote:

> Steven, you're making a pretty good case here, but a couple questions:
> 
> 1) The case of newer versions of python adding methods to builtins, like
> .bit_length is really compelling. But I do wonder how frequently that comes
> up. It seems to me on this list, that people are very reluctant to add
> methods to anything builtin (other than dunders) -- particularly ABCs, or
> classes that might be subclassed, as that might break anything that
> currently uses that same name. Anyway, is this a one-off? or something that
> is likely to come up semi-frequently in the future?

Of course there is no way of knowing how conservative future Steering 
Councils will be. As Python gets older, all the obviously useful methods 
will already exist, and the rate of new methods being added will likely
decrease.

But if you look at the docs for builtins:

https://docs.python.org/3/library/stdtypes.html

and search for "New in version" you will get an idea of how often 
builtins gain new methods since Python 3. Similarly for other modules.


> Note also that the py2 to py3 transition was (Hopefully) an anomaly -- more
> subtle changes between versions make it less compelling to support old
> versions for very long.

Large frameworks and libraries will surely continue to support a large 
range of versions, even as they drop support for 2.7.


> 2) Someone asked a question about the term "Extension Methods" --  I assume
> it's "Extension attributes", where the new attribute could be anything, yes?

In principle, sure. That's a design question to be agreed upon.


[...]
> No -- we're not assuming Python users are idiots -- there is an important
> difference here:
> 
> from extensions import flatten
> flatten(mylist)
> 
> very clearly adds the name `flatten` to the current module namespace. 
> That itself can be confusing to total newbies, but yes, you can't get 
> anywhere with Python without knowing that. Granted, you still need 
> need to know what `flatten` is, and what it does some other way in any 
> case.

Indeed. There's nothing "very clearly" about that until you have learned 
how imports work and why `from... import...` is different from `import`.

> Whereas:
> 
> from extensions use flatten
> mylist.flatten()
> 
> does NOT import the name `flatten` into the local namespace

Correct -- it "imports" it into the relevant class namespace, or 
whatever language people will use to describe it.

> -- which I suppose will be "clear" because it's using "use" rather 
> than a regular import, but that might be subtle. But importantly, what 
> it has done is add a name to some particular type -- what type? who 
> knows?

That's a fair observation. Do you think that C# uses of LINQ are 
confused by what is being modified when they say this?

using System.Linq;

Of course extension methods will be *new and different* until they 
become old and familiar. It may be that the syntax will make it clear 
not just where you are "importing" extensions from but what types will 
be included.

That is an excellent point to raise, thank you.


> In this example, you used the name "mylist", so I can assume it's an
> extension to list. But if that variable were called "stuff", I"d have
> absolutely no idea.

Yes, and we can write obfuscated variable names in Python today too :-)


> And as above, you'd need to go find the documentation
> for the flatten extension method, just as you would for any name in a
> module, but somehow functions feel more obvious to me.

You're not wrong. I dare say that there will be a learning curve 
involved with extension methods, like any new technology. If you've 
never learned about them, it might be confusing to see:

mylist.flatten()

in code and then try `help(list.flatten)` in the interactive interpreter 
and get an AttributeError exception, because you didn't notice the 
"using" at the top of the module.

But how is that different from seeing:

now = time()

in code and then `help(time)` raises a NameError because you didn't 
notice the import at the top of the module?

There's a learning curve in learning to use any tool, and that includes 
learning to program.


[...]
> Are you thinking that you could extend an ABC? Or if not that, then at
> least a superclass and get all subclasses? I'm a bit confused about how the
> MRO might work there.

If ABCs use normal attribute lookup, there's no reason why extension 
methods shouldn't work with them.


-- 
Steve
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/GVZBZ3VDMHQURYHQSIQ3KM6K5ZDNZ2Z4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

[Steven D'Aprano]

On Thu, Jun 24, 2021 at 12:17:17AM +1000, Chris Angelico wrote:
> On Wed, Jun 23, 2021 at 11:25 PM Steven D'Aprano  wrote:

> > To a first approximation (ignoring shadowing) every dot lookup can be
> > replaced with getattr and vice versa:
> >
> > obj.name <--> getattr(obj, 'name')
> >
> > A simple source code transformation could handle that, and the behaviour
> > of the code should be the same. Extension methods shouldn't change that.
> 
> Alright. In that case, getattr() has stopped being a function, and is
> now a magical construct of the compiler.

How do you come to that conclusion?

Did you miss the part where I said "To a first approximation (ignoring 
shadowing)"?

What I'm describing is not some proposed change, it is the status quo. 
getattr is equivalent to dot notation, as it always has been, all the 
way back to 1.5 or older. There's no change there.

getattr is a function. A regular, plain, ordinary builtin function. You 
can shadow it, or bind it to another name, or reach into builtins and 
delete it, just like every other builtin function. But if you don't do 
any of those things, then it is functionally equivalent to dot lookup.


> What happens if I do this?
> 
> if random.randrange(2):
> def getattr(obj, attr): return lambda: "Hello, world"
> 
> def foo(thing):
> return getattr(thing, "in_order")()

According to the result of the random number generator, either the 
lambda will be returned by the shadowed getattr, or the attribute 
"in_order" will be looked up on obj.

Just like today.


> If getattr is a perfectly ordinary function, as it now is, then it 
> should be perfectly acceptable to shadow it.

Correct.

> It should also be perfectly acceptable to use
> any other way of accessing attributes - for instance, the
> PyObject_GetAttr() function in C.

Again, correct.


> Why should getattr() become magical?

It doesn't.


> > > What exactly are the semantics of getattr?
> >
> > Oh gods, I don't know the exact semantics of attribute look ups now!
> > Something like this, I think:
> >
> > obj.attr (same as getattr(obj, 'attr'):
> 
> That is exactly what's weird about it. Instead of looking up the name
> getattr and then calling a perfectly ordinary function, now it has to
> be a magical construct of the compiler, handled right there. It is, in
> fact, impossible to craft equivalent semantics in a third-party
> function.

I don't think that it is impossible to emulate attribute lookup in pure 
Python code. It's complicated, to be sure, but I'm confident it can be 
done. Check out the Descriptor How To Guide, which is old but as far as 
I can tell still pretty accurate in its description of how attributes 
are looked up.


> Currently, getattr() can be defined in C on top of the C API function
> PyObject_GetAttr, which looks solely at the object and not the
> execution context. By your proposal, getattr() can only be compiler
> magic.

The only "magic" that is needed is the ability to inspect the call stack 
to find out the module being called from. CPython provides functions to 
do that in the inspect library: `inspect.stack` and `inspect.getmodule`. 
Strictly speaking, they are not portable Python, but any interpreter 
ought to be able to provide analogous abilities.

Do you think that the functions in the gc library are "compiler magic"? 
It would be next to impossible to emulate them from pure Python in an 
interpeter-independent fashion. Some interpreters don't even have 
reference counts.

How about locals()? That too has a privileged implementation, capable of 
doing things likely impossible from pure, implementation-independent 
Python code. Its still a plain old regular builtin function that can be 
shadowed, renamed and deleted.


> > Aside from the possibility that it might be shadowed or deleted from
> > builtins, can you give me any examples where `obj.attr` and
> > `getattr(obj. 'attr')` behave differently? Even *one* example?
> 
> That is *precisely* the possibility.

And it will remain the possibility.

> That is exactly why it is magical by your definition

It really won't.


> But if it's possible to do a source code transformation from
> getattr(obj, "attr") to obj.attr, then it is no longer possible to do
> *ANY* of this. You can't have an alias for getattr, you can't have a
> wrapper around it, you can't write your own version of it.

In the absence of any shadowing or monkey-patching of builtins.

 
-- 
Steve
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/4U3UB7GLN2N7KQ4MVWJJPSFEJUW2KFRU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

[Simão Afonso]

On 2021-06-24 09:19:36, Christopher Barker wrote:
> No -- we're not assuming Python users are idiots -- there is an important
> difference here:
> 
> from extensions import flatten
> flatten(mylist)
> 
> very clearly adds the name `flatten` to the current module namespace. That
> itself can be confusing to total newbies, but yes, you can't get anywhere
> with Python without knowing that. Granted, you still need need to know what
> `flatten` is, and what it does some other way in any case. Whereas:
> 
> from extensions use flatten
> mylist.flatten()
> 
> does NOT import the name `flatten` into the local namespace -- which I
> suppose will be "clear" because it's using "use" rather than a regular
> import, but that might be subtle. But importantly, what it has done is add
> a name to some particular type -- what type? who knows?

This explicit namespacing is an important objection, IMHO.

What about this syntax:

> from extensions use flatten in list

And symmetrically:

> from extensions use * in list

Going even further (maybe too far):

> from extensions use * in *
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/WOINSXVW3HNDIY5ZKI72KCSGIZMYI55J/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

[Christopher Barker]

Steven, you're making a pretty good case here, but a couple questions:

1) The case of newer versions of python adding methods to builtins, like
.bit_length is really compelling. But I do wonder how frequently that comes
up. It seems to me on this list, that people are very reluctant to add
methods to anything builtin (other than dunders) -- particularly ABCs, or
classes that might be subclassed, as that might break anything that
currently uses that same name. Anyway, is this a one-off? or something that
is likely to come up semi-frequently in the future?

Note also that the py2 to py3 transition was (Hopefully) an anomaly -- more
subtle changes between versions make it less compelling to support old
versions for very long.

2) Someone asked a question about the term "Extension Methods" --  I assume
it's "Extension attributes", where the new attribute could be anything, yes?

2) Comprehensibility:

Seriously, there's a time to realise when arguments against a feature
> devolve down to utterly spurious claims that Python programmers are
> idiots who will be confused by:
>
> from extensions use flatten
> mylist.flatten()
>
> but can instantly understand:
>
> from extensions import flatten
> flatten(mylist)
>

No -- we're not assuming Python users are idiots -- there is an important
difference here:

from extensions import flatten
flatten(mylist)

very clearly adds the name `flatten` to the current module namespace. That
itself can be confusing to total newbies, but yes, you can't get anywhere
with Python without knowing that. Granted, you still need need to know what
`flatten` is, and what it does some other way in any case. Whereas:

from extensions use flatten
mylist.flatten()

does NOT import the name `flatten` into the local namespace -- which I
suppose will be "clear" because it's using "use" rather than a regular
import, but that might be subtle. But importantly, what it has done is add
a name to some particular type -- what type? who knows?

In this example, you used the name "mylist", so I can assume it's an
extension to list. But if that variable were called "stuff", I"d have
absolutely no idea. And as above, you'd need to go find the documentation
for the flatten extension method, just as you would for any name in a
module, but somehow functions feel more obvious to me.

Thinking about this I've found what I think is a key issue for why this may
be far less useful for Python that it is for other languages. Using teh
"flatten" example, which I imagine you did due to the recent discussion on
this list about the such a function as a potential new builtin:

Python is dynamically Polymorphic (I may have just made that term up -- I
guess it's the same as duck typed) -- but what that means in that context
is that I don't usually care exactly what type an object is -- only that it
supports particular functionality, so, for instance:

from my_utilities import flatten

def func_that_works_with_nested_objects(the_things):
all_the_things_in_one  = flatten(the_things)
...

Presumably, that could work with any iterable with iterables in it.

but

from my_extensions use flatten

def func_that_works_with_nested_objects(the_things):
all_the_things_in_one  = the_things.flatten

OOPS! that's only going to work with actual lists.

Are you thinking that you could extend an ABC? Or if not that, then at
least a superclass and get all subclasses? I'm a bit confused about how the
MRO might work there.

Anyway, in my mind THAT is the big difference between Python and at least
mony  of the languages that support extension methods.

A "solution" would be to do what we do with numpy -- it has an "asarray()"
function that is a no-op if the argument is a numpy array, and creates an
array if it's not. WE often put that at the top of a function, so that we
can then use all the nifty array stuff inside the function, but not
requires the caller to create an array firat. But that buys ALL the numpy
functionality, it would be serious overkill for a method or two.

It's not a reason it couldn't work, or be useful, but certainly a lot less
useful than it might be.

In fact, the example for int.bit_length may b e the only compelling use
case -- not that method per se, but a built-in type that is rarely
duck-typed. That would be integers, floats and strings, at least those are
the most common. even ints and floats are two types that are frequently
used  interchangeably.

Side note:

I don't see the relevance to extension methods. You seem to be just
> listing random facts as if they were objections to the extension method
> proposal.
>

Let's keep this civil, and assume good intentions -- if something is
irrelevant, it's irrelevant, but please don't assume that the argument was
not made in good faith. For my part I've been following this thread, but
only recently understood the scope of the proposal well enough to know that
e.g. the above issue was not relevant.

-Chris B

-- 
Christopher 

[Python-ideas] Re: Extension methods in Python

[Rob Cliffe via Python-ideas]

On 24/06/2021 12:44, Richard Damon wrote:

On 6/24/21 7:09 AM, Simão Afonso wrote:

On 2021-06-24 20:59:31, Steven D'Aprano wrote:

Seriously, there's a time to realise when arguments against a feature
devolve down to utterly spurious claims that Python programmers are
idiots who will be confused by:

 from extensions use flatten
 mylist.flatten()

but can instantly understand:

 from extensions import flatten
 flatten(mylist)

Does this mean importing a module can modify other objects, including
builtins? Should this spooky-action-at-a-distance be encouraged?

OTOH, this already happens in the stdlib with rlcompleter, I assume
using monkey-patching. This is a special case for interactive use,
though.

https://docs.python.org/3/library/rlcompleter.html

Yes, importing a module runs the global code in that module, and that
code can not only define the various things in that module but can also
manipulate the contents of other modules.

This doesn't mean that spooky-action-at-a-distance is always good, but
sometimes it is what is needed. You need to be aware of the power that
you wield.

+1.  E.g. I have a module that converts `print` to a version that 
prompts for continuation after each screenful of output.  Very handy!  
Power is good, as long as it's used with discretion.

Rob Cliffe
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/3CLX65EN4JQVNGTPYFQKKJYOV4FRFVSW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

[Richard Damon]

On 6/24/21 7:09 AM, Simão Afonso wrote:
> On 2021-06-24 20:59:31, Steven D'Aprano wrote:
>> Seriously, there's a time to realise when arguments against a feature 
>> devolve down to utterly spurious claims that Python programmers are 
>> idiots who will be confused by:
>>
>> from extensions use flatten
>> mylist.flatten()
>>
>> but can instantly understand:
>>
>> from extensions import flatten
>> flatten(mylist)
> Does this mean importing a module can modify other objects, including
> builtins? Should this spooky-action-at-a-distance be encouraged?
>
> OTOH, this already happens in the stdlib with rlcompleter, I assume
> using monkey-patching. This is a special case for interactive use,
> though.
>
> https://docs.python.org/3/library/rlcompleter.html

Yes, importing a module runs the global code in that module, and that
code can not only define the various things in that module but can also
manipulate the contents of other modules.

This doesn't mean that spooky-action-at-a-distance is always good, but
sometimes it is what is needed. You need to be aware of the power that
you wield.

-- 
Richard Damon

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/I555D55DJBOV4V3MTZ2IJQ3MIDDOHQKV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

[Simão Afonso]

On 2021-06-24 20:59:31, Steven D'Aprano wrote:
> Seriously, there's a time to realise when arguments against a feature 
> devolve down to utterly spurious claims that Python programmers are 
> idiots who will be confused by:
> 
> from extensions use flatten
> mylist.flatten()
> 
> but can instantly understand:
> 
> from extensions import flatten
> flatten(mylist)

Does this mean importing a module can modify other objects, including
builtins? Should this spooky-action-at-a-distance be encouraged?

OTOH, this already happens in the stdlib with rlcompleter, I assume
using monkey-patching. This is a special case for interactive use,
though.

https://docs.python.org/3/library/rlcompleter.html
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/JVBVM6QM3DOEGMOJNCAOOHK4V5RLYPQS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

[Steven D'Aprano]

On Wed, Jun 23, 2021 at 11:22:26AM -0700, Brendan Barnwell wrote:
> On 2021-06-23 03:02, Steven D'Aprano wrote:
> >Attribute lookups are just another form of name lookup. Name lookups
> >depend on the current execution scope, not the caller's scope. With
> >extension methods, so do attribute lookups.
> 
>   But that's the thing, they aren't.

Of course attribute lookups are another form of name lookup. One hint 
that this is the case is that some languages, such as Java, call 
attributes *variables*, just like local and global variables. Both name 
and attribute lookups are looking up some named variable in some 
namespace. This shouldn't be controversial.

When you look up bare name:

x

the interpreter follows the LEGB rule and looks for `x` in the local 
scope, the enclosing scope, the global scope and then the builtin scope. 
There are some complications to do with locals and class scopes, and 
comprehensions, etc, but fundamentally you are searching namespaces for 
names. Think of the LEGB rule as analogous to an MRO.


When you look up a dotted name:

obj.x

the interpreter looks for `x` in the instance scope, the class scope, 
and any superclass scopes. The detailed search rules are different, what 
with decorators, inheritance, etc, the MRO could be indefinitely long, 
and there are dynamic attributes (`__getattr__`) too.

But fundamentally, although the details are different, attribute access 
and name lookup are both looking up names in namespaces.


> You gave a bunch of examples of 
> lexical scope semantics with imports and function locals vs globals. 
> But attribute lookups do not work that way.  Attribute lookups are 
> defined to work via a FUNCTION CALL to the __getattribute__ (and thence 
> often the __getattr__) of the OBJECT whose attribute is being looked up. 

There's that weird obsession with "function call" again. Why do you 
think that makes a difference?


> They do not in any way depend on the name via which that object is 
> accessed.

U... yes? Why is that relevant?


>   Now of course you can say that you want to make a new rule that 
>   throws the old rules out the window.

That's pure FUD. Extension methods don't require throwing the old rules 
out the window. The old rules continue to apply.


> We can do that for anything.  We can 
> define a new rule that says now when you do attribute lookups it will 
> call a global function called attribute_lookups_on_tuesdays if it's a 
> Tuesday in your timezone.

If you want to do that then go ahead and propose it in another thread, 
but I don't want anything like that strawman.


> But what I'm saying is that the way attribute 
> lookups currently work is not the same as the way bare-name lookups 
> work, because attribute lookups are localized to the object

You might have heard of something called "inheritance". And dynamic 
attributes. Maybe even mixins or traits.

Attribute lookups are not localised to the object.


> (not the name!) and bare-name lookups are not.  I consider this 
> difference fundamental to Python.

There are plenty of differences between name lookups and attribute 
lookups, but locality is not one of them.


> It's why locals() isn't really how local name lookups work (which came 
> up elsewhere in this thread).

I don't see the relevance to extension methods.


> It's why you 
> can't magically hook into "x = my_obj" and create some magical behavior 
> that depends on my_obj.

I don't see the relevance to extension methods. You seem to be just 
listing random facts as if they were objections to the extension method 
proposal.

You're right, we can't hook into assignment to a bare name. So what? We 
*can* hook into attribute lookups.


>   As for other languages, you keep referencing them as if the 
> fact that something known as "extension methods" exists in those other 
> languages makes it self-evident that it would be useful in Python.  
> Python isn't those other languages.  I'm not familiar with all of the 
> other languages you mentioned, but I'll bet that at least some of them 
> do not have the same name/attribute lookup rules and dunder-governed 
> object-customization setup as Python.  So that's the difference.

I think that's a difference that makes no difference.

What if I told you that it is likely that Python's name/attribute lookup 
rules and dunder-governed object-customization are the key features that 
would make extension methods possible with little or no interpreter 
support?

As I described in a previous post, adding extension methods would be a 
very small change to the existing attribute lookup rules. The tricky 
part is to come up with a fast, efficient registration system for 
determining when to use them.


> The fact that extension methods happen to exist and be useful in those 
> languages is really neither here nor there.

Extension methods don't just "happen to exist", they were designed to 
solve a real problem. That's a problem that can apply 

[Python-ideas] Re: Extension methods in Python

[Chris Angelico]

On Thu, Jun 24, 2021 at 7:51 PM Steven D'Aprano  wrote:
>
> I'm not sure if you completely understand the use-case I was describing,
> so let me clarify for you with a concrete example.
>
> Ints have a "bit_length" method, starting from Python 2.7. I needed to
> use that method going all the way back to version 2.4. I have an
> implementation that works, so I could backport that method to 2.4
> through 2.6, except that you can't monkey-patch builtins in Python.
>
> So monkey-patching is out.
>
> (And besides, I wouldn't want to monkey-patch it: I only need that
> method in one module. I want to localise the change to only where it is
> needed.)
>
> Subclassing int wouldn't help. I need it to work on actual ints, and any
> third-party subclasses of int, not just my own custom subclass.
>
> (And besides, have you tried to subclass int? It's a real PITA. It's
> easy enough to write a subclass, but every operation on it returns an
> actual int instead of the subclass. So you have to write a ton of
> boilerplate to make int subclasses workable. But I digress.)
>
> So a subclass is not a good solution either.
>
> That leaves only a function.
>
> But that hurts code readability and maintainance. In 2.7 and above,
> bit_length is a method, not a function. All the documentation for
> bit_length assumes it is a method. Every tutorial that uses it has it as
> a method. Other code that uses it treats it as a method.
>
> Except my code, where it is a function.
>
> Using a function is not a *terrible* solution to the problem of
> backporting a new feature to older versions of Python. I've done it
> dozens of times and it's not awful. **But it could be better.**
>
> Why can't the backport be a method, just like in 2.7 and above?
>
> With extension methods, it can be.
>

You've given some great arguments for why (5).bit_length() should be
allowed to be a thing. (By the way - we keep saying "extension
METHODS", but should this be allowed to give non-function attributes
too?) But not once have you said where getattr(), hasattr(), etc come
into this. The biggest pushback against this proposal has been the
assumption that getattr(5, "bit_length")() would have to be the same
as (5).bit_length(). Why is that necessary? I've never seen any
examples of use-cases for that.

Let's tighten this up into a real proposal. (I'm only +0.5 on this,
but am willing to be swayed.)

* Each module has a registration of (type, name, function) triples.
* Each code object is associated with a module.
* Compiled code automatically links the module with the code object.
(If you instantiate a code object manually, it's on you to pick a
module appropriately.)
* Attribute lookups use three values: object, attribute name, and module.
* If the object does not have the attribute, its MRO is scanned
sequentially for a registered method. If one is found, use it.

Not mentioned in this proposal: anything relating to getattr or
hasattr, which will continue to look only at real methods. There may
need to be an enhanced version of PyObject_GetAttr which is able to
look up extension methods, but the current one simply wouldn't.

Also not mentioned: ABC registration. If you register a class as a
subclass of an ABC and then register an extension method on that
class, isinstance() will say that it's an instance of the ABC, but the
extension method won't be there. I'm inclined to say "tough luck,
don't do that", but if there are strong enough use cases, that could
be added.

But otherwise, I would FAR prefer a much simpler proposal, one which
changes only the things that need to be changed.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/WNDEVXU57APUPLYXHKJCRGG4YAM3H75Z/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

[Steven D'Aprano]

Oops, sorry, I neglected to trim my response to João. Please ignore my  
previous response, with the untrimmed quoting at the start, and give any 
replies to this.


On Wed, Jun 23, 2021 at 07:23:19PM +0200, João Santos wrote:

> Of course that means the the standard library might also introduce 
> something new that will be shadowed by one of your custom methods, 

Extension methods have lower priority than actual methods on the class. 
So that won't happen. The actual method on the class will shadow the 
extension method.

I'm not sure if you completely understand the use-case I was describing, 
so let me clarify for you with a concrete example.

Ints have a "bit_length" method, starting from Python 2.7. I needed to 
use that method going all the way back to version 2.4. I have an 
implementation that works, so I could backport that method to 2.4 
through 2.6, except that you can't monkey-patch builtins in Python.

So monkey-patching is out.

(And besides, I wouldn't want to monkey-patch it: I only need that 
method in one module. I want to localise the change to only where it is 
needed.)

Subclassing int wouldn't help. I need it to work on actual ints, and any 
third-party subclasses of int, not just my own custom subclass.

(And besides, have you tried to subclass int? It's a real PITA. It's 
easy enough to write a subclass, but every operation on it returns an 
actual int instead of the subclass. So you have to write a ton of 
boilerplate to make int subclasses workable. But I digress.)

So a subclass is not a good solution either.

That leaves only a function.

But that hurts code readability and maintainance. In 2.7 and above, 
bit_length is a method, not a function. All the documentation for 
bit_length assumes it is a method. Every tutorial that uses it has it as 
a method. Other code that uses it treats it as a method.

Except my code, where it is a function.

Using a function is not a *terrible* solution to the problem of 
backporting a new feature to older versions of Python. I've done it 
dozens of times and it's not awful. **But it could be better.**

Why can't the backport be a method, just like in 2.7 and above?

With extension methods, it can be.

Obviously not for Python 2.x code. But plan for the future: if we have 
extension methods in the language, eventually every version of Python we 
care about will support it. And then writing compatibility layers will 
be much simpler.


> and then you'll wish you had just used functions or a wrapper class.

Believe me, I won't. I've written dozens of compatibility functions over 
the last decade or more, going back to Python 2.3. I've written hybrid 
2/3 code. Extension methods would not always be useful, but for cases 
like int.bit_length, it would be a far superior solution.


> If you can import extension methods wholesale, you might even be 
> monkeypatching something without realising it

Extension methods is not monkey-patching. It is like adding a global 
name to one module. If I write:

def func(arg):
...

in module A.py, that does not introduce func to any other module unless 
those other modules explicitly import it.

Extension methods are exactly analogous: they are only visible in the 
module where you opt-in to use them. They don't monkey-patch the entire 
interpreter-wide environment.

And because extension methods have a lower priority than actual methods, 
you cannot override an existing method on a class. You can only extend 
the class with a new method.



-- 
Steve
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/O43B665WNRF5W3PIUPRBIOWAFJ4H7CS3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

[Steven D'Aprano]

On Wed, Jun 23, 2021 at 07:23:19PM +0200, João Santos wrote:
> 
> 
> On Wed, Jun 23 2021 at 20:48:39 +1000, Steven D'Aprano 
>  wrote:
> >I've just thought of a great use-case for extension methods.
> >
> >Hands up who has to write code that runs under multiple versions of
> >Python? *raises my hand*
> >
> >I'm sure I'm not the only one. You probably have written compatibility
> >functions like this:
> >
> >def bit_length(num):
> >try:
> >return num.bit_length()
> >except AttributeError:
> ># fallback implementation goes here
> >...
> >
> >
> >and then everywhere you want to write `n.bit_length()`, you write
> >`bit_length(n)` instead.
> >
> >Extension methods would let us do this:
> >
> ># compatibility.py
> >@extends(int):
> >def bit_length(self):
> ># fallback implementation goes here
> > ...
> >
> >
> ># mylibrary.py
> >using compatibility
> >num = 42
> >num.bit_length()
> >
> >
> >Now obviously that isn't going to help with versions too old to
> >support extension methods, but eventually extension methods will
> >be available in the oldest version of Python you care about:
> >
> ># supports Python 3.14 and above
> >
> >Once we reach that point, then backporting new methods to classes
> >becomes a simple matter of using an extension method. No mess, no 
> >fuss.
> >
> >As someone who has written a lot of code like that first bit_length
> >compatibility function in my time, I think I've just gone from "Yeah,
> >extension methods seem useful..." to "OMG I WANT THEM TEN YEARS AGO 
> >SO I
> >CAN USE THEM RIGHT NOW!!!".
> >
> >
> >Backporting might not be your killer-app for extension methods, but I
> >really do think they might be mine.
> >
> >
> >--
> >Steve
> >___
> >Python-ideas mailing list -- python-ideas@python.org 
> >
> >To unsubscribe send an email to python-ideas-le...@python.org 
> >
> >
> >Message archived at 
> >
> >Code of Conduct: 
> 



> Of course that means the the standard library might also introduce 
> something new that will be shadowed by one of your custom methods, 

Extension methods have lower priority than actual methods on the class. 
So that won't happen. The actual method on the class will shadow the 
extension method.

I'm not sure if you completely understand the use-case I was describing, 
so let me clarify for you with a concrete example.

Ints have a "bit_length" method, starting from Python 2.7. I needed to 
use that method going all the way back to version 2.4. I have an 
implementation that works, so I could backport that method to 2.4 
through 2.6, except that you can't monkey-patch builtins in Python.

So monkey-patching is out.

(And besides, I wouldn't want to monkey-patch it: I only need that 
method in one module. I want to localise the change to only where it is 
needed.)

Subclassing int wouldn't help. I need it to work on actual ints, and any 
third-party subclasses of int, not just my own custom subclass.

(And besides, have you tried to subclass int? It's a real PITA. It's 
easy enough to write a subclass, but every operation on it returns an 
actual int instead of the subclass. So you have to write a ton of 
boilerplate to make int subclasses workable. But I digress.)

So a subclass is not a good solution either.

That leaves only a function.

But that hurts code readability and maintainance. In 2.7 and above, 
bit_length is a method, not a function. All the documentation for 
bit_length assumes it is a method. Every tutorial that uses it has it as 
a method. Other code that uses it treats it as a method.

Except my code, where it is a function.

Using a function is not a *terrible* solution to the problem of 
backporting a new feature to older versions of Python. I've done it 
dozens of times and it's not awful. **But it could be better.**

Why can't the backport be a method, just like in 2.7 and above?

With extension methods, it can be.

Obviously not for Python 2.x code. But plan for the future: if we have 
extension methods in the language, eventually every version of Python we 
care about will support it. And then writing compatibility layers will 
be much simpler.


> and then you'll wish you had just used functions or a wrapper class.

Believe me, I won't. I've written dozens of compatibility functions over 
the last decade or more, going back to Python 2.3. I've written hybrid 
2/3 code. Extension methods would not always be useful, but for cases 
like int.bit_length, it would be a far superior solution.


> If you can import extension methods wholesale, you might even be 
> monkeypatching something without realising it

[Python-ideas] Re: Fill in missing contextvars/asyncio task support

[Mark Gordon]

I've read the PEP and understand what's implemented. However there is pretty 
limited discussion about what the design constraints were and what 
intended/recommended usage would look like. I'll answer my own question:

1. If all we wanted was a version of TLS that worked in an analogous way 
extending (synchronous code, threads) to (async code, tasks) then you don't 
need anything fancy, a simple dictionary backing the "context" will do. This is 
all that you need to solve the "decimal formatting" problem, for instance.

2. However, the scope of PEP 567 was increased to something greater. It was 
decided that we want tasks/threads to be able to inherit an existing context. 
This is a unique feature with no analog in TLS. I believe a motivating use case 
was for a request/response server that may spawn worker tasks off the main task 
and want to store request context information in contextvars.

3. Additionally, to continue to have the "decimal formatting" solution work 
correctly it's necessary that no two tasks/threads are running on the same 
context. This means "inherting" should mean "running on a copy of".

These constraints strongly suggest an interface of:

contextvars.get_context() -> Context
Context.run_in_copy(func) -> Context
Context.async_run_in_copy(coro) -> Context
*** NoContext.run method, no copy methods needed either ***

So what was the motivation behind having a copy_context() and a non-copying 
Context.run method? It seems to break the third design constraint allowing you 
to have multiple threads try and run on the same Context.

Nonetheless, "asyncio.run_in_context()" is a direct analog of "Context.run" and 
should be a clear add to the API from a symmetry point of view. We are already 
beyond the point where constraint three is being strictly enforced. I'm not 
sure what argument against this API wouldn't apply to "Context.run()" as well.

If it's still a -1 on "asyncio.run_in_context()" what about 
"asyncio.run_in_context_copy(context, coro) -> Context" that copies the passed 
context and runs the coroutine in the task using that context? If we go this 
route maybe we would plan on deprecating Context.run and replacing it with a 
Context.run_in_copy method?
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/4SYYJEVIV3ZUSVBZIL34EZTIEMBRV533/
Code of Conduct: http://python.org/psf/codeofconduct/