subject:"\[Python\-ideas\] Re\: Extension methods in Python"

Here's a quick and dirty proof of concept I knocked up in about 20 
minutes, demonstrating that no deep compiler magic is needed. It's just 
a small change to the way `object.__getattribute__` works.

I've emulated it with my own base class, since `object` can't be 
monkey-patched.

The proof of concept is probably buggy and incomplete. It isn't intended 
to be a final, polished production-ready implementation. It's not 
implementation-agnostic: it requires the ability to inspect the call 
stack. If you're using IronPython, this may not work.

You will notice I didn't need to touch getattr to have it work, let 
alone hack the interpreter to make it some sort of magical construct. It 
all works through `__getattribute__`.

The registration system is just the easiest thing that I could throw 
together. There are surely better designs.

Run A.py to see it in action.


-- 
Steve
# Extension method helpers and proof of concept.
import inspect

MODULE_REGISTRY = set()
METHOD_REGISTRY = {}

def get_execution_scope():
frm = inspect.stack()[2]
return inspect.getmodule(frm[0])

def extends(cls):
def inner(func):
METHOD_REGISTRY.setdefault(cls, set()).add(func)
return func
return inner

def using():
MODULE_REGISTRY.add(get_execution_scope())

class MyObject:
# Base class that supports extension methods.
def __getattribute__(self, name):
try:
return super().__getattribute__(name)
except AttributeError:
mod = get_execution_scope()
if mod in MODULE_REGISTRY:
xmethods = METHOD_REGISTRY.get(type(self), set())
for func in xmethods:
if func.__name__ == name:
return func.__get__(self)
raise

class Demo(MyObject):
pass

@extends(Demo)
def xmethod(self):
return "called extension method"
from extmethods import Demo

instance = Demo()
print('(Module A) has xmethod?', hasattr(instance, 'xmethod'))

import B
B.main()

print('(Module A) has xmethod?', hasattr(instance, 'xmethod'))
from extmethods import using, Demo

using()

def main():
obj = Demo()
print('(Module B) has xmethod?', hasattr(obj, 'xmethod'))
print(getattr(obj, 'xmethod')())
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/GINVBO2BIJNPSTT65DQ37YQF2PFZQYST/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

On Thu, Jun 24, 2021 at 09:19:36AM -0700, Christopher Barker wrote:

> Steven, you're making a pretty good case here, but a couple questions:
> 
> 1) The case of newer versions of python adding methods to builtins, like
> .bit_length is really compelling. But I do wonder how frequently that comes
> up. It seems to me on this list, that people are very reluctant to add
> methods to anything builtin (other than dunders) -- particularly ABCs, or
> classes that might be subclassed, as that might break anything that
> currently uses that same name. Anyway, is this a one-off? or something that
> is likely to come up semi-frequently in the future?

Of course there is no way of knowing how conservative future Steering 
Councils will be. As Python gets older, all the obviously useful methods 
will already exist, and the rate of new methods being added will likely
decrease.

But if you look at the docs for builtins:

https://docs.python.org/3/library/stdtypes.html

and search for "New in version" you will get an idea of how often 
builtins gain new methods since Python 3. Similarly for other modules.

> Note also that the py2 to py3 transition was (Hopefully) an anomaly -- more
> subtle changes between versions make it less compelling to support old
> versions for very long.

Large frameworks and libraries will surely continue to support a large 
range of versions, even as they drop support for 2.7.

> 2) Someone asked a question about the term "Extension Methods" --  I assume
> it's "Extension attributes", where the new attribute could be anything, yes?

In principle, sure. That's a design question to be agreed upon.

[...]
> No -- we're not assuming Python users are idiots -- there is an important
> difference here:
> 
> from extensions import flatten
> flatten(mylist)
> 
> very clearly adds the name `flatten` to the current module namespace. 
> That itself can be confusing to total newbies, but yes, you can't get 
> anywhere with Python without knowing that. Granted, you still need 
> need to know what `flatten` is, and what it does some other way in any 
> case.

Indeed. There's nothing "very clearly" about that until you have learned 
how imports work and why `from... import...` is different from `import`.

> Whereas:
> 
> from extensions use flatten
> mylist.flatten()
> 
> does NOT import the name `flatten` into the local namespace

Correct -- it "imports" it into the relevant class namespace, or 
whatever language people will use to describe it.

> -- which I suppose will be "clear" because it's using "use" rather 
> than a regular import, but that might be subtle. But importantly, what 
> it has done is add a name to some particular type -- what type? who 
> knows?

That's a fair observation. Do you think that C# uses of LINQ are 
confused by what is being modified when they say this?

using System.Linq;

Of course extension methods will be *new and different* until they 
become old and familiar. It may be that the syntax will make it clear 
not just where you are "importing" extensions from but what types will 
be included.

That is an excellent point to raise, thank you.

> In this example, you used the name "mylist", so I can assume it's an
> extension to list. But if that variable were called "stuff", I"d have
> absolutely no idea.

Yes, and we can write obfuscated variable names in Python today too :-)

> And as above, you'd need to go find the documentation
> for the flatten extension method, just as you would for any name in a
> module, but somehow functions feel more obvious to me.

You're not wrong. I dare say that there will be a learning curve 
involved with extension methods, like any new technology. If you've 
never learned about them, it might be confusing to see:

mylist.flatten()

in code and then try `help(list.flatten)` in the interactive interpreter 
and get an AttributeError exception, because you didn't notice the 
"using" at the top of the module.

But how is that different from seeing:

now = time()

in code and then `help(time)` raises a NameError because you didn't 
notice the import at the top of the module?

There's a learning curve in learning to use any tool, and that includes 
learning to program.

[...]
> Are you thinking that you could extend an ABC? Or if not that, then at
> least a superclass and get all subclasses? I'm a bit confused about how the
> MRO might work there.

If ABCs use normal attribute lookup, there's no reason why extension 
methods shouldn't work with them.

-- 
Steve
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/GVZBZ3VDMHQURYHQSIQ3KM6K5ZDNZ2Z4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

On Thu, Jun 24, 2021 at 12:17:17AM +1000, Chris Angelico wrote:
> On Wed, Jun 23, 2021 at 11:25 PM Steven D'Aprano  wrote:

> > To a first approximation (ignoring shadowing) every dot lookup can be
> > replaced with getattr and vice versa:
> >
> > obj.name <--> getattr(obj, 'name')
> >
> > A simple source code transformation could handle that, and the behaviour
> > of the code should be the same. Extension methods shouldn't change that.
> 
> Alright. In that case, getattr() has stopped being a function, and is
> now a magical construct of the compiler.

How do you come to that conclusion?

Did you miss the part where I said "To a first approximation (ignoring 
shadowing)"?

What I'm describing is not some proposed change, it is the status quo. 
getattr is equivalent to dot notation, as it always has been, all the 
way back to 1.5 or older. There's no change there.

getattr is a function. A regular, plain, ordinary builtin function. You 
can shadow it, or bind it to another name, or reach into builtins and 
delete it, just like every other builtin function. But if you don't do 
any of those things, then it is functionally equivalent to dot lookup.

> What happens if I do this?
> 
> if random.randrange(2):
> def getattr(obj, attr): return lambda: "Hello, world"
> 
> def foo(thing):
> return getattr(thing, "in_order")()

According to the result of the random number generator, either the 
lambda will be returned by the shadowed getattr, or the attribute 
"in_order" will be looked up on obj.

Just like today.

> If getattr is a perfectly ordinary function, as it now is, then it 
> should be perfectly acceptable to shadow it.

Correct.

> It should also be perfectly acceptable to use
> any other way of accessing attributes - for instance, the
> PyObject_GetAttr() function in C.

Again, correct.

> Why should getattr() become magical?

It doesn't.

> > > What exactly are the semantics of getattr?
> >
> > Oh gods, I don't know the exact semantics of attribute look ups now!
> > Something like this, I think:
> >
> > obj.attr (same as getattr(obj, 'attr'):
> 
> That is exactly what's weird about it. Instead of looking up the name
> getattr and then calling a perfectly ordinary function, now it has to
> be a magical construct of the compiler, handled right there. It is, in
> fact, impossible to craft equivalent semantics in a third-party
> function.

I don't think that it is impossible to emulate attribute lookup in pure 
Python code. It's complicated, to be sure, but I'm confident it can be 
done. Check out the Descriptor How To Guide, which is old but as far as 
I can tell still pretty accurate in its description of how attributes 
are looked up.

> Currently, getattr() can be defined in C on top of the C API function
> PyObject_GetAttr, which looks solely at the object and not the
> execution context. By your proposal, getattr() can only be compiler
> magic.

The only "magic" that is needed is the ability to inspect the call stack 
to find out the module being called from. CPython provides functions to 
do that in the inspect library: `inspect.stack` and `inspect.getmodule`. 
Strictly speaking, they are not portable Python, but any interpreter 
ought to be able to provide analogous abilities.

Do you think that the functions in the gc library are "compiler magic"? 
It would be next to impossible to emulate them from pure Python in an 
interpeter-independent fashion. Some interpreters don't even have 
reference counts.

How about locals()? That too has a privileged implementation, capable of 
doing things likely impossible from pure, implementation-independent 
Python code. Its still a plain old regular builtin function that can be 
shadowed, renamed and deleted.

> > Aside from the possibility that it might be shadowed or deleted from
> > builtins, can you give me any examples where `obj.attr` and
> > `getattr(obj. 'attr')` behave differently? Even *one* example?
> 
> That is *precisely* the possibility.

And it will remain the possibility.

> That is exactly why it is magical by your definition

It really won't.

> But if it's possible to do a source code transformation from
> getattr(obj, "attr") to obj.attr, then it is no longer possible to do
> *ANY* of this. You can't have an alias for getattr, you can't have a
> wrapper around it, you can't write your own version of it.

In the absence of any shadowing or monkey-patching of builtins.

-- 
Steve
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/4U3UB7GLN2N7KQ4MVWJJPSFEJUW2KFRU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

2021-06-24 Thread Simão Afonso

On 2021-06-24 09:19:36, Christopher Barker wrote:
> No -- we're not assuming Python users are idiots -- there is an important
> difference here:
> 
> from extensions import flatten
> flatten(mylist)
> 
> very clearly adds the name `flatten` to the current module namespace. That
> itself can be confusing to total newbies, but yes, you can't get anywhere
> with Python without knowing that. Granted, you still need need to know what
> `flatten` is, and what it does some other way in any case. Whereas:
> 
> from extensions use flatten
> mylist.flatten()
> 
> does NOT import the name `flatten` into the local namespace -- which I
> suppose will be "clear" because it's using "use" rather than a regular
> import, but that might be subtle. But importantly, what it has done is add
> a name to some particular type -- what type? who knows?

This explicit namespacing is an important objection, IMHO.

What about this syntax:

> from extensions use flatten in list

And symmetrically:

> from extensions use * in list

Going even further (maybe too far):

> from extensions use * in *
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/WOINSXVW3HNDIY5ZKI72KCSGIZMYI55J/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

2021-06-24 Thread Christopher Barker

Steven, you're making a pretty good case here, but a couple questions:

1) The case of newer versions of python adding methods to builtins, like
.bit_length is really compelling. But I do wonder how frequently that comes
up. It seems to me on this list, that people are very reluctant to add
methods to anything builtin (other than dunders) -- particularly ABCs, or
classes that might be subclassed, as that might break anything that
currently uses that same name. Anyway, is this a one-off? or something that
is likely to come up semi-frequently in the future?

Note also that the py2 to py3 transition was (Hopefully) an anomaly -- more
subtle changes between versions make it less compelling to support old
versions for very long.

2) Someone asked a question about the term "Extension Methods" --  I assume
it's "Extension attributes", where the new attribute could be anything, yes?

2) Comprehensibility:

Seriously, there's a time to realise when arguments against a feature
> devolve down to utterly spurious claims that Python programmers are
> idiots who will be confused by:
>
> from extensions use flatten
> mylist.flatten()
>
> but can instantly understand:
>
> from extensions import flatten
> flatten(mylist)
>

No -- we're not assuming Python users are idiots -- there is an important
difference here:

from extensions import flatten
flatten(mylist)

very clearly adds the name `flatten` to the current module namespace. That
itself can be confusing to total newbies, but yes, you can't get anywhere
with Python without knowing that. Granted, you still need need to know what
`flatten` is, and what it does some other way in any case. Whereas:

from extensions use flatten
mylist.flatten()

does NOT import the name `flatten` into the local namespace -- which I
suppose will be "clear" because it's using "use" rather than a regular
import, but that might be subtle. But importantly, what it has done is add
a name to some particular type -- what type? who knows?

In this example, you used the name "mylist", so I can assume it's an
extension to list. But if that variable were called "stuff", I"d have
absolutely no idea. And as above, you'd need to go find the documentation
for the flatten extension method, just as you would for any name in a
module, but somehow functions feel more obvious to me.

Thinking about this I've found what I think is a key issue for why this may
be far less useful for Python that it is for other languages. Using teh
"flatten" example, which I imagine you did due to the recent discussion on
this list about the such a function as a potential new builtin:

Python is dynamically Polymorphic (I may have just made that term up -- I
guess it's the same as duck typed) -- but what that means in that context
is that I don't usually care exactly what type an object is -- only that it
supports particular functionality, so, for instance:

from my_utilities import flatten

def func_that_works_with_nested_objects(the_things):
all_the_things_in_one  = flatten(the_things)
...

Presumably, that could work with any iterable with iterables in it.

but

from my_extensions use flatten

def func_that_works_with_nested_objects(the_things):
all_the_things_in_one  = the_things.flatten

OOPS! that's only going to work with actual lists.

Are you thinking that you could extend an ABC? Or if not that, then at
least a superclass and get all subclasses? I'm a bit confused about how the
MRO might work there.

Anyway, in my mind THAT is the big difference between Python and at least
mony  of the languages that support extension methods.

A "solution" would be to do what we do with numpy -- it has an "asarray()"
function that is a no-op if the argument is a numpy array, and creates an
array if it's not. WE often put that at the top of a function, so that we
can then use all the nifty array stuff inside the function, but not
requires the caller to create an array firat. But that buys ALL the numpy
functionality, it would be serious overkill for a method or two.

It's not a reason it couldn't work, or be useful, but certainly a lot less
useful than it might be.

In fact, the example for int.bit_length may b e the only compelling use
case -- not that method per se, but a built-in type that is rarely
duck-typed. That would be integers, floats and strings, at least those are
the most common. even ints and floats are two types that are frequently
used  interchangeably.

Side note:

I don't see the relevance to extension methods. You seem to be just
> listing random facts as if they were objections to the extension method
> proposal.
>

Let's keep this civil, and assume good intentions -- if something is
irrelevant, it's irrelevant, but please don't assume that the argument was
not made in good faith. For my part I've been following this thread, but
only recently understood the scope of the proposal well enough to know that
e.g. the above issue was not relevant.

-Chris B

-- 
Christopher

[Python-ideas] Re: Extension methods in Python

2021-06-24 Thread Rob Cliffe via Python-ideas




On 24/06/2021 12:44, Richard Damon wrote:

On 6/24/21 7:09 AM, Simão Afonso wrote:

On 2021-06-24 20:59:31, Steven D'Aprano wrote:

Seriously, there's a time to realise when arguments against a feature
devolve down to utterly spurious claims that Python programmers are
idiots who will be confused by:

 from extensions use flatten
 mylist.flatten()

but can instantly understand:

 from extensions import flatten
 flatten(mylist)

Does this mean importing a module can modify other objects, including
builtins? Should this spooky-action-at-a-distance be encouraged?

OTOH, this already happens in the stdlib with rlcompleter, I assume
using monkey-patching. This is a special case for interactive use,
though.

https://docs.python.org/3/library/rlcompleter.html

Yes, importing a module runs the global code in that module, and that
code can not only define the various things in that module but can also
manipulate the contents of other modules.

This doesn't mean that spooky-action-at-a-distance is always good, but
sometimes it is what is needed. You need to be aware of the power that
you wield.

+1.  E.g. I have a module that converts `print` to a version that 
prompts for continuation after each screenful of output.  Very handy!  
Power is good, as long as it's used with discretion.

Rob Cliffe
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/3CLX65EN4JQVNGTPYFQKKJYOV4FRFVSW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

2021-06-24 Thread Richard Damon

On 6/24/21 7:09 AM, Simão Afonso wrote:
> On 2021-06-24 20:59:31, Steven D'Aprano wrote:
>> Seriously, there's a time to realise when arguments against a feature 
>> devolve down to utterly spurious claims that Python programmers are 
>> idiots who will be confused by:
>>
>> from extensions use flatten
>> mylist.flatten()
>>
>> but can instantly understand:
>>
>> from extensions import flatten
>> flatten(mylist)
> Does this mean importing a module can modify other objects, including
> builtins? Should this spooky-action-at-a-distance be encouraged?
>
> OTOH, this already happens in the stdlib with rlcompleter, I assume
> using monkey-patching. This is a special case for interactive use,
> though.
>
> https://docs.python.org/3/library/rlcompleter.html

Yes, importing a module runs the global code in that module, and that
code can not only define the various things in that module but can also
manipulate the contents of other modules.

This doesn't mean that spooky-action-at-a-distance is always good, but
sometimes it is what is needed. You need to be aware of the power that
you wield.

-- 
Richard Damon

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/I555D55DJBOV4V3MTZ2IJQ3MIDDOHQKV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

2021-06-24 Thread Simão Afonso

On 2021-06-24 20:59:31, Steven D'Aprano wrote:
> Seriously, there's a time to realise when arguments against a feature 
> devolve down to utterly spurious claims that Python programmers are 
> idiots who will be confused by:
> 
> from extensions use flatten
> mylist.flatten()
> 
> but can instantly understand:
> 
> from extensions import flatten
> flatten(mylist)

Does this mean importing a module can modify other objects, including
builtins? Should this spooky-action-at-a-distance be encouraged?

OTOH, this already happens in the stdlib with rlcompleter, I assume
using monkey-patching. This is a special case for interactive use,
though.

https://docs.python.org/3/library/rlcompleter.html
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/JVBVM6QM3DOEGMOJNCAOOHK4V5RLYPQS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

On Wed, Jun 23, 2021 at 11:22:26AM -0700, Brendan Barnwell wrote:
> On 2021-06-23 03:02, Steven D'Aprano wrote:
> >Attribute lookups are just another form of name lookup. Name lookups
> >depend on the current execution scope, not the caller's scope. With
> >extension methods, so do attribute lookups.
> 
>   But that's the thing, they aren't.

Of course attribute lookups are another form of name lookup. One hint 
that this is the case is that some languages, such as Java, call 
attributes *variables*, just like local and global variables. Both name 
and attribute lookups are looking up some named variable in some 
namespace. This shouldn't be controversial.

When you look up bare name:

x

the interpreter follows the LEGB rule and looks for `x` in the local 
scope, the enclosing scope, the global scope and then the builtin scope. 
There are some complications to do with locals and class scopes, and 
comprehensions, etc, but fundamentally you are searching namespaces for 
names. Think of the LEGB rule as analogous to an MRO.

When you look up a dotted name:

obj.x

the interpreter looks for `x` in the instance scope, the class scope, 
and any superclass scopes. The detailed search rules are different, what 
with decorators, inheritance, etc, the MRO could be indefinitely long, 
and there are dynamic attributes (`__getattr__`) too.

But fundamentally, although the details are different, attribute access 
and name lookup are both looking up names in namespaces.

> You gave a bunch of examples of 
> lexical scope semantics with imports and function locals vs globals. 
> But attribute lookups do not work that way.  Attribute lookups are 
> defined to work via a FUNCTION CALL to the __getattribute__ (and thence 
> often the __getattr__) of the OBJECT whose attribute is being looked up. 

There's that weird obsession with "function call" again. Why do you 
think that makes a difference?

> They do not in any way depend on the name via which that object is 
> accessed.

U... yes? Why is that relevant?

>   Now of course you can say that you want to make a new rule that 
>   throws the old rules out the window.

That's pure FUD. Extension methods don't require throwing the old rules 
out the window. The old rules continue to apply.

> We can do that for anything.  We can 
> define a new rule that says now when you do attribute lookups it will 
> call a global function called attribute_lookups_on_tuesdays if it's a 
> Tuesday in your timezone.

If you want to do that then go ahead and propose it in another thread, 
but I don't want anything like that strawman.

> But what I'm saying is that the way attribute 
> lookups currently work is not the same as the way bare-name lookups 
> work, because attribute lookups are localized to the object

You might have heard of something called "inheritance". And dynamic 
attributes. Maybe even mixins or traits.

Attribute lookups are not localised to the object.

> (not the name!) and bare-name lookups are not.  I consider this 
> difference fundamental to Python.

There are plenty of differences between name lookups and attribute 
lookups, but locality is not one of them.

> It's why locals() isn't really how local name lookups work (which came 
> up elsewhere in this thread).

I don't see the relevance to extension methods.

> It's why you 
> can't magically hook into "x = my_obj" and create some magical behavior 
> that depends on my_obj.

I don't see the relevance to extension methods. You seem to be just 
listing random facts as if they were objections to the extension method 
proposal.

You're right, we can't hook into assignment to a bare name. So what? We 
*can* hook into attribute lookups.

>   As for other languages, you keep referencing them as if the 
> fact that something known as "extension methods" exists in those other 
> languages makes it self-evident that it would be useful in Python.  
> Python isn't those other languages.  I'm not familiar with all of the 
> other languages you mentioned, but I'll bet that at least some of them 
> do not have the same name/attribute lookup rules and dunder-governed 
> object-customization setup as Python.  So that's the difference.

I think that's a difference that makes no difference.

What if I told you that it is likely that Python's name/attribute lookup 
rules and dunder-governed object-customization are the key features that 
would make extension methods possible with little or no interpreter 
support?

As I described in a previous post, adding extension methods would be a 
very small change to the existing attribute lookup rules. The tricky 
part is to come up with a fast, efficient registration system for 
determining when to use them.

> The fact that extension methods happen to exist and be useful in those 
> languages is really neither here nor there.

Extension methods don't just "happen to exist", they were designed to 
solve a real problem. That's a problem that can apply

[Python-ideas] Re: Extension methods in Python

2021-06-24 Thread Chris Angelico

On Thu, Jun 24, 2021 at 7:51 PM Steven D'Aprano  wrote:
>
> I'm not sure if you completely understand the use-case I was describing,
> so let me clarify for you with a concrete example.
>
> Ints have a "bit_length" method, starting from Python 2.7. I needed to
> use that method going all the way back to version 2.4. I have an
> implementation that works, so I could backport that method to 2.4
> through 2.6, except that you can't monkey-patch builtins in Python.
>
> So monkey-patching is out.
>
> (And besides, I wouldn't want to monkey-patch it: I only need that
> method in one module. I want to localise the change to only where it is
> needed.)
>
> Subclassing int wouldn't help. I need it to work on actual ints, and any
> third-party subclasses of int, not just my own custom subclass.
>
> (And besides, have you tried to subclass int? It's a real PITA. It's
> easy enough to write a subclass, but every operation on it returns an
> actual int instead of the subclass. So you have to write a ton of
> boilerplate to make int subclasses workable. But I digress.)
>
> So a subclass is not a good solution either.
>
> That leaves only a function.
>
> But that hurts code readability and maintainance. In 2.7 and above,
> bit_length is a method, not a function. All the documentation for
> bit_length assumes it is a method. Every tutorial that uses it has it as
> a method. Other code that uses it treats it as a method.
>
> Except my code, where it is a function.
>
> Using a function is not a *terrible* solution to the problem of
> backporting a new feature to older versions of Python. I've done it
> dozens of times and it's not awful. **But it could be better.**
>
> Why can't the backport be a method, just like in 2.7 and above?
>
> With extension methods, it can be.
>

You've given some great arguments for why (5).bit_length() should be
allowed to be a thing. (By the way - we keep saying "extension
METHODS", but should this be allowed to give non-function attributes
too?) But not once have you said where getattr(), hasattr(), etc come
into this. The biggest pushback against this proposal has been the
assumption that getattr(5, "bit_length")() would have to be the same
as (5).bit_length(). Why is that necessary? I've never seen any
examples of use-cases for that.

Let's tighten this up into a real proposal. (I'm only +0.5 on this,
but am willing to be swayed.)

* Each module has a registration of (type, name, function) triples.
* Each code object is associated with a module.
* Compiled code automatically links the module with the code object.
(If you instantiate a code object manually, it's on you to pick a
module appropriately.)
* Attribute lookups use three values: object, attribute name, and module.
* If the object does not have the attribute, its MRO is scanned
sequentially for a registered method. If one is found, use it.

Not mentioned in this proposal: anything relating to getattr or
hasattr, which will continue to look only at real methods. There may
need to be an enhanced version of PyObject_GetAttr which is able to
look up extension methods, but the current one simply wouldn't.

Also not mentioned: ABC registration. If you register a class as a
subclass of an ABC and then register an extension method on that
class, isinstance() will say that it's an instance of the ABC, but the
extension method won't be there. I'm inclined to say "tough luck,
don't do that", but if there are strong enough use cases, that could
be added.

But otherwise, I would FAR prefer a much simpler proposal, one which
changes only the things that need to be changed.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/WNDEVXU57APUPLYXHKJCRGG4YAM3H75Z/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

Oops, sorry, I neglected to trim my response to João. Please ignore my  
previous response, with the untrimmed quoting at the start, and give any 
replies to this.

On Wed, Jun 23, 2021 at 07:23:19PM +0200, João Santos wrote:

> Of course that means the the standard library might also introduce 
> something new that will be shadowed by one of your custom methods, 

Extension methods have lower priority than actual methods on the class. 
So that won't happen. The actual method on the class will shadow the 
extension method.

I'm not sure if you completely understand the use-case I was describing, 
so let me clarify for you with a concrete example.

Ints have a "bit_length" method, starting from Python 2.7. I needed to 
use that method going all the way back to version 2.4. I have an 
implementation that works, so I could backport that method to 2.4 
through 2.6, except that you can't monkey-patch builtins in Python.

So monkey-patching is out.

(And besides, I wouldn't want to monkey-patch it: I only need that 
method in one module. I want to localise the change to only where it is 
needed.)

Subclassing int wouldn't help. I need it to work on actual ints, and any 
third-party subclasses of int, not just my own custom subclass.

(And besides, have you tried to subclass int? It's a real PITA. It's 
easy enough to write a subclass, but every operation on it returns an 
actual int instead of the subclass. So you have to write a ton of 
boilerplate to make int subclasses workable. But I digress.)

So a subclass is not a good solution either.

That leaves only a function.

But that hurts code readability and maintainance. In 2.7 and above, 
bit_length is a method, not a function. All the documentation for 
bit_length assumes it is a method. Every tutorial that uses it has it as 
a method. Other code that uses it treats it as a method.

Except my code, where it is a function.

Using a function is not a *terrible* solution to the problem of 
backporting a new feature to older versions of Python. I've done it 
dozens of times and it's not awful. **But it could be better.**

Why can't the backport be a method, just like in 2.7 and above?

With extension methods, it can be.

Obviously not for Python 2.x code. But plan for the future: if we have 
extension methods in the language, eventually every version of Python we 
care about will support it. And then writing compatibility layers will 
be much simpler.

> and then you'll wish you had just used functions or a wrapper class.

Believe me, I won't. I've written dozens of compatibility functions over 
the last decade or more, going back to Python 2.3. I've written hybrid 
2/3 code. Extension methods would not always be useful, but for cases 
like int.bit_length, it would be a far superior solution.

> If you can import extension methods wholesale, you might even be 
> monkeypatching something without realising it

Extension methods is not monkey-patching. It is like adding a global 
name to one module. If I write:

def func(arg):
...

in module A.py, that does not introduce func to any other module unless 
those other modules explicitly import it.

Extension methods are exactly analogous: they are only visible in the 
module where you opt-in to use them. They don't monkey-patch the entire 
interpreter-wide environment.

And because extension methods have a lower priority than actual methods, 
you cannot override an existing method on a class. You can only extend 
the class with a new method.

-- 
Steve
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/O43B665WNRF5W3PIUPRBIOWAFJ4H7CS3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

On Wed, Jun 23, 2021 at 07:23:19PM +0200, João Santos wrote:
> 
> 
> On Wed, Jun 23 2021 at 20:48:39 +1000, Steven D'Aprano 
>  wrote:
> >I've just thought of a great use-case for extension methods.
> >
> >Hands up who has to write code that runs under multiple versions of
> >Python? *raises my hand*
> >
> >I'm sure I'm not the only one. You probably have written compatibility
> >functions like this:
> >
> >def bit_length(num):
> >try:
> >return num.bit_length()
> >except AttributeError:
> ># fallback implementation goes here
> >...
> >
> >
> >and then everywhere you want to write `n.bit_length()`, you write
> >`bit_length(n)` instead.
> >
> >Extension methods would let us do this:
> >
> ># compatibility.py
> >@extends(int):
> >def bit_length(self):
> ># fallback implementation goes here
> > ...
> >
> >
> ># mylibrary.py
> >using compatibility
> >num = 42
> >num.bit_length()
> >
> >
> >Now obviously that isn't going to help with versions too old to
> >support extension methods, but eventually extension methods will
> >be available in the oldest version of Python you care about:
> >
> ># supports Python 3.14 and above
> >
> >Once we reach that point, then backporting new methods to classes
> >becomes a simple matter of using an extension method. No mess, no 
> >fuss.
> >
> >As someone who has written a lot of code like that first bit_length
> >compatibility function in my time, I think I've just gone from "Yeah,
> >extension methods seem useful..." to "OMG I WANT THEM TEN YEARS AGO 
> >SO I
> >CAN USE THEM RIGHT NOW!!!".
> >
> >
> >Backporting might not be your killer-app for extension methods, but I
> >really do think they might be mine.
> >
> >
> >--
> >Steve
> >___
> >Python-ideas mailing list -- python-ideas@python.org 
> >
> >To unsubscribe send an email to python-ideas-le...@python.org 
> >
> >
> >Message archived at 
> >
> >Code of Conduct: 
> 

> Of course that means the the standard library might also introduce 
> something new that will be shadowed by one of your custom methods, 

Extension methods have lower priority than actual methods on the class. 
So that won't happen. The actual method on the class will shadow the 
extension method.

I'm not sure if you completely understand the use-case I was describing, 
so let me clarify for you with a concrete example.

Ints have a "bit_length" method, starting from Python 2.7. I needed to 
use that method going all the way back to version 2.4. I have an 
implementation that works, so I could backport that method to 2.4 
through 2.6, except that you can't monkey-patch builtins in Python.

So monkey-patching is out.

(And besides, I wouldn't want to monkey-patch it: I only need that 
method in one module. I want to localise the change to only where it is 
needed.)

Subclassing int wouldn't help. I need it to work on actual ints, and any 
third-party subclasses of int, not just my own custom subclass.

(And besides, have you tried to subclass int? It's a real PITA. It's 
easy enough to write a subclass, but every operation on it returns an 
actual int instead of the subclass. So you have to write a ton of 
boilerplate to make int subclasses workable. But I digress.)

So a subclass is not a good solution either.

That leaves only a function.

But that hurts code readability and maintainance. In 2.7 and above, 
bit_length is a method, not a function. All the documentation for 
bit_length assumes it is a method. Every tutorial that uses it has it as 
a method. Other code that uses it treats it as a method.

Except my code, where it is a function.

Using a function is not a *terrible* solution to the problem of 
backporting a new feature to older versions of Python. I've done it 
dozens of times and it's not awful. **But it could be better.**

Why can't the backport be a method, just like in 2.7 and above?

With extension methods, it can be.

Obviously not for Python 2.x code. But plan for the future: if we have 
extension methods in the language, eventually every version of Python we 
care about will support it. And then writing compatibility layers will 
be much simpler.

> and then you'll wish you had just used functions or a wrapper class.

Believe me, I won't. I've written dozens of compatibility functions over 
the last decade or more, going back to Python 2.3. I've written hybrid 
2/3 code. Extension methods would not always be useful, but for cases 
like int.bit_length, it would be a far superior solution.

> If you can import extension methods wholesale, you might even be 
> monkeypatching something without realising it

[Python-ideas] Re: Extension methods in Python

2021-06-23 Thread Brendan Barnwell

On 2021-06-23 03:02, Steven D'Aprano wrote:

Attribute lookups are just another form of name lookup. Name lookups
depend on the current execution scope, not the caller's scope. With
extension methods, so do attribute lookups.

But that's the thing, they aren't. You gave a bunch of examples of
lexical scope semantics with imports and function locals vs globals.
But attribute lookups do not work that way. Attribute lookups are
defined to work via a FUNCTION CALL to the __getattribute__ (and thence
often the __getattr__) of the OBJECT whose attribute is being looked up.
They do not in any way depend on the name via which that object is
accessed.

Now of course you can say that you want to make a new rule that throws
the old rules out the window. We can do that for anything. We can
define a new rule that says now when you do attribute lookups it will
call a global function called attribute_lookups_on_tuesdays if it's a
Tuesday in your timezone. But what I'm saying is that the way attribute
lookups currently work is not the same as the way bare-name lookups
work, because attribute lookups are localized to the object (not the
name!) and bare-name lookups are not. I consider this difference
fundamental to Python. It's why locals() isn't really how local name
lookups work (which came up elsewhere in this thread). It's why you
can't magically hook into "x = my_obj" and create some magical behavior
that depends on my_obj. Attribute lookups are under the control of the
object; they come after the scope-based name resolution is all over with
and they don't use the scope-based rules.

As for other languages, you keep referencing them as if the fact that
something known as "extension methods" exists in those other languages
makes it self-evident that it would be useful in Python. Python isn't
those other languages. I'm not familiar with all of the other languages
you mentioned, but I'll bet that at least some of them do not have the
same name/attribute lookup rules and dunder-governed
object-customization setup as Python. So that's the difference. The
fact that extension methods happen to exist and be useful in those
languages is really neither here nor there. The attribute lookup
procedure you are proposing is deeply inconsistent with the way Python
currently does attribute lookup and currently does other things (like
operator overloading), and doesn't fit into Python's overall mechanism
of object-based hooks. A spatula attachment may be useful on a kitchen
mixer; that doesn't mean it's a good idea to add one to your car's
dashboard.

Apart from that, I will say that I also don't generally assume that
because other languages have a feature it's good or worth considering.
Some languages are better designed than others. I think Python is a
very well designed language. Certainly we can learn from other
languages, but even apart from the issues of "fit" that I describe
above, the mere fact that some feature is available or even considered
useful in other languages doesn't by itself even convince me it's a good
idea at all. It could just be a mistake. We need to specifically show
that this will make writing and/or reading code easier and better in
Python, and I think this proposal would do the opposite, making code
harder to read.

--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."

--author unknown
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/FTHXJDHPE5UFVQE5WSBJN6ZURZKWUFFY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

2021-06-23 Thread João Santos




On Wed, Jun 23 2021 at 20:48:39 +1000, Steven D'Aprano 
 wrote:

I've just thought of a great use-case for extension methods.

Hands up who has to write code that runs under multiple versions of
Python? *raises my hand*

I'm sure I'm not the only one. You probably have written compatibility
functions like this:

def bit_length(num):
try:
return num.bit_length()
except AttributeError:
# fallback implementation goes here
...


and then everywhere you want to write `n.bit_length()`, you write
`bit_length(n)` instead.

Extension methods would let us do this:

# compatibility.py
@extends(int):
def bit_length(self):
# fallback implementation goes here
...


# mylibrary.py
using compatibility
num = 42
num.bit_length()


Now obviously that isn't going to help with versions too old to
support extension methods, but eventually extension methods will
be available in the oldest version of Python you care about:

# supports Python 3.14 and above

Once we reach that point, then backporting new methods to classes
becomes a simple matter of using an extension method. No mess, no 
fuss.


As someone who has written a lot of code like that first bit_length
compatibility function in my time, I think I've just gone from "Yeah,
extension methods seem useful..." to "OMG I WANT THEM TEN YEARS AGO 
SO I

CAN USE THEM RIGHT NOW!!!".


Backporting might not be your killer-app for extension methods, but I
really do think they might be mine.


--
Steve
___
Python-ideas mailing list -- python-ideas@python.org 

To unsubscribe send an email to python-ideas-le...@python.org 



Message archived at 


Code of Conduct: 


Of course that means the the standard library might also introduce 
something new that will be shadowed by one of your custom methods, and 
then you'll wish you had just used functions or a wrapper class. If you 
can import extension methods wholesale, you might even be 
monkeypatching something without realising it, in which case you'll be 
lucky if things break in an obvious way.
Honestly, all the use cases in this thread seem to be much better 
served by using plain old functions.




___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/XRNH7HUFATC4QJLMQD5WYEUIC36PC3GA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

2021-06-23 Thread Chris Angelico

On Wed, Jun 23, 2021 at 11:25 PM Steven D'Aprano  wrote:
>
> On Wed, Jun 23, 2021 at 03:47:05PM +1000, Chris Angelico wrote:
>
> > Okay. Lemme give it to you *even more clearly* since the previous
> > example didn't satisfy.
> >
> > # file1.py
> >
> > @extend(list)
> > def in_order(self):
> > return sorted(self)
> >
> > def frob(stuff):
> > return stuff.in_order()
> >
> > # file2.py
> >
> > from file1 import frob
> > thing = [1, 5, 2]
> > frob(thing) # == [1, 2, 5]
> > def otherfrob(stuff):
> > return stuff.in_order()
> > otherfrob(thing) # AttributeError
> >
> >
> >
> > Am I correct so far? The function imported from file1 has the
> > extension method, the code in file2 does not. That's the entire point
> > here, right?
>
> Correct so far.
>
>
> > Okay. Now, what if getattr is brought into the mix?
>
> To a first approximation (ignoring shadowing) every dot lookup can be
> replaced with getattr and vice versa:
>
> obj.name <--> getattr(obj, 'name')
>
> A simple source code transformation could handle that, and the behaviour
> of the code should be the same. Extension methods shouldn't change that.

Alright. In that case, getattr() has stopped being a function, and is
now a magical construct of the compiler. What happens if I do this?

if random.randrange(2):
def getattr(obj, attr): return lambda: "Hello, world"

def foo(thing):
return getattr(thing, "in_order")()

Does it respect extension methods or not? If getattr is a perfectly
ordinary function, as it now is, then it should be perfectly
acceptable to shadow it. It should also be perfectly acceptable to use
any other way of accessing attributes - for instance, the
PyObject_GetAttr() function in C.

Why should getattr() become magical?

> > import random
> >
> > @extend(list)
> > def unordered(self):
> > return random.shuffle(self[:])
>
> I think that's going to always return None :-)

Oops, my bad :) Not that it changes anything, given that we care more
about whether they trigger AttributeError than what they actually do.

(Chomp the rest of the discussion, since that was all based on the
assumption that getattr was a normal function.)

> > What exactly are the semantics of getattr?
>
> Oh gods, I don't know the exact semantics of attribute look ups now!
> Something like this, I think:
>
> obj.attr (same as getattr(obj, 'attr'):

That is exactly what's weird about it. Instead of looking up the name
getattr and then calling a perfectly ordinary function, now it has to
be a magical construct of the compiler, handled right there. It is, in
fact, impossible to craft equivalent semantics in a third-party
function.

Currently, getattr() can be defined in C on top of the C API function
PyObject_GetAttr, which looks solely at the object and not the
execution context. By your proposal, getattr() can only be compiler
magic.

> > Please explain exactly what the semantics of getattr are, and exactly
> > which modules it is supposed to be able to see. Remember, it is not a
> > compiler construct or an operator. It is a function, and it lives in
> > its own module (the builtins).
>
> You seem to think that getattr being a function makes a difference. Why?
>
> Aside from the possibility that it might be shadowed or deleted from
> builtins, can you give me any examples where `obj.attr` and
> `getattr(obj. 'attr')` behave differently? Even *one* example?

That is *precisely* the possibility. That is exactly why it is magical
by your definition, and nonmagical by the current definition. At the
moment, getattr() is just a function, hasattr() is just a function. I
can do things like this:

def ga(obj, attr): return getattr(obj, attr)

Or this:

ga = getattr

Or this:

PyObject *my_getattr(PyObject *obj, PyObject *attr) {return
PyObject_GetAttr(obj, attr);}

But if it's possible to do a source code transformation from
getattr(obj, "attr") to obj.attr, then it is no longer possible to do
*ANY* of this. You can't have an alias for getattr, you can't have a
wrapper around it, you can't write your own version of it.

In some languages, this is acceptable and unsurprising, because the
getattr-like feature is actually an operator. (For instance,
JavaScript fundamentally defines obj.attr as being equivalent to
obj["attr"], so if you want dynamic lookups, you just use square
brackets.) In Python, that is simply not the case. Are you proposing
to break backward compatibility and all consistency just for the sake
of this?

> > > Not a rhetorical question: is that how it works in something like Swift,
> > > or Kotlin?
> >
> > I have no idea. I'm just asking how you intend it to work in Python.
> > If you want to cite other languages, go ahead, but I'm not assuming
> > that they already have the solution, because they are different
> > languages. Also not a rhetorical question: Is their getattr equivalent
> > actually an operator or compiler construct, rather than being a
> > regular function? Because if it is, then the entire problem doesn't
> > exist.
>

[Python-ideas] Re: Extension methods in Python

2021-06-23 Thread Soni L.

On 2021-06-23 10:21 a.m., Steven D'Aprano wrote:
> > What about other functions implemented in C? If I write a C module
> > that calls PyObject_GetAttr, does it behave as if dot notation were
> > used in the module that called me, or does it use my module's
> > extension methods?
>
> That depends. If you write a C module that calls PyObject_GetAttr right 
> now, is that *exactly* the same as dot notation in pure-Python code?
>
> The documentation is terse:
>
> https://docs.python.org/3.8/c-api/object.html#c.PyObject_GetAttr
>
> but if it is correct that it is precisely equivalent to dot syntax, then 
> the same rules will apply. Has the current module opted in? If so, then 
> does the class have an extension method of the requested name?
>
> Same applies to code objects evaluated without a function, or whatever 
> other exotic corner cases you think of. Whatever you think of, the 
> answer will always be the same:
>
> - if the execution context is a module that has opted to use 
>   extension methods, then attribute access will see extension methods;
>
> - if not, then it won't.
>
> If you think of a scenario where you are executing code where there is 
> no module scope at all, and all global lookups fail, then "no module" 
> cannot opt in to use extension methods and so the code won't see them.
>
> If you can think of a scenario where you are executing code where there 
> are multiple module scopes that fight for supremacy using their two 
> weapons of fear, surprise and a fanatical devotion to the Pope, then the 
> winner will determine the result.
>
> *wink*
>
>
>
>

But if getattr is part of the builtins module, written in C, and the
builtins module is calling PyObject_GetAttr, and PyObject_GetAttr is
exactly the same as the dot notation...

Then getattr is exactly the same as the dot notation, **in the builtins
module**! The builtins module doesn't use any extension methods, as it
is written in C. As such getattr(foo, "bar") MUST NOT produce the same
result as foo.bar if extension methods are at play!

(You're still missing the point of extension methods. Do check out our
other reply.)
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/XARO3NEUWZ7OVS47Y36KPPSDLKQWAUQZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

2021-06-23 Thread Steven D'Aprano

On Wed, Jun 23, 2021 at 03:47:05PM +1000, Chris Angelico wrote:

> Okay. Lemme give it to you *even more clearly* since the previous
> example didn't satisfy.
> 
> # file1.py
> 
> @extend(list)
> def in_order(self):
> return sorted(self)
> 
> def frob(stuff):
> return stuff.in_order()
> 
> # file2.py
> 
> from file1 import frob
> thing = [1, 5, 2]
> frob(thing) # == [1, 2, 5]
> def otherfrob(stuff):
> return stuff.in_order()
> otherfrob(thing) # AttributeError
> 
> 
> 
> Am I correct so far? The function imported from file1 has the
> extension method, the code in file2 does not. That's the entire point
> here, right?

Correct so far.

> Okay. Now, what if getattr is brought into the mix?

To a first approximation (ignoring shadowing) every dot lookup can be 
replaced with getattr and vice versa:

obj.name <--> getattr(obj, 'name')

A simple source code transformation could handle that, and the behaviour 
of the code should be the same. Extension methods shouldn't change that.

> # file3.py
> @extend(list)
> def in_order(self):
> return sorted(self)
> 
> def fetch1(stuff, attr):
> if attr == "in_order": return stuff.in_order
> if attr == "unordered": return stuff.unordered
> return getattr(stuff, attr)
> 
> def fetch2(stuff, attr):
> return getattr(stuff, attr)

In file3's scope, there is no list.unordered method, so any call like

some_list.unordered
getattr(some_list, 'unordered')

will fail, regardless of which list some_list is, or where it was 
created. That implies that:

fetch1(some_list, 'unordered')
fetch2(some_list, 'unordered')

will also fail. It doesn't matter who is calling the functions, or what 
module they are called from. What matters is the context where the 
attribute lookup occurs, which in fetch1 and fetch2 is the file3 scope.

> # file4.py
> from file3 import fetch1, fetch2

Doesn't matter that fetch1 and fetch2 are imported into file4. They are 
still executed in the global scope of file3. If they called `globals()`, 
they would see file3's globals, not file4's. Same thing for extension 
methods.

> import random
>
> @extend(list)
> def unordered(self):
> return random.shuffle(self[:])

I think that's going to always return None :-)

> def fetch3(stuff, attr):
> if attr == "in_order": return stuff.in_order
> if attr == "unordered": return stuff.unordered
> return getattr(stuff, attr)
> 
> def fetch4(stuff, attr):
> return getattr(stuff, attr)

In the scope of file4, there is no list method "in_order", but there is 
a list method "unordered". So

some_list.in_order
getattr(some_list, 'in_order')

will fail. That implies that:

fetch3(some_list, 'unordered')
fetch4(some_list, 'unordered')

will also fail. It doesn't matter who is calling the functions, or what 
module they are called from. What matters is the context where the 
attribute lookup occurs, which in fetch3 and fetch4 is the file4 scope.

(By the way, I think that your example here is about ten times more 
obfuscated than it need be, because of the use of generic, uninformative 
names with numbers.)

> thing = [1, 5, 2]
> fetch1(thing, "in_order")()
> fetch2(thing, "in_order")()
> fetch3(thing, "in_order")()
> fetch4(thing, "in_order")()
> fetch1(thing, "unordered")()
> fetch2(thing, "unordered")()
> fetch3(thing, "unordered")()
> fetch4(thing, "unordered")()
>
> Okay. *NOW* which ones raise AttributeError, and which ones give the
> extension method?

Look at the execution context.

fetch1(thing, "in_order") and fetch2(thing, "in_order") execute in the 
scope of file3, where lists have an in_order extension method.

It doesn't matter that they are called from file4: the body of the 
fetchN functions, where the attribute access takes place, executes where 
the global scope is file3 and hence the extension method "in_order" is 
found and returned.

For the same reason, both fetch1(thing, "unordered") and fetch2(thing, 
"unordered") will fail.

It doesn't matter that they are called from file4: their execution 
context is their global scope, file3, and just as they see file3's 
globals, not the callers, they will see file3's extension methods.

(I say "the module's extension methods", not necessarily to imply that 
the extension methods are somehow attached to the module, but only that 
there is some sort of registry that says, in effect, "if your execution 
context is module X, then these extension methods are in use".)

Similarly, the body of fetch3 and fetch4 execute in the execution 
context of file4, where list has been extended with an unordered method. 
So fetch3(thing, "unordered") and fetch4(thing, "unordered") both return 
that unordered method.

For the same reason (the execution context), fetch3(thing, "in_order") 
and fetch4(thing, "in_order") both fail.

> What exactly are the semantics of getattr?

Oh gods, I don't know the exact semantics of attribute look ups now! 
Something like this, I think:

obj.attr (same as

[Python-ideas] Re: Extension methods in Python

2021-06-23 Thread Soni L.



On 2021-06-23 5:21 a.m., Steven D'Aprano wrote:
> On Tue, Jun 22, 2021 at 08:44:56AM -0300, Soni L. wrote:
>
> > Oh this is a long one.
> > 
> > Hypothetically, let's say you have a proxy object:
> > 
> > class Foo:
> >   def __getattribute__(self, thing):
> >     return getattr(super().__getattribute__(self, "proxied"), thing)
> > 
> > Should this really include extension methods into it by default?
>
> By default? Absolutely not. Extension methods are opt-in.
>
>
> > This is clearly wrong.
>
> What is clearly wrong? Your question? A "yes" answer? A "no" answer? 
> Your proxy object?

We're saying including local extension methods into the proxy object's
attribute lookup is wrong.

> Soni, and Chris, you seem to be responding as if extension methods are 
> clearly, obviously and self-evidently a stupid idea. Let me remind you 
> that at least ten languages (C#, Java, Typescript, Oxygene, Ruby, 
> Smalltalk, Kotlin, Dart, VB.NET and Swift) support it. Whatever the pros 
> and cons of the technique, it is not self-evidently wrong or stupid.

That's funny because we (Soni) have been arguing about and pushing for a
specific implementation of them. We're not opposed to them, quite the
opposite we even have an implementation we'd like to see, altho we don't
really see much of a use for them ourselves.

>
> > The local override for the LOAD_ATTR opcode
> > should NOT apply to proxy methods except where explicitly requested.
>
> Why not?
>
> Let me ask you this:
>
> - should proxy objects really include `__slots__` by default?
>
> - should they really include dynamic attributes generated by 
>   `__getattr__`?
>
> - should they include attributes in the inheritence hierarchy?
>
> - why should extension methods be any different?
>
>
> Let's step back from extension methods and consider a similar technique, 
> the dreaded monkey-patch. If I extend a class by monkey-patching it with 
> a new method:
>
> import library
> library.Klass.method = patch_method
>
> would you expect that (by default) the patched method are invisible to 
> proxies of Klass? Would you expect there to be a way to "opt-out" of 
> proxying that method?
>
> I hope that your answers are "No, and no", because if either answer is 
> "yes", you will be very disappointed in Python.
>
> Why should extension methods be different from any other method? Let's 
> go through the list of methods which are all treated the same:

Why shouldn't extension methods be different from monkey-patching? If
they were to be the same, why call them something different?

>
> - methods defined on the class;
> - methods defined on a superclass or mixin;
> - methods added onto the instance;
> - methods created dynamically by `__getattr__`.
>
> (Did I miss any?)
>
> And the list of those which are handled differently, with ways to 
> opt-out of seeing them:
>
> - ... um... er...
>
> Have I missed any?
>
>
> > The point is that the caller using your proxy object should opt-in to
> > the extension methods, rather than break with no way to opt-out of them.
>
> You opt-out by not opting in.
>
>
> > Your extension methods shouldn't propagate to proxy objects.
>
> Fundamentally, your proxy object is just doing attribute lookups on 
> another object. If you have a proxy to an instance `obj`, there should 
> be no difference in behaviour between extension methods and regular 
> methods. If `obj.method` succeeds, so should `proxy.method`, because 
> that's what proxies do.
>
> The origin of obj.method should not make any difference. I'm sorry to 
> have to keep harping on this, but it doesn't matter to the proxy whether 
> the method exists in the instance `__dict__`, or the class `__dict__`, 
> or `__slots__`, or a superclass, or is dynamically generated by 
> `__getattr__`. A method is a method.
>
> Extension methods are methods.

Extension methods are functions and are scoped like other functions.

>
>
> > To go even further, should all your class definitions that happen to
> > extend a class with in-scope extension methods automatically gain those
> > extension methods? Because with actual extension methods, that doesn't
> > happen.
>
> We might want to follow the state of the art here, assuming there was 
> consensus in other languages about inheriting extension methods. But I 
> would expect this behaviour:
>
>
> # --- extensions.py library ---
>
> @extends(list)
> def flatten(self):
> ...
>
>
> # --- module A.py ---
>
> uses extensions  # opt-in to use the extension method
>
> class MyListA(list):
> pass
>
> MyListA.flatten  # inherits from list
>
>
> # --- module B.py ---
>
> class MyListB(list):
> pass
>
> MyListB.flatten  # raises AttributeError
>
>
> However, there may be factors I haven't considered.
>
>
> > You can have class MyList(list): pass and other callers would
> > not get MyList.flatten even with you being able to use MyList.flatten
> > locally.
> > 
> > Extension methods are more like

[Python-ideas] Re: Extension methods in Python

2021-06-23 Thread Steven D'Aprano

On Tue, Jun 22, 2021 at 11:43:09AM -0700, Brendan Barnwell wrote:

> ### file1.py
> @extend(list)
> def len2(self):
>   return len(self)**2
> 
> ### file2.py
> # or whatever I do to say "I want to use extensions to list defined in 
> file1"
> from file1 extend list
> 
> def coolness(some_list):
>   return some_list.len2() + 1
> 
> my_list = [1, 2, 3]
> print("My list len2:", my_list.len2())
> print("My list coolness:", coolness(my_list))

Right, as far as code executing within the "file2" scope is concerned, 
lists have a "len2" method.

> ### file3.py
> import file2
> 
> other_list = [1, 2, 3, 4]
> print("Other list len2:", other_list.len2())
> print("other list coolness:", file2.coolness(other_list))

The call to other_list.len2 would fail with AttributeError, because the 
current execution scope when the attribute is looked up is file3, not 
file2.

The call to file2.coolness() would succeed, because the function 
coolness is executed in the scope of file2, which is using the 
extension method.

This is similar to the way name lookups work:

# A.py
x = 1
def func():
print(x)  # This always looks up x in A's scope.

Obviously calling func() in its own module will succeed. But if you call 
func() from another module, it doesn't look up "x" in the caller's 
module, it still refers back to the module it was defined. Namely A.

# B.py
x = 
from A import func
func()  # prints 1, not 

This is standard Python semantics, and you probably don't even think 
about it. Extension methods would work similarly: whether the attribute 
name is visible or not depends on which scope the attribute lookup 
occurs in, not who is calling it.

So if you can understand Python scoping, extension methods will be quite 
similar.

Remember:

- global `x` in A.py and global `x` in B.py are different names in 
  different scopes;

- functions remember their original global scope and lookup names
  in that, not the caller's global scope.

Attribute lookups involving extension methods would be similar:

- whether the extension method is seen or not depends on the
  which scope the look up occurs, not where the caller is.

That lets us see what will happen in file3.py here:

> print("My list len2 from outside:", file2.my_list.len2())

It doesn't matter where the list comes from. What matters is where 
the attribute access occurs. In the first line there, the attribute 
access occurs in the scope of file3, so it fails.

It doesn't matter whether you write `other_list.len2` or 
`file2.my_list.len2`, both are equivalent to:

- get a list instance (file2.my_list, but it could be anything)
- doesn't matter where the instance comes from
- look up the method "len2" on that instance **in the file3 scope**

Hence it fails.

The line that follows:

> print("My list coolness from outside:", file2.coolness(file2.my_list))

does this:

- get a list instance (file2.my_list)
- pass it to file2.coolness
- which looks up the method "len2" **in the file2 scope**

and hence it succeeds.

>   The list object my_list in file2 is the same object accessed as 
> file2.my_list in file3.  Likewise coolness and file2.coolness.  It is 
> going to be super confusing if calling the same function object with the 
> same list object argument gives different results depending on which 
> file you're in.

"It is going to be super confusing if calling the same function with the 
same *name* gives different results depending on which file you're in."

-- me, 25 years ago or so, when I first started learning Python

And probably you to.

Attribute lookups are just another form of name lookup. Name lookups 
depend on the current execution scope, not the caller's scope. With 
extension methods, so do attribute lookups.

If you can cope with name lookups, you can cope with extension methods.

[...]
> But what if the extension is an override of an existing method?  Is that 
> not allowed?)

In C#, you can define extension methods with the same name as an 
existing method, but they will never be seen. The extension methods are 
only used if the normal method lookup fails.

>   In addition, if there is a difference between my_list and 
>   other_list,

No, absolutely not. It isn't two different sorts of list.

Here is some pseudo-code that might help clarify the behaviour.

_getattr = getattr  # original that we know and love

def getattr(obj, name):  # called on obj.name
# New, improved version
try:
return _getattr(obj, name)
except AttributeError:
if current execution scope is using extensions for type(obj):
return extension_method(type(self), name)

Think of that as a hand-wavy sketch of behaviour, not an exact 
specification.

> Are both objects lists?  If they are, then how can they have different 
> methods?

Here's another sketch of behaviour.

class object:
def __getattr__(self, name):
# only called

[Python-ideas] Re: Extension methods in Python

2021-06-23 Thread Chris Angelico

On Wed, Jun 23, 2021 at 6:49 PM Brendan Barnwell  wrote:
> There aren't many things in Python that work this way.  Future imports
> are the main one, but those are rare (and rightly so).  The import
> machinery itself provides some possibility for this (as used by stuff
> like macropy) but is mostly not used for such things (and again rightly so).

Future imports are generally about syntax. They only very seldom
affect execution, and when they do, there has to be a mechanic like a
flag on the code object (as generator_stop does). In order to maintain
compatibility between modules with and without the directive, the
execution has to be independent of that. (For example, Py2's "from
__future__ import division" changes the bytecode created from the
division operator, and barry_as_FLUFL generates the same code for "<>"
that normal mode generates for "!=".)

There's no fundamental problem with having things defined per-module.
You can override the print function for the current module, and other
modules aren't affected. I think that per-module operator overloading
would be extremely confusing, but mainly because type-free operator
overloading is an inherently confusing thing to do. Python allows an
object to override operators, and C++ allows you to write a function
called "operator+" that takes two arguments, but the types of those
arguments is what makes everything make sense. That usually removes
the problem of recursion because the implementation of
operator+(my_obj, my_obj) is usually going to require addition of
simpler types like integers.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/VYKWJHEO5MCKUU6ISORO6IHXFL4TZ6ZZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

2021-06-23 Thread Chris Angelico

On Wed, Jun 23, 2021 at 6:25 PM Steven D'Aprano  wrote:
> Soni, and Chris, you seem to be responding as if extension methods are
> clearly, obviously and self-evidently a stupid idea. Let me remind you
> that at least ten languages (C#, Java, Typescript, Oxygene, Ruby,
> Smalltalk, Kotlin, Dart, VB.NET and Swift) support it. Whatever the pros
> and cons of the technique, it is not self-evidently wrong or stupid.

It's not self-evidently wrong or stupid. But the semantics, as given,
don't make sense. Before this could ever become part of the language,
it will need some VERY well-defined semantics. (Preferably, semantics
that don't depend on definitions involving CPython bytecode, although
I'm fine with it being described like that for the time being. But
ultimately, other Pythons will have to be able to match the
semantics.) You have been saying certain things as if they are
self-evidently right, without any justification or explanation.

> Let me ask you this:
>
> - should proxy objects really include `__slots__` by default?
>
> - should they really include dynamic attributes generated by
>   `__getattr__`?
>
> - should they include attributes in the inheritence hierarchy?
>
> - why should extension methods be any different?

They are different because they are *context-sensitive*. Every other
example you have given is attached to the object itself. The object
defines whether it has __slots__, __dict__, a __getattr__ method, a
__getattribute__ method, and superclasses. The object defines, in
those ways, which attributes can be looked up, and it doesn't matter
how you ask the question, you'll get the same answer. Calling
getattr(obj, "thing") is the same as obj.thing is the same as
PyObject_GetAttr(ptr_to_obj, ptr_to_string_thing) is the same as any
other way you would look it up.

Extension methods change that. Now it depends on which module you are
in. That means you're either going to have to forfeit these
consistencies, or they are going to need to figure out WHICH module
you are working with.

The simplest definition is this: Extension methods apply *only* to dot
notation here in the current module. Every piece of code compiled in
this module will look up dotted attributes using extensions active in
this module. (In CPython terms, that affects the behaviour of
LOAD_ATTR only, and would mean that the code object retains a
reference to that module.) That's pretty reasonable. But to accept
this simple definition, you *must* forfeit the parallel with
getattr(), since getattr() is defined in the builtins, NOT in your
module. Yet you assert that, self-evidently, getattr(obj, "thing")
MUST be the same as obj.thing, no matter what.

> Let's step back from extension methods and consider a similar technique,
> the dreaded monkey-patch. If I extend a class by monkey-patching it with
> a new method:
>
> import library
> library.Klass.method = patch_method
>
> would you expect that (by default) the patched method are invisible to
> proxies of Klass? Would you expect there to be a way to "opt-out" of
> proxying that method?
>
> I hope that your answers are "No, and no", because if either answer is
> "yes", you will be very disappointed in Python.

For my part, I absolutely agree with you - the proxy should see it.
But that's because the *object*, not the module, is making that
decision.

> Why should extension methods be different from any other method? Let's
> go through the list of methods which are all treated the same:
>
> - methods defined on the class;
> - methods defined on a superclass or mixin;
> - methods added onto the instance;
> - methods created dynamically by `__getattr__`.
>
> (Did I miss any?)

Yes, all things that are defined by the object, regardless of its context.

> And the list of those which are handled differently, with ways to
> opt-out of seeing them:
>
> - ... um... er...
>
> Have I missed any?

Nope. Python currently has a grand total of zero ways to have
attributes whose existence depends on the module of the caller.
(Barring shenanigans with __getattr__ and sys._getframe. Or ctypes. I
think we can all agree that that sort of thing doesn't count.)

> The origin of obj.method should not make any difference. I'm sorry to
> have to keep harping on this, but it doesn't matter to the proxy whether
> the method exists in the instance `__dict__`, or the class `__dict__`,
> or `__slots__`, or a superclass, or is dynamically generated by
> `__getattr__`. A method is a method.
>
> Extension methods are methods.

By definition, extension methods are methods in one module, and not
methods in another module.

> > Also note that they use instance method syntax, but no other. That is
> > they apply to LOAD_ATTR opcodes but should not apply to getattr!
> > (Indeed, reflection in C#/Kotlin doesn't see the extension methods!)
>
> Okay, that's a good data point. The question is, why doesn't reflection
> see the extension methods? That will help us decide whether that's a
> limitation of

[Python-ideas] Re: Extension methods in Python

2021-06-23 Thread Brendan Barnwell

On 2021-06-22 15:35, Soni L. wrote:

Imagine if Python didn't have an + operator, but instead an + *infix
function*.

Thus, every module would automatically include the global

def infix +(left, right):
...

And indeed, you could say we already have this. Except currently you
can't define your own local infix +. But what if you*could*?

Then that would be bad.

All the examples you give seem bad to me. They just make the code more
confusing.

Python combines various paradigms, but I think one way in which it very
smoothly leverage object orientation is by making objects the locus of
so much behavior. Operator overloads are defined at the object level,
as are "quasi-operator" overloads for things like attribute lookup,
iteration, context managers, etc. This means that the semantics of an
expression are, by and large, determined by the types of the objects in
that expression.

What you're describing is basically moving a lot of that out to the
module level. Now instead of operator overloads being governed by
objects, they'd be governed by the module in which the code appears.
The semantics of an expression would be determined not (only) by the
types of the objects involved, but also by the module in which the
expression textually occurs.

There aren't many things in Python that work this way. Future imports
are the main one, but those are rare (and rightly so). The import
machinery itself provides some possibility for this (as used by stuff
like macropy) but is mostly not used for such things (and again rightly so).

Beyond that, I just think this kind of thing is a bad idea. Objects
naturally cross module boundaries, in that an object may be created in
one module and used in many other modules. It is good for an object's
behavior to be consistent across modules, so that someone using an
object (or readng code that uses an object) can look at the
documentation for that object's type and understand how it will work in
any context. It is good for code to be understandable in a "bottom up"
way in which you understand the parts (the objects, expressions,
syntactic structures, etc.) and can combine your understanding of those
parts to understand the whole. It is bad for an object to shapeshift
and do different things in different contexts. It is bad for code to
heavily depend on "top down" information that requires you to know
"where you are" (in one module or another) to understand how things
work. That increases cognitive burden and makes code more difficult to
understand.

Personally I'm opposed to anything that moves in that direction.

--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."

--author unknown
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/4OUETMGSZEYXLQ23CPDUIKEJCUEEZKUO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

2021-06-23 Thread Steven D'Aprano

On Tue, Jun 22, 2021 at 08:44:56AM -0300, Soni L. wrote:

> Oh this is a long one.
> 
> Hypothetically, let's say you have a proxy object:
> 
> class Foo:
>   def __getattribute__(self, thing):
>     return getattr(super().__getattribute__(self, "proxied"), thing)
> 
> Should this really include extension methods into it by default?

By default? Absolutely not. Extension methods are opt-in.

> This is clearly wrong.

What is clearly wrong? Your question? A "yes" answer? A "no" answer? 
Your proxy object?

Soni, and Chris, you seem to be responding as if extension methods are 
clearly, obviously and self-evidently a stupid idea. Let me remind you 
that at least ten languages (C#, Java, Typescript, Oxygene, Ruby, 
Smalltalk, Kotlin, Dart, VB.NET and Swift) support it. Whatever the pros 
and cons of the technique, it is not self-evidently wrong or stupid.

> The local override for the LOAD_ATTR opcode
> should NOT apply to proxy methods except where explicitly requested.

Why not?

Let me ask you this:

- should proxy objects really include `__slots__` by default?

- should they really include dynamic attributes generated by 
  `__getattr__`?

- should they include attributes in the inheritence hierarchy?

- why should extension methods be any different?

Let's step back from extension methods and consider a similar technique, 
the dreaded monkey-patch. If I extend a class by monkey-patching it with 
a new method:

import library
library.Klass.method = patch_method

would you expect that (by default) the patched method are invisible to 
proxies of Klass? Would you expect there to be a way to "opt-out" of 
proxying that method?

I hope that your answers are "No, and no", because if either answer is 
"yes", you will be very disappointed in Python.

Why should extension methods be different from any other method? Let's 
go through the list of methods which are all treated the same:

- methods defined on the class;
- methods defined on a superclass or mixin;
- methods added onto the instance;
- methods created dynamically by `__getattr__`.

(Did I miss any?)

And the list of those which are handled differently, with ways to 
opt-out of seeing them:

- ... um... er...

Have I missed any?

> The point is that the caller using your proxy object should opt-in to
> the extension methods, rather than break with no way to opt-out of them.

You opt-out by not opting in.

> Your extension methods shouldn't propagate to proxy objects.

Fundamentally, your proxy object is just doing attribute lookups on 
another object. If you have a proxy to an instance `obj`, there should 
be no difference in behaviour between extension methods and regular 
methods. If `obj.method` succeeds, so should `proxy.method`, because 
that's what proxies do.

The origin of obj.method should not make any difference. I'm sorry to 
have to keep harping on this, but it doesn't matter to the proxy whether 
the method exists in the instance `__dict__`, or the class `__dict__`, 
or `__slots__`, or a superclass, or is dynamically generated by 
`__getattr__`. A method is a method.

Extension methods are methods.

> To go even further, should all your class definitions that happen to
> extend a class with in-scope extension methods automatically gain those
> extension methods? Because with actual extension methods, that doesn't
> happen.

We might want to follow the state of the art here, assuming there was 
consensus in other languages about inheriting extension methods. But I 
would expect this behaviour:

# --- extensions.py library ---

@extends(list)
def flatten(self):
...

# --- module A.py ---

uses extensions  # opt-in to use the extension method

class MyListA(list):
pass

MyListA.flatten  # inherits from list

# --- module B.py ---

class MyListB(list):
pass

MyListB.flatten  # raises AttributeError

However, there may be factors I haven't considered.

> You can have class MyList(list): pass and other callers would
> not get MyList.flatten even with you being able to use MyList.flatten
> locally.
> 
> Extension methods are more like Rust traits than inheritance-based OOP.

I understand that Rust doesn't support inheritance at all, and that Rust 
traits are more like what everyone else calls "interfaces".

> Also note that they use instance method syntax, but no other. That is
> they apply to LOAD_ATTR opcodes but should not apply to getattr!
> (Indeed, reflection in C#/Kotlin doesn't see the extension methods!)

Okay, that's a good data point. The question is, why doesn't reflection 
see the extension methods? That will help us decide whether that's a 
limitation of reflection in those languages, or a deliberate design 
feature we should follow.

A brief search suggests that people using C# do want to access extension 
methods via reflection, and that there are ways to do so:

https://duckduckgo.com/?q=c%23+invoke+extension+method+via+reflection

-- 
Steve

[Python-ideas] Re: Extension methods in Python

On Wed, Jun 23, 2021 at 11:40 AM Steven D'Aprano  wrote:
>
> On Tue, Jun 22, 2021 at 10:25:33PM +1000, Chris Angelico wrote:
>
> > > If its a problem for getattr, it is a problem for dot syntax, because
> > > they are essentially the same thing.
> >
> > Ahh but that is precisely the problem.
>
> Is it? Don't be shy. Tell us what the problem is and why its a problem.

That's exactly what the rest of the post is about.

> > > > How is getattr defined?
> > >
> > > The same as it is defined now, except with some minor tweaks to support
> > > extension methods.
> >
> > Do those tweaks include reaching back into the module that called it?
> > How magical will it be?
>
> I thought you agreed that we didn't need to discuss implementation until
> we had decided on the desired semantics?
>
> Let's just say it will be a well-defined, totally non-magical
> implementation (like everything else in Python) that manages to be
> equally efficient as regular attribute access. A high bar to set.
> Implementation issues may require us to dial that back a bit, or might
> even rule out the concept altogether, but let's start off by assuming
> the best and decide on the semantics first.

Semantics are *exactly* what I'm talking about.

> > This means that the getattr() function, being a perfectly
> > straight-forward function, is not going to see any extension methods.
>
> Does getattr see slots (both C-level and Python)? Yes. Does it see
> attributes in instance and class dicts? Yes. Does it see dynamic
> attributes that use `__getattr__`? Yes. Does it understand the
> descriptor protocol? Yes.
>
> It does everything else dot notation does. Why wouldn't it see extension
> methods? (Apart from spite.)
>
> The getattr builtin is just a public interface to whatever internal
> function or functions the interpreter uses to look up attributes.

Okay. Lemme give it to you *even more clearly* since the previous
example didn't satisfy.

# file1.py

@extend(list)
def in_order(self):
return sorted(self)

def frob(stuff):
return stuff.in_order()

# file2.py

from file1 import frob
thing = [1, 5, 2]
frob(thing) # == [1, 2, 5]
def otherfrob(stuff):
return stuff.in_order()
otherfrob(thing) # AttributeError

Am I correct so far? The function imported from file1 has the
extension method, the code in file2 does not. That's the entire point
here, right?

Okay. Now, what if getattr is brought into the mix?

# file3.py
@extend(list)
def in_order(self):
return sorted(self)

def fetch1(stuff, attr):
if attr == "in_order": return stuff.in_order
if attr == "unordered": return stuff.unordered
return getattr(stuff, attr)

def fetch2(stuff, attr):
return getattr(stuff, attr)

# file4.py
from file3 import fetch1, fetch2
import random

@extend(list)
def unordered(self):
return random.shuffle(self[:])

def fetch3(stuff, attr):
if attr == "in_order": return stuff.in_order
if attr == "unordered": return stuff.unordered
return getattr(stuff, attr)

def fetch4(stuff, attr):
return getattr(stuff, attr)

thing = [1, 5, 2]
fetch1(thing, "in_order")()
fetch2(thing, "in_order")()
fetch3(thing, "in_order")()
fetch4(thing, "in_order")()
fetch1(thing, "unordered")()
fetch2(thing, "unordered")()
fetch3(thing, "unordered")()
fetch4(thing, "unordered")()

Okay. *NOW* which ones raise AttributeError, and which ones give the
extension method? What exactly are the semantics of getattr? Is it a
magical function that can reach back into the module that called it,
or is it actually a function of its own? And if getattr is supposed to
reach back into the other module, why shouldn't other functions be
able to?

Please explain exactly what the semantics of getattr are, and exactly
which modules it is supposed to be able to see. Remember, it is not a
compiler construct or an operator. It is a function, and it lives in
its own module (the builtins).

> Not a rhetorical question: is that how it works in something like Swift,
> or Kotlin?

I have no idea. I'm just asking how you intend it to work in Python.
If you want to cite other languages, go ahead, but I'm not assuming
that they already have the solution, because they are different
languages. Also not a rhetorical question: Is their getattr equivalent
actually an operator or compiler construct, rather than being a
regular function? Because if it is, then the entire problem doesn't
exist.

> > And what about this?
> >
> > f = functools.partial(getattr, stuff)
> > f("in_order")
> >
> > NOW which extension methods should apply? Those registered here? Those
> > registered in the builtins? Those registered in functools?
>
> partial is just a wrapper around its function argument, so that should
> behave *exactly* the same as `getattr(stuff, 'in_order')`.

So if it behaves exactly the same way that getattr would, then is it
exactly the same as fetch2 and fetch4? If not, how is it different?

What about other functions implemented in C? If I write a C module
that calls

[Python-ideas] Re: Extension methods in Python

On Tue, Jun 22, 2021 at 10:25:33PM +1000, Chris Angelico wrote:

> > If its a problem for getattr, it is a problem for dot syntax, because
> > they are essentially the same thing.
> 
> Ahh but that is precisely the problem.

Is it? Don't be shy. Tell us what the problem is and why its a problem.

> > > How is getattr defined?
> >
> > The same as it is defined now, except with some minor tweaks to support
> > extension methods.
> 
> Do those tweaks include reaching back into the module that called it?
> How magical will it be?

I thought you agreed that we didn't need to discuss implementation until 
we had decided on the desired semantics?

Let's just say it will be a well-defined, totally non-magical 
implementation (like everything else in Python) that manages to be 
equally efficient as regular attribute access. A high bar to set. 
Implementation issues may require us to dial that back a bit, or might 
even rule out the concept altogether, but let's start off by assuming 
the best and decide on the semantics first.

[...]
> Let me clarify then.

Thank you, that would be helpful.

> We shall assume for the moment that the builtins module does not have
> any extension methods registered. (I suppose it could, but then you
> get all the action-at-a-distance of monkey-patching AND the problems
> of extension methods, so I would hope people don't do this.)

How do you know that the builtins aren't already using extension 
methods?

Let's pretend that you didn't know that CPython's implementation was C 
rather than C#. Or that C has support for something similar to extension 
methods. (I daresay you could simulate it, somehow.) Or that we're 
talking about IronPython, for example, which is implemented in C#. I 
might tell you that list.sort and list.index are regular methods, and 
that list.append and list.reverse are extension methods.

Short of looking at the source code, there would be absolutely no way 
for you to tell if I were correct or not.

So if builtins used extension methods, that would be indistinguishable 
from builtins not using extension methods.

(To be pedantic: this would only be true if those extension methods were 
added at interpreter startup, before the interpreter ran any user code. 
Otherwise you could take a snapshot of `dir(list)` before and after, and 
inspect the differences.)

To be clear, this is distinct from *a user module* using extension 
methods on a builtin type, which is normal and the point of the 
exercise.

Do I need to explain the difference between the interpreter using 
extension methods as part of the builtins implementation, and user- 
written modules ("spam.py") extending builtin classes with extension 
types? Because they are completely different things.

> This means that the getattr() function, being a perfectly 
> straight-forward function, is not going to see any extension methods.

Does getattr see slots (both C-level and Python)? Yes. Does it see 
attributes in instance and class dicts? Yes. Does it see dynamic 
attributes that use `__getattr__`? Yes. Does it understand the 
descriptor protocol? Yes.

It does everything else dot notation does. Why wouldn't it see extension 
methods? (Apart from spite.)

The getattr builtin is just a public interface to whatever internal 
function or functions the interpreter uses to look up attributes.

> # whatever the actual syntax is
> @extend(list)
> def in_order(self):
> return sorted(self)
> 
> stuff = [1, 5, 2]
> stuff.in_order() # == [1, 2, 5]
> getattr(stuff, "in_order")() # AttributeError

What reason do you have for thinking that would be how it works?

Not a rhetorical question: is that how it works in something like Swift,
or Kotlin?

> Does the getattr function see the extension methods? If so, which?

Yes, and the same ones you would see if you used dot syntax.

> If not, how can getattr return the same thing as attribute lookup 
> does?

I think you've just answered your own question. getattr has to return 
the same thing as attribute lookup, because if it didn't, it wouldn't be 
returning the same thing as attribute lookup, which is getattr's reason 
to exist.

> How do you inform getattr of which extension methods it should be looking at?

You don't. You inform the interpreter tha you are opting in to use 
extension methods on a type, the interpreter does whatever it needs to 
do to make it work (implementation), and then it Just Works™.

> And what about this?
> 
> f = functools.partial(getattr, stuff)
> f("in_order")
>
> NOW which extension methods should apply? Those registered here? Those
> registered in the builtins? Those registered in functools?

partial is just a wrapper around its function argument, so that should 
behave *exactly* the same as `getattr(stuff, 'in_order')`.

> Yes, monkey-patching *is* cleaner, because the object is the same
> object no matter how you look it up.

Oh for heaven's sake, I'm not proposing changes to Python's object 
identity model! Please don't

[Python-ideas] Re: Extension methods in Python




On 2021-06-22 8:11 p.m., Chris Angelico wrote:
> On Wed, Jun 23, 2021 at 9:06 AM Soni L.  wrote:
> > On 2021-06-22 7:38 p.m., Chris Angelico wrote:
> > > Have you actually tried designing this into a larger project to see
> > > what problems you run into, or is this something you've only
> > > considered at this trivial level?
> >
> > 1. It's opt-in.
> > 2. It's designed to be used by a hypothetical extension methods module,
> > but without imposing any design constraints on such module. It could
> > return a named function every time a given name is looked up (a la "bind
> > the first argument" operator), or do dynamic dispatch based on types or
> > ABCs (a la proper extension methods).
> >
> > In practice, you don't def your own __dot__, but rather use someone
> > else's "__dot__ builder". If you don't wanna deal with it, just don't
> > use __dot__.
> >
> > It's also useful for the occasional domain-specific language.
> >
>
> What I'm hearing from you is: "No". Lots of words to say it, but, the
> answer to my question seems to be no.

We have never felt a need for extension methods in python, no.

>
> ChrisA
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/BZ7BR5ACN2VXJQA6LVV6FNSOCKFVQTWT/
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/Q57V3QDZ6O6WZZXYQDG4E3RRFSPZO62L/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

On Wed, Jun 23, 2021 at 9:06 AM Soni L.  wrote:
> On 2021-06-22 7:38 p.m., Chris Angelico wrote:
> > Have you actually tried designing this into a larger project to see
> > what problems you run into, or is this something you've only
> > considered at this trivial level?
>
> 1. It's opt-in.
> 2. It's designed to be used by a hypothetical extension methods module,
> but without imposing any design constraints on such module. It could
> return a named function every time a given name is looked up (a la "bind
> the first argument" operator), or do dynamic dispatch based on types or
> ABCs (a la proper extension methods).
>
> In practice, you don't def your own __dot__, but rather use someone
> else's "__dot__ builder". If you don't wanna deal with it, just don't
> use __dot__.
>
> It's also useful for the occasional domain-specific language.
>

What I'm hearing from you is: "No". Lots of words to say it, but, the
answer to my question seems to be no.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/BZ7BR5ACN2VXJQA6LVV6FNSOCKFVQTWT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python




On 2021-06-22 7:38 p.m., Chris Angelico wrote:
> On Wed, Jun 23, 2021 at 8:30 AM Soni L.  wrote:
> >
> >
> >
> > On 2021-06-22 5:54 p.m., Chris Angelico wrote:
> > > On Wed, Jun 23, 2021 at 6:41 AM Soni L.  wrote:
> > > > It would have local scope, similar to uh... locals. Y'know how locals
> > > > are just sugar for locals()['foo'] and stuff? Yeah.
> > >
> > > Not really, no, they're not. :) The dictionary returned by locals()
> > > isn't actually an implementation detail of local name lookups.
> >
> > It's... part of the language. Not an implementation detail. The
> > dictionary returned by locals() is an inherent part of local name
> > lookups, isn't it?
>
> No, it's not. Most definitely not.
>
> https://docs.python.org/3/library/functions.html#locals

Ohh. Fair enough, sorry.

> > > Have you put any thought into how you would deal with the problem of
> > > recursive __dot__ calls?
> >
> > Let it recurse!
> >
> > Globals and locals don't go through __dot__, so you can just... use
> > them. In particular, you can always use getattr(), and probably should.
> > Or even set __dot__ to getattr inside it, like so:
> >
> > def __dot__(left, right):
> >   __dot__ = getattr
> >   foo.bar # same as getattr(foo, "bar") because we set (local) __dot__
> > to getattr above
>
> I can't actually pin down what I'm averse to here, but it gives me a
> really REALLY bad feeling. You're expecting every attribute lookup to
> now look for a local or global name __dot__ (or, presumably, a
> nonlocal, class, or builtin), and do whatever that does. That seems
> like a really effective foot-gun.
>
> Have you actually tried designing this into a larger project to see
> what problems you run into, or is this something you've only
> considered at this trivial level?

1. It's opt-in.
2. It's designed to be used by a hypothetical extension methods module,
but without imposing any design constraints on such module. It could
return a named function every time a given name is looked up (a la "bind
the first argument" operator), or do dynamic dispatch based on types or
ABCs (a la proper extension methods).

In practice, you don't def your own __dot__, but rather use someone
else's "__dot__ builder". If you don't wanna deal with it, just don't
use __dot__.

It's also useful for the occasional domain-specific language.

>
> ChrisA
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/YOPTCIYV4GSW3N7EA7KPLJBKNVSNANXZ/
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/LQW3GLWRGPOZDOAS7LGUL4WLP6JOUKUR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

On Wed, Jun 23, 2021 at 8:30 AM Soni L.  wrote:
>
>
>
> On 2021-06-22 5:54 p.m., Chris Angelico wrote:
> > On Wed, Jun 23, 2021 at 6:41 AM Soni L.  wrote:
> > > It would have local scope, similar to uh... locals. Y'know how locals
> > > are just sugar for locals()['foo'] and stuff? Yeah.
> >
> > Not really, no, they're not. :) The dictionary returned by locals()
> > isn't actually an implementation detail of local name lookups.
>
> It's... part of the language. Not an implementation detail. The
> dictionary returned by locals() is an inherent part of local name
> lookups, isn't it?

No, it's not. Most definitely not.

https://docs.python.org/3/library/functions.html#locals

> > Have you put any thought into how you would deal with the problem of
> > recursive __dot__ calls?
>
> Let it recurse!
>
> Globals and locals don't go through __dot__, so you can just... use
> them. In particular, you can always use getattr(), and probably should.
> Or even set __dot__ to getattr inside it, like so:
>
> def __dot__(left, right):
>   __dot__ = getattr
>   foo.bar # same as getattr(foo, "bar") because we set (local) __dot__
> to getattr above

I can't actually pin down what I'm averse to here, but it gives me a
really REALLY bad feeling. You're expecting every attribute lookup to
now look for a local or global name __dot__ (or, presumably, a
nonlocal, class, or builtin), and do whatever that does. That seems
like a really effective foot-gun.

Have you actually tried designing this into a larger project to see
what problems you run into, or is this something you've only
considered at this trivial level?

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/YOPTCIYV4GSW3N7EA7KPLJBKNVSNANXZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python



On 2021-06-22 5:34 p.m., Brendan Barnwell wrote:
> On 2021-06-22 13:09, Soni L. wrote:
>> Think about it like this, extension methods give you the ability to make
>> imported functions that look like this:
>>
>> foo(bar, baz)
>>
>> look like this instead:
>>
>> bar.foo(baz)
>>
>> That's all there is to them. They're just a lie to change how you
>> read/write the code. Some languages have an whole operator that has a
>> similar function, where something like bar->foo(baz) is sugar for
>> foo(bar, baz). The OP doesn't specify any particular mechanism for
>> extension methods, so e.g. making the dot operator be implemented by a
>> local function in the module, which delegates to the current attribute
>> lookup mechanism by default, would be perfectly acceptable. It's like
>> deprecating the existing dot operator and introducing a completely
>> different one that has nothing to do with attribute lookup!
>
> Okay, if that's the case, then I just think it's a bad idea.  :-)
>
> We already have a definition for what bar.foo does, and it's
> totally under the control of the bar object (via the
> __getattr__/__getattribute__ mechanism).  The idea that other things
> would be able to hook in there does not appeal to me at all.
>
> I don't really understand why you would want such a thing, to be
> honest.  I feel it would make code way more difficult to reason about,
> as it would break locality constraints every which way.  Now every
> time you see`bar.foo` you would have to think about all kinds of other
> modules that may be hooking in and adding their own complications.
> What's the point?
>
> Mostly the whole benefit of the dot notation is that it specifies
> a locally constrained relationship between the object and the
> attribute: you know that bar.foo does what bar decides, and no one
> else gets any say (unless bar asks for their opinion, e.g. by
> consulting global variables or whatever).  If we want to write
> foo(bar, baz). . . well, we can just do that!  What you're describing
> would just make existing attribute usages harder to understand while
> only "adding" something we can already do quite straightforwardly.
>
> Imagine a similar proposal for other syntax.  Suppose that in any
> module I could define a function called operator_add and then other
> modules could "import" this "extension" so that every use of the +
> operator would somehow hook into this operator_add function.  So now
> every time you do 2 + 2 you might be invoking some extension behavior.
> In my view that is unambiguously a road to madness, and as far as I
> can tell the extension mechanism you're proposing is equally ill-advised.
>

Imagine if Python didn't have an + operator, but instead an + *infix
function*.

Thus, every module would automatically include the global

def infix +(left, right):
  ...

And indeed, you could say we already have this. Except currently you
can't define your own local infix +. But what if you *could*?

What if you could just,

# file1.py
def infix +(left, right):
  return left << right
x = 4 + 4

# file2.py
def infix +(left, right):
  return left ** right
x = 4 + 4

# file3.py
import file1
import file2
print(file1.x) # 64
print(file2.x) # 256
print(4 + 4) # 8

How does this break locality?

Same idea with the dot operator, really.

(Some languages don't have operators, but only functions. They let you
do just this.)
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/QBQCSJELOHSPOG24X2LMSYBSKCGQRVYP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python



On 2021-06-22 5:54 p.m., Chris Angelico wrote:
> On Wed, Jun 23, 2021 at 6:41 AM Soni L.  wrote:
> >
> >
> >
> > On 2021-06-22 5:23 p.m., Chris Angelico wrote:
> > > On Wed, Jun 23, 2021 at 6:13 AM Soni L.  wrote:
> > > > Think about it like this, extension methods give you the ability to make
> > > > imported functions that look like this:
> > > >
> > > > foo(bar, baz)
> > > >
> > > > look like this instead:
> > > >
> > > > bar.foo(baz)
> > > >
> > > > That's all there is to them. They're just a lie to change how you
> > > > read/write the code. Some languages have an whole operator that has a
> > > > similar function, where something like bar->foo(baz) is sugar for
> > > > foo(bar, baz). The OP doesn't specify any particular mechanism for
> > > > extension methods, so e.g. making the dot operator be implemented by a
> > > > local function in the module, which delegates to the current attribute
> > > > lookup mechanism by default, would be perfectly acceptable. It's like
> > > > deprecating the existing dot operator and introducing a completely
> > > > different one that has nothing to do with attribute lookup!
> > >
> > > uh... I'm lost. Are you saying that that's a good thing? You *want* to
> > > replace the existing dot operator with one that has nothing to do with
> > > attribute lookup?? I don't get it.
> >
> > Sure! As long as the new one can call getattr!
> >
> > Let's say the new dot operator looks like this:
> >
> > # file1.py
> > def __dot__(left, right):
> >   print(left)
> >   print(right)
> >   ...
> > foo = []
> > foo.bar
> >
> > Now, this would actually print the list [] and the string "bar".
> >
> > Then you can just use getattr to get attribute lookup behaviour out of it!
> >
> > def __dot__(left, right):
> >   return getattr(left, right)
> > foo = []
> > foo.bar
> >
> > It would have local scope, similar to uh... locals. Y'know how locals
> > are just sugar for locals()['foo'] and stuff? Yeah.
>
> Not really, no, they're not. :) The dictionary returned by locals()
> isn't actually an implementation detail of local name lookups.

It's... part of the language. Not an implementation detail. The
dictionary returned by locals() is an inherent part of local name
lookups, isn't it?

> Have you put any thought into how you would deal with the problem of
> recursive __dot__ calls?

Let it recurse!

Globals and locals don't go through __dot__, so you can just... use
them. In particular, you can always use getattr(), and probably should.
Or even set __dot__ to getattr inside it, like so:

def __dot__(left, right):
  __dot__ = getattr
  foo.bar # same as getattr(foo, "bar") because we set (local) __dot__
to getattr above

In languages with lexical scoping (instead of block scoping), the
compiler doesn't see things that haven't yet been declared. In those
languages, such a __dot__ function would actually inherit the global
__dot__ rather than recursing. But as you can see from the above
example, it's really not a big deal.

>
> ChrisA
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/NEI23TBWA3QKRGYIWTBJV3SK5O6MWXZC/
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/X7J5SNYPDJTCC5LKBG4SHA5YEQAU5AMA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

On Wed, Jun 23, 2021 at 6:41 AM Soni L.  wrote:
>
>
>
> On 2021-06-22 5:23 p.m., Chris Angelico wrote:
> > On Wed, Jun 23, 2021 at 6:13 AM Soni L.  wrote:
> > > Think about it like this, extension methods give you the ability to make
> > > imported functions that look like this:
> > >
> > > foo(bar, baz)
> > >
> > > look like this instead:
> > >
> > > bar.foo(baz)
> > >
> > > That's all there is to them. They're just a lie to change how you
> > > read/write the code. Some languages have an whole operator that has a
> > > similar function, where something like bar->foo(baz) is sugar for
> > > foo(bar, baz). The OP doesn't specify any particular mechanism for
> > > extension methods, so e.g. making the dot operator be implemented by a
> > > local function in the module, which delegates to the current attribute
> > > lookup mechanism by default, would be perfectly acceptable. It's like
> > > deprecating the existing dot operator and introducing a completely
> > > different one that has nothing to do with attribute lookup!
> >
> > uh... I'm lost. Are you saying that that's a good thing? You *want* to
> > replace the existing dot operator with one that has nothing to do with
> > attribute lookup?? I don't get it.
>
> Sure! As long as the new one can call getattr!
>
> Let's say the new dot operator looks like this:
>
> # file1.py
> def __dot__(left, right):
>   print(left)
>   print(right)
>   ...
> foo = []
> foo.bar
>
> Now, this would actually print the list [] and the string "bar".
>
> Then you can just use getattr to get attribute lookup behaviour out of it!
>
> def __dot__(left, right):
>   return getattr(left, right)
> foo = []
> foo.bar
>
> It would have local scope, similar to uh... locals. Y'know how locals
> are just sugar for locals()['foo'] and stuff? Yeah.

Not really, no, they're not. :) The dictionary returned by locals()
isn't actually an implementation detail of local name lookups.

Have you put any thought into how you would deal with the problem of
recursive __dot__ calls?

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/NEI23TBWA3QKRGYIWTBJV3SK5O6MWXZC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

2021-06-22 Thread Brendan Barnwell

On 2021-06-22 13:09, Soni L. wrote:

Think about it like this, extension methods give you the ability to make
imported functions that look like this:

foo(bar, baz)

look like this instead:

bar.foo(baz)

That's all there is to them. They're just a lie to change how you
read/write the code. Some languages have an whole operator that has a
similar function, where something like bar->foo(baz) is sugar for
foo(bar, baz). The OP doesn't specify any particular mechanism for
extension methods, so e.g. making the dot operator be implemented by a
local function in the module, which delegates to the current attribute
lookup mechanism by default, would be perfectly acceptable. It's like
deprecating the existing dot operator and introducing a completely
different one that has nothing to do with attribute lookup!

Okay, if that's the case, then I just think it's a bad idea. :-)

We already have a definition for what bar.foo does, and it's totally
under the control of the bar object (via the
__getattr__/__getattribute__ mechanism). The idea that other things
would be able to hook in there does not appeal to me at all.

I don't really understand why you would want such a thing, to be
honest. I feel it would make code way more difficult to reason about,
as it would break locality constraints every which way. Now every time
you see`bar.foo` you would have to think about all kinds of other
modules that may be hooking in and adding their own complications.
What's the point?

Mostly the whole benefit of the dot notation is that it specifies a
locally constrained relationship between the object and the attribute:
you know that bar.foo does what bar decides, and no one else gets any
say (unless bar asks for their opinion, e.g. by consulting global
variables or whatever). If we want to write foo(bar, baz). . . well, we
can just do that! What you're describing would just make existing
attribute usages harder to understand while only "adding" something we
can already do quite straightforwardly.

Imagine a similar proposal for other syntax. Suppose that in any
module I could define a function called operator_add and then other
modules could "import" this "extension" so that every use of the +
operator would somehow hook into this operator_add function. So now
every time you do 2 + 2 you might be invoking some extension behavior.
In my view that is unambiguously a road to madness, and as far as I can
tell the extension mechanism you're proposing is equally ill-advised.

--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."

--author unknown
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/4T35JHWPGIPWONX4PD7YPWEA2RXROSZI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python



On 2021-06-22 5:23 p.m., Chris Angelico wrote:
> On Wed, Jun 23, 2021 at 6:13 AM Soni L.  wrote:
> > Think about it like this, extension methods give you the ability to make
> > imported functions that look like this:
> >
> > foo(bar, baz)
> >
> > look like this instead:
> >
> > bar.foo(baz)
> >
> > That's all there is to them. They're just a lie to change how you
> > read/write the code. Some languages have an whole operator that has a
> > similar function, where something like bar->foo(baz) is sugar for
> > foo(bar, baz). The OP doesn't specify any particular mechanism for
> > extension methods, so e.g. making the dot operator be implemented by a
> > local function in the module, which delegates to the current attribute
> > lookup mechanism by default, would be perfectly acceptable. It's like
> > deprecating the existing dot operator and introducing a completely
> > different one that has nothing to do with attribute lookup!
>
> uh... I'm lost. Are you saying that that's a good thing? You *want* to
> replace the existing dot operator with one that has nothing to do with
> attribute lookup?? I don't get it.

Sure! As long as the new one can call getattr!

Let's say the new dot operator looks like this:

# file1.py
def __dot__(left, right):
  print(left)
  print(right)
  ...
foo = []
foo.bar

Now, this would actually print the list [] and the string "bar".

Then you can just use getattr to get attribute lookup behaviour out of it!

def __dot__(left, right):
  return getattr(left, right)
foo = []
foo.bar

It would have local scope, similar to uh... locals. Y'know how locals
are just sugar for locals()['foo'] and stuff? Yeah.
>
> ChrisA
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/XDY3VAYX3FXJI7ZVT3AYAFA7PGQMBWAA/
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/RGLMGLHAHLJVN3G5WEQSEKURUY7OP2UD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

On Wed, Jun 23, 2021 at 6:13 AM Soni L.  wrote:
> Think about it like this, extension methods give you the ability to make
> imported functions that look like this:
>
> foo(bar, baz)
>
> look like this instead:
>
> bar.foo(baz)
>
> That's all there is to them. They're just a lie to change how you
> read/write the code. Some languages have an whole operator that has a
> similar function, where something like bar->foo(baz) is sugar for
> foo(bar, baz). The OP doesn't specify any particular mechanism for
> extension methods, so e.g. making the dot operator be implemented by a
> local function in the module, which delegates to the current attribute
> lookup mechanism by default, would be perfectly acceptable. It's like
> deprecating the existing dot operator and introducing a completely
> different one that has nothing to do with attribute lookup!

uh... I'm lost. Are you saying that that's a good thing? You *want* to
replace the existing dot operator with one that has nothing to do with
attribute lookup?? I don't get it.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/XDY3VAYX3FXJI7ZVT3AYAFA7PGQMBWAA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python



On 2021-06-22 3:43 p.m., Brendan Barnwell wrote:
> On 2021-06-22 05:14, Chris Angelico wrote:
>> Fair point. However, I've worked with a good number of languages that
>> have some notion of object methods, and generally, an object has or
>> doesn't have a method based on what the object*is*, not on who's
>> asking.
>
> I agree, and this is the aspect of the proposal that most confuses
> me. I still can't understand concretely what is being proposed,
> though, so I'm not sure I even understand it.  Can someone clarify? 
> Suppose I have this
>
> *
> ### file1.py
> @extend(list)
> def len2(self):
> return len(self)**2
>
> ### file2.py
> # or whatever I do to say "I want to use extensions to list defined in
> file1"
> from file1 extend list
>
> def coolness(some_list):
> return some_list.len2() + 1
>
> my_list = [1, 2, 3]
> print("My list len2:", my_list.len2())
> print("My list coolness:", coolness(my_list))
>
> ### file3.py
> import file2
>
> other_list = [1, 2, 3, 4]
> print("Other list len2:", other_list.len2())
> print("other list coolness:", file2.coolness(other_list))
> print("My list len2 from outside:", file2.my_list.len2())
> print("My list coolness from outside:", file2.coolness(file2.my_list))
> *
>
> What exactly is supposed to happen here if I run file3?  file2
> declares use of file1's extensions.  file2 does not.  But file3 uses a
> function in file2 that makes use of such extensions.  Who sees the
> extension?
>

NameError, value, NameError, value, respectively.

> The list object my_list in file2 is the same object accessed as
> file2.my_list in file3.  Likewise coolness and file2.coolness.  It is
> going to be super confusing if calling the same function object with
> the same list object argument gives different results depending on
> which file you're in.  Likewise it's going to be confusing if the same
> list object sometimes has a .len2 method and sometimes doesn't.

It isn't the list object that has the extension method.

>
> But if it doesn't work that way, then it would seem to mean either
> every module sees the extensions (even if they didn't opt in), or else
> my_list in file2 is not the same object as file2.my_list in file3. 
> And that would be even worse.  (In this example it may seem okay
> because you can ask why I would call len2 from file3 if I didn't want
> to use it. But what if the extension is an override of an existing
> method?  Is that not allowed?)
>
> In addition, if there is a difference between my_list and
> other_list, then that apparently means that the syntax for lists now
> does something different in the two files.  This is maybe the most
> reasonable approach, since it's at least remotely reminiscent of a
> __future__ import, which changes syntactic behavior.  But what exactly
> is the difference between the two objects here?  Are both objects
> lists?  If they are, then how can they have different methods?  If
> they're not, then what are they?
>
> Most __future__ imports don't work like this.  Maybe the closest
> thing is the generator_stop one, but at least that places a flag on
> the code object to indicate the difference.  Would "extended lists"
> have some kind of magic attribute indicating which extensions they're
> using?  That may have been marginally acceptable in the case of PEP
> 479, which was essentially a bugfix, and set the attribute on code
> objects which are an obscure internal data structure.  But allowing
> this kind of thing for "user-facing" objects like lists would create a
> profusion of different list objects with different behavior depending
> on some combination of attributes indicating "what extends me" --- or,
> even worse, create different behavior without any such overt
> indication of which extensions are in use for a given object.
>
> The idea that the file in which code is written would somehow
> determine this type of runtime behavior seems to me to break my
> assumption that by knowing an object's identity I should have all the
> information I need to know about how to use it.  Some of the posts
> earlier in this thread seem to suggest that somehow the module where
> something was defined (something --- not sure what --- maybe the
> object with the extended method?  maybe the extended method itself?)
> would somehow get a hook to override attribute access on some objects
> (again, not sure which objects).
>
> That to me is the exact opposite of encapsulation.  Encapsulation
> means the object itself contains all its behavior.  If there is some
> getattr-like hook in some other module somewhere that is lying in wait
> to override attribute access on a given object "only sometimes" then
> that's not encapsulation at all.  It's almost as bad as the infamous
> COME FROM statement!
>
> Existing mechanisms like __getattribute__ are not parallel at all.
> When you know an object's identity, you know its MRO, which tells you
> all you need to know about what __getattribute__

[Python-ideas] Re: Extension methods in Python

2021-06-22 Thread Brendan Barnwell


On 2021-06-22 05:14, Chris Angelico wrote:

Fair point. However, I've worked with a good number of languages that
have some notion of object methods, and generally, an object has or
doesn't have a method based on what the object*is*, not on who's
asking.


	I agree, and this is the aspect of the proposal that most confuses me. 
 I still can't understand concretely what is being proposed, though, so 
I'm not sure I even understand it.  Can someone clarify?  Suppose I have 
this


*
### file1.py
@extend(list)
def len2(self):
return len(self)**2

### file2.py
# or whatever I do to say "I want to use extensions to list defined in 
file1"

from file1 extend list

def coolness(some_list):
return some_list.len2() + 1

my_list = [1, 2, 3]
print("My list len2:", my_list.len2())
print("My list coolness:", coolness(my_list))

### file3.py
import file2

other_list = [1, 2, 3, 4]
print("Other list len2:", other_list.len2())
print("other list coolness:", file2.coolness(other_list))
print("My list len2 from outside:", file2.my_list.len2())
print("My list coolness from outside:", file2.coolness(file2.my_list))
*

	What exactly is supposed to happen here if I run file3?  file2 declares 
use of file1's extensions.  file2 does not.  But file3 uses a function 
in file2 that makes use of such extensions.  Who sees the extension?


	The list object my_list in file2 is the same object accessed as 
file2.my_list in file3.  Likewise coolness and file2.coolness.  It is 
going to be super confusing if calling the same function object with the 
same list object argument gives different results depending on which 
file you're in.  Likewise it's going to be confusing if the same list 
object sometimes has a .len2 method and sometimes doesn't.


	But if it doesn't work that way, then it would seem to mean either 
every module sees the extensions (even if they didn't opt in), or else 
my_list in file2 is not the same object as file2.my_list in file3.  And 
that would be even worse.  (In this example it may seem okay because you 
can ask why I would call len2 from file3 if I didn't want to use it. 
But what if the extension is an override of an existing method?  Is that 
not allowed?)


	In addition, if there is a difference between my_list and other_list, 
then that apparently means that the syntax for lists now does something 
different in the two files.  This is maybe the most reasonable approach, 
since it's at least remotely reminiscent of a __future__ import, which 
changes syntactic behavior.  But what exactly is the difference between 
the two objects here?  Are both objects lists?  If they are, then how 
can they have different methods?  If they're not, then what are they?


	Most __future__ imports don't work like this.  Maybe the closest thing 
is the generator_stop one, but at least that places a flag on the code 
object to indicate the difference.  Would "extended lists" have some 
kind of magic attribute indicating which extensions they're using?  That 
may have been marginally acceptable in the case of PEP 479, which was 
essentially a bugfix, and set the attribute on code objects which are an 
obscure internal data structure.  But allowing this kind of thing for 
"user-facing" objects like lists would create a profusion of different 
list objects with different behavior depending on some combination of 
attributes indicating "what extends me" --- or, even worse, create 
different behavior without any such overt indication of which extensions 
are in use for a given object.


	The idea that the file in which code is written would somehow determine 
this type of runtime behavior seems to me to break my assumption that by 
knowing an object's identity I should have all the information I need to 
know about how to use it.  Some of the posts earlier in this thread seem 
to suggest that somehow the module where something was defined 
(something --- not sure what --- maybe the object with the extended 
method?  maybe the extended method itself?) would somehow get a hook to 
override attribute access on some objects (again, not sure which objects).


	That to me is the exact opposite of encapsulation.  Encapsulation means 
the object itself contains all its behavior.  If there is some 
getattr-like hook in some other module somewhere that is lying in wait 
to override attribute access on a given object "only sometimes" then 
that's not encapsulation at all.  It's almost as bad as the infamous 
COME FROM statement!


	Existing mechanisms like __getattribute__ are not parallel at all. 
When you know an object's identity, you know its MRO, which tells you 
all you need to know about what __getattribute__ calls might happen. 
You don't need to know anything about where the object "came from" or 
what file you're using it in.  But it seems with this proposal you would 
need to know, and that's kind of creepy to me.


--
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is no

[Python-ideas] Re: Extension methods in Python




On 2021-06-22 9:25 a.m., Chris Angelico wrote:
> (Oh, and another wrinkle, although a small one: Code objects would
> need to keep track of their modules. Currently functions do, but code
> objects don't. But that seems unlikely to introduce further
> complications.)

What? No you just stop emitting LOAD_ATTR and instead emit LOAD_LOCAL
'__opcode_load_attr_impl__' and call it with the relevant values.

>
> ChrisA
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/H5QSWNLWC4HOR3JPXBNDWCSE2QFHXPS6/
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/GLM2YWGRWHOAIQ7EJOXUNPGPY7HEEZPN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

On Tue, Jun 22, 2021 at 9:56 PM Steven D'Aprano  wrote:
>
> On Tue, Jun 22, 2021 at 09:12:53PM +1000, Chris Angelico wrote:
>
> > > The must be no semantic difference between:
> > >
> > > obj.method(arg)
> > >
> > > and
> > >
> > > getattr(obj, 'method')(arg)
> > >
> > > regardless of whether `method` is a regular method or an extension
> > > method.
> >
> > And this is a problem.
>
> If its a problem for getattr, it is a problem for dot syntax, because
> they are essentially the same thing.

Ahh but that is precisely the problem.

> > How is getattr defined?
>
> The same as it is defined now, except with some minor tweaks to support
> extension methods.

Do those tweaks include reaching back into the module that called it?
How magical will it be?

> > Is it counted as being in the current module?
>
> `getattr`? No, that's a builtin. You can shadow it or delete it if you
> want, it's just a public API to the underlying functionality built
> into the interpreter. Dot syntax won't be affected.

Let me clarify then.

We shall assume for the moment that the builtins module does not have
any extension methods registered. (I suppose it could, but then you
get all the action-at-a-distance of monkey-patching AND the problems
of extension methods, so I would hope people don't do this.) This
means that the getattr() function, being a perfectly straight-forward
function, is not going to see any extension methods.

Okay then.

# whatever the actual syntax is
@extend(list)
def in_order(self):
return sorted(self)

stuff = [1, 5, 2]
stuff.in_order() # == [1, 2, 5]
getattr(stuff, "in_order")() # AttributeError

Does the getattr function see the extension methods? If so, which? If
not, how can getattr return the same thing as attribute lookup does?

How do you inform getattr of which extension methods it should be looking at?

And what about this?

f = functools.partial(getattr, stuff)
f("in_order")

NOW which extension methods should apply? Those registered here? Those
registered in the builtins? Those registered in functools?

Yes, monkey-patching *is* cleaner, because the object is the same
object no matter how you look it up.

(Oh, and another wrinkle, although a small one: Code objects would
need to keep track of their modules. Currently functions do, but code
objects don't. But that seems unlikely to introduce further
complications.)

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/H5QSWNLWC4HOR3JPXBNDWCSE2QFHXPS6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

On Tue, Jun 22, 2021 at 9:23 PM Steven D'Aprano  wrote:
>
> On Tue, Jun 22, 2021 at 05:50:48PM +1000, Chris Angelico wrote:
>
> > Hmm, that's not what I'd usually understand "encapsulation" to mean.
> > That's what would normally be called "namespacing".
>
> Pfft, who you going to believe, me or some random folx on the internet
> editing Wikipedia? *wink*
>
> Okay, using the Wikipedia/OOP definition still applies. Presumably most
> extension methods are going to be regular instance methods or class
> methods, rather than staticmethod. So they will take a `self` (or `cls`)
> parameter, and presumably most such extension methods will actually act
> on that self parameter in some way.
>
> There is your "bundling of data with the methods that operate on that
> data", as required :-)

Okay, that's fair. Granted. It's not ALL of encapsulation, but it is,
to an extent encapsulation. (It is also namespacing, and your
justification of it was actually a justification of namespacing; but
this is Python, and I think we all agree that namespacing is good!)

> Regarding your example, you're only confused because you haven't take on
> board the fact that extension methods aren't interpreter-global,
> just module-global. Because it's new and unfamiliar. But we can do
> exactly the same thing, right now, with functions instead of methods,
> and you will find it trivially easy to diagnose the fault:
>
>
> # file1.py
> from file2 import func
> # instead of "applying an extension method" from elsewhere,
> # import a function from elsewhere
> from extra_functions import g
> def spamify(obj):
>  print(g(obj)) # works fine
>  print(func(obj))  # fails
>
>
> # file2.py
> def func(obj):
> return g(obj)  # NameError
>
>
> This example looks easy and not the least bit scary to you because
> you've been using Python for a while and it has become second nature to
> you. But you might remember back when you were a n00b, it probably
> confused you: why doesn't `g(obj)` work when you imported it? How
> weird and confusing! What do you mean, if I want to use g, I have to
> import it in each and every module where I want to use it? That's just
> dumb. Importing g once should make it available EVERYWHERE, right?
>
> Been there, done that.

Fair point. However, I've worked with a good number of languages that
have some notion of object methods, and generally, an object has or
doesn't have a method based on what the object *is*, not on who's
asking. It's going to make for some extremely confusing results. Is
getattr() going to act as part of the builtins module or the module
that's calling it? What about hasattr? What about an ABC's instance
check, or anything else?

How do other languages deal with this? How do they have a getattr-like
function? Does it have to be a compiler construct?

> Chris, here you are defending monkey-patching, not just as a necessary
> evil under some circumstances, but as a "cleaner" option, and then in
> your very next sentence:
>
> > And the Ruby community is starting to see the risks of
> > monkey-patching.
>

Yes. That is exactly right. I am claiming that monkey-patching is, in
many MANY cases, a cleaner option than extension methods. And then I
am saying that monkey-patching is usually a bad thing. This is not
incompatible, and it forms a strong view of my opinion of extension
methods.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/RCIR6GEJ6YA2Z2ZAFJJ7V2CJOGT6BRKP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

On Tue, Jun 22, 2021 at 09:12:53PM +1000, Chris Angelico wrote:

> > The must be no semantic difference between:
> >
> > obj.method(arg)
> >
> > and
> >
> > getattr(obj, 'method')(arg)
> >
> > regardless of whether `method` is a regular method or an extension
> > method.
>
> And this is a problem.

If its a problem for getattr, it is a problem for dot syntax, because 
they are essentially the same thing.

> How is getattr defined?

The same as it is defined now, except with some minor tweaks to support 
extension methods.

Do you remember when we introduced `__slots__` (version 2.2 or 2.3, I 
think?), and added a whole new mechanism to look up ordinary attributes 
and slot attributes? No, neither do I, because we didn't.

We have a single mechanism for looking up attributes, including methods, 
which works with instances and classes, descriptors and non-descripters, 
C-level slots and Python `__slots__` and `__dicts__` and `__getattr__` 
and `__getattribute__`, and I am absolutely positively sure that if 
Python ever adds a new implementation for attribute lookup, it will 
still be handled by the same getattr mechanism, which is built into the 
interpreter.

> Is it counted as being in the current module?

`getattr`? No, that's a builtin. You can shadow it or delete it if you 
want, it's just a public API to the underlying functionality built 
into the interpreter. Dot syntax won't be affected.

> If it is, then has getattr magically become
> part of the module it's called from? Or do ALL lookups depend on where
> the function was called, rather than where it's defined?

Can you be a bit more precise? I'm not suggesting that we introduce 
dynamic scoping instead of lexical scoping, if that's what you mean.

Attribute lookups already depend on the state of the object at the time 
the lookup is made. This is just more of the same.

class K:
pass

K.attr  # this is an AttributeError

K.attr = 'extension'
K.attr  # this is fine

> If 'method' is an extension method, where exactly is it visible?

I believe that TypeScript uses "import" for this, so it would be visible 
from anywhere that imports it:

https://putridparrot.com/blog/extension-methods-in-typescript/

-- 
Steve
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/5CEA6XWEX7VP554RIC4TDTJMURRVAOJL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

Oh this is a long one.

Hypothetically, let's say you have a proxy object:

class Foo:
  def __getattribute__(self, thing):
    return getattr(super().__getattribute__(self, "proxied"), thing)

Should this really include extension methods into it by default?

This is clearly wrong. The local override for the LOAD_ATTR opcode
should NOT apply to proxy methods except where explicitly requested.
Also sometimes you are supposed to call the dunder directly, like in the
above example. It's not *bad* to do it if you know what you're doing.

The point is that the caller using your proxy object should opt-in to
the extension methods, rather than break with no way to opt-out of them.
Your extension methods shouldn't propagate to proxy objects.

To go even further, should all your class definitions that happen to
extend a class with in-scope extension methods automatically gain those
extension methods? Because with actual extension methods, that doesn't
happen. You can have class MyList(list): pass and other callers would
not get MyList.flatten even with you being able to use MyList.flatten
locally.

Extension methods are more like Rust traits than inheritance-based OOP.
Also note that they use instance method syntax, but no other. That is
they apply to LOAD_ATTR opcodes but should not apply to getattr!
(Indeed, reflection in C#/Kotlin doesn't see the extension methods!)

On 2021-06-22 6:57 a.m., Steven D'Aprano wrote:
> I'm sorry Soni, I don't understand what you are arguing here. See below.
>
>
> On Mon, Jun 21, 2021 at 10:09:17PM -0300, Soni L. wrote:
> > 
> > 
> > On 2021-06-21 9:39 p.m., Steven D'Aprano wrote:
> > 
> > 
> > >
> > > Fourth step is that you go ahead and use lists as normal. Whether you 
> > > use getattr or dot syntax, any extension methods defined in spam.py will 
> > > show up, as if they were actual list methods.
> > >
> > > hasattr([], 'head')  # returns True
> > > list.tail  # returns the spam.tail function object (unbound method)
> > >
> > > They're not monkey-patched: other modules don't see that.
> > >
> > >
> > 
> > Python is a dynamic language. Maybe you're using hasattr/getattr to
> > forward something from A to B. If "other modules don't see that" then
> > this must work as if there were no extension methods in place.
>
> What's "forward something from A to B" mean? What are A and B?
>
> If "this" (method lookups) "must work as if there were no extension 
> methods in place" then extension methods are a no-op and are pointless. 
> You write an extension method, register it as applying to a type, the 
> caller opts-in to use it, and then... nothing happens, because it "must 
> work as if there were no extension methods in place".
>
> Surely that isn't what you actually want to happen. But if not, I have 
> no idea what you mean.
>
> The whole point of extension methods is that once the caller opts in to 
> use them, method look ups (and that includes hasattr and getattr) must 
> work as if the extension methods **are in place**.
>
> The must be no semantic difference between:
>
> obj.method(arg)
>
> and
>
> getattr(obj, 'method')(arg)
>
> regardless of whether `method` is a regular method or an extension 
> method.
>
>
> > So you
> > actually wouldn't want the local load_attr override to apply to those.
> > If you did... well, just call the override directly.
>
> I have no idea what that means. What is "the local load_attr override"?
>
>
> > If the override was
> > called __opcode_load_attr_impl__ you'd just call
> > __opcode_load_attr_impl__ directly instead of going through getattr.
>
> As a general rule, you should not be calling dunders directly.
>
> You seem to have missed the point that extension methods are intended as 
> a mechanism to **extend a type** by giving it new methods on an opt-in 
> basis. I want to call them "virtual methods" except that would add 
> confusion regarding virtual subclasses and ABCs etc.
>
> Maybe you need to read the Kotlin docs:
>
> https://kotlinlang.org/docs/extensions.html
>
> and the C# docs:
>
> https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/classes-and-structs/extension-methods
>
> Wikipedia also has a broad overview from a language-agnostic 
> perspective:
>
> https://en.wikipedia.org/wiki/Extension_method
>
> Here's an example in TypeScript and Javascript:
>
> https://putridparrot.com/blog/extension-methods-in-typescript/
>
>
>
> In particular note these comments:
>
> # Kotlin
> "Such functions are available for calling in the usual way as if they 
> were methods of the original class."
>
> # C#
> "Extension methods are only in scope when you explicitly import the 
> namespace into your source code with a using directive."
>
> Both C# and Kotlin are statically typed languages, and Python is not, 
> but we ought to aim to minimise the differences in semantics. Aside from 
> extension methods being resolved at runtime instead of at compile time, 
> the behaviour ought to be as close as possible.
>
> Just as

[Python-ideas] Re: Extension methods in Python

On Tue, Jun 22, 2021 at 05:50:48PM +1000, Chris Angelico wrote:

> Hmm, that's not what I'd usually understand "encapsulation" to mean.
> That's what would normally be called "namespacing".

Pfft, who you going to believe, me or some random folx on the internet 
editing Wikipedia? *wink*

Okay, using the Wikipedia/OOP definition still applies. Presumably most 
extension methods are going to be regular instance methods or class 
methods, rather than staticmethod. So they will take a `self` (or `cls`) 
parameter, and presumably most such extension methods will actually act 
on that self parameter in some way.

There is your "bundling of data with the methods that operate on that 
data", as required :-)

The fact that the methods happen to be written in an separate file, and 
(in some sense) added to the class as extension methods, is neither here 
nor there. While we *could* write an extension method that totally 
ignored `self` and instead operated entirely on global variables, most 
people won't -- and besides, we can already do that with regular 
methods.

class Weird:
def method(self, arg):
global data, more_data, unbundled_data, extra_data
del self  # don't need it, don't want it
do_stuff_with(data, more_data, unbundled_data, extra_data)

So the point is that extension methods are no less object-orientey than 
regular methods. They ought to behave just like regular methods with 
respect to encapsulation, namespacing, inheritance etc, modulo any minor 
and necessary differences.

E.g. in C# extension methods can only extend a class, not override an 
existing method.

[...]
> I don't think it's safer necessarily. With this proposal, we have the
> notion that obj.method() can mean two completely different things *at
> the same time* and *on the same object* depending on how you refactor
> the code.
> 
> # file1.py
> from file2 import func
> # and apply some extension methods
> def spamify(obj):
> print(obj.method())
> print(func(obj))
> 
> # file2.py
> def func(obj):
> return obj.method()

Yes, we can write non-obvious code in any language, using all sorts of 
"confusing" techniques, especially when you do stuff dynamically.

class K:
def __getattr__(self, attrname):
if attrname == 'method':
if __name__ == '__main__':
raise AttributeError
return something()

Regarding your example, you're only confused because you haven't take on 
board the fact that extension methods aren't interpreter-global, 
just module-global. Because it's new and unfamiliar. But we can do 
exactly the same thing, right now, with functions instead of methods, 
and you will find it trivially easy to diagnose the fault:

# file1.py
from file2 import func
# instead of "applying an extension method" from elsewhere,
# import a function from elsewhere
from extra_functions import g
def spamify(obj):
 print(g(obj)) # works fine
 print(func(obj))  # fails

# file2.py
def func(obj):
return g(obj)  # NameError

This example looks easy and not the least bit scary to you because 
you've been using Python for a while and it has become second nature to 
you. But you might remember back when you were a n00b, it probably 
confused you: why doesn't `g(obj)` work when you imported it? How 
weird and confusing! What do you mean, if I want to use g, I have to 
import it in each and every module where I want to use it? That's just 
dumb. Importing g once should make it available EVERYWHERE, right?

Been there, done that.

You learned about modules and namespaces, and why Python's design is 
*safer and better* than a single interpreter-global namespace, and now 
that doesn't confuse you one bit.

And if you were using Kotlin, or C#, or Swift, or any one of a number of 
other languages with extension methods, you would likewise learn that 
extension methods work in a similar fashion. Why does obj.method raise 
AttributeError from file2? *Obviously* its because you neglected to 
"apply the extension method", duh.

That's as obvious as neglecting to import something and getting a 
NameError. Maybe even more obvious, if your IDE or linter knows about 
extension methods.

And its *safer and better* than monkey-patching.

We have two people in this thread who know Kotlin and C#, at least one 
of them is a fan of the technique. Why don't we ask them how often this 
sort of error is a problem within the Kotlin and C# communities?

> Is that really beneficial? All I'm seeing is myriad ways for things to
> get confusing - just like in the very worst examples of
> monkey-patching.
> 
> And yes, I have some experience of monkey-patching in Python,
> including a situation where I couldn't just "import A; import B", I
> had to first import a helper for A, then import B, and finally import
> A, because there were conflicting monkey-patches. But here's the
> thing: extension

[Python-ideas] Re: Extension methods in Python

On Tue, Jun 22, 2021 at 8:01 PM Steven D'Aprano  wrote:
> The whole point of extension methods is that once the caller opts in to
> use them, method look ups (and that includes hasattr and getattr) must
> work as if the extension methods **are in place**.
>
> The must be no semantic difference between:
>
> obj.method(arg)
>
> and
>
> getattr(obj, 'method')(arg)
>
> regardless of whether `method` is a regular method or an extension
> method.
>

And this is a problem. How is getattr defined? Is it counted as being
in the current module? If it is, then has getattr magically become
part of the module it's called from? Or do ALL lookups depend on where
the function was called, rather than where it's defined?

If 'method' is an extension method, where exactly is it visible?

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/5YRFIE3O42WN3TCLLLR2NR5M6DZ2SUJU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

I'm sorry Soni, I don't understand what you are arguing here. See below.

On Mon, Jun 21, 2021 at 10:09:17PM -0300, Soni L. wrote:
> 
> 
> On 2021-06-21 9:39 p.m., Steven D'Aprano wrote:
> 
> 
> >
> > Fourth step is that you go ahead and use lists as normal. Whether you 
> > use getattr or dot syntax, any extension methods defined in spam.py will 
> > show up, as if they were actual list methods.
> >
> > hasattr([], 'head')  # returns True
> > list.tail  # returns the spam.tail function object (unbound method)
> >
> > They're not monkey-patched: other modules don't see that.
> >
> >
> 
> Python is a dynamic language. Maybe you're using hasattr/getattr to
> forward something from A to B. If "other modules don't see that" then
> this must work as if there were no extension methods in place.

What's "forward something from A to B" mean? What are A and B?

If "this" (method lookups) "must work as if there were no extension 
methods in place" then extension methods are a no-op and are pointless. 
You write an extension method, register it as applying to a type, the 
caller opts-in to use it, and then... nothing happens, because it "must 
work as if there were no extension methods in place".

Surely that isn't what you actually want to happen. But if not, I have 
no idea what you mean.

The whole point of extension methods is that once the caller opts in to 
use them, method look ups (and that includes hasattr and getattr) must 
work as if the extension methods **are in place**.

The must be no semantic difference between:

obj.method(arg)

and

getattr(obj, 'method')(arg)

regardless of whether `method` is a regular method or an extension 
method.

> So you
> actually wouldn't want the local load_attr override to apply to those.
> If you did... well, just call the override directly.

I have no idea what that means. What is "the local load_attr override"?

> If the override was
> called __opcode_load_attr_impl__ you'd just call
> __opcode_load_attr_impl__ directly instead of going through getattr.

As a general rule, you should not be calling dunders directly.

You seem to have missed the point that extension methods are intended as 
a mechanism to **extend a type** by giving it new methods on an opt-in 
basis. I want to call them "virtual methods" except that would add 
confusion regarding virtual subclasses and ABCs etc.

Maybe you need to read the Kotlin docs:

https://kotlinlang.org/docs/extensions.html

and the C# docs:

https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/classes-and-structs/extension-methods

Wikipedia also has a broad overview from a language-agnostic 
perspective:

https://en.wikipedia.org/wiki/Extension_method

Here's an example in TypeScript and Javascript:

https://putridparrot.com/blog/extension-methods-in-typescript/

In particular note these comments:

# Kotlin
"Such functions are available for calling in the usual way as if they 
were methods of the original class."

# C#
"Extension methods are only in scope when you explicitly import the 
namespace into your source code with a using directive."

Both C# and Kotlin are statically typed languages, and Python is not, 
but we ought to aim to minimise the differences in semantics. Aside from 
extension methods being resolved at runtime instead of at compile time, 
the behaviour ought to be as close as possible.

Just as single dispatch in Python is resolved dynamically, but aims to 
behave as close as possible to single dispatch in statically typed 
languages.

Another important quote:

"Because extension methods are called by using instance method syntax, 
no special knowledge is required to use them from client code. To enable 
extension methods for a particular type, just add a `using` directive 
for the namespace in which the methods are defined."

"No special knowledge is required" implies that, aside from the opt-in 
step itself, extension methods must behave precisely the same as regular 
methods. That means they will be accessible as bound methods on the 
instance:

obj.method

and unbound methods (functions) on the type:

type(obj).method

and using dynamic lookup:

getattr(obj, 'method')

and they will fully participate in inheritance heirarchies if you have 
opted in to use them.

> There needs to be an escape hatch for this.

The escape hatch is to *not* opt-in to the extension method. If the 
caller doesn't opt-in, they don't get the extension methods.

That is the critical difference between extension methods and monkey- 
patching the type. Monkey-patching effects everyone. Extension methods 
have to be opt-in.

> Or you *could* have getattr be special (called by load_attr) and
> overridable, and builtins.getattr be the escape hatch, but nobody would
> like that.

Huh? Unless you have shadowed getattr with a module-level function, 
getattr *is* builtins.getattr.

-- 
Steve
___
Python-ideas mailing list --

[Python-ideas] Re: Extension methods in Python

On Tue, Jun 22, 2021 at 10:40 AM Steven D'Aprano  wrote:
> > True, all true, but considering that this is *not* actually part of
> > the class, some of that doesn't really apply. For instance, is it
> > really encapsulation? What does that word even mean when you're
> > injecting methods in from the outside?
>
> Sure it's encapsulation. We can already do this with non-builtin
> classes:
>
> class SpammySpam:
> def spam(self, arg):
> ...
>
> from another_module import eggy_method
>
> def aardvarks(self, foo, bar):
> ...
>
> SpammySpam.aardvarks = aardvarks
>
>
> The fact that two of those methods have source code that wasn't indented
> under the class statement is neither here nor there. Even the fact that
> eggy_method was defined in another module is irrelevant. What matters is
> that once I've put the class together, all three methods are fully
> encapsulated into the SpammySpam class, and other classes can define
> different methods with the same name.
>
> Encapsulation is less about where you write the source code, and more
> about the fact that I can have
>
> SpammySpam().spam
>
> and
>
> Advertising().spam
>
> without the two spam methods stomping on each other.

Hmm, that's not what I'd usually understand "encapsulation" to mean.
That's what would normally be called "namespacing".

"... encapsulation refers to the bundling of data with the methods
that operate on that data, or the restricting of direct access to some
of an object's components."
https://en.wikipedia.org/wiki/Encapsulation_(computer_programming)

> Extension methods let us extend classes without the downsides of
> monkey-patching. Extension methods are completely opt-in while
> monkey-patching is mandatory for everyone. If we could only have one,
> extension methods would clearly be the safer choice.

I don't think it's safer necessarily. With this proposal, we have the
notion that obj.method() can mean two completely different things *at
the same time* and *on the same object* depending on how you refactor
the code.

# file1.py
from file2 import func
# and apply some extension methods
def spamify(obj):
print(obj.method())
print(func(obj))

# file2.py
def func(obj):
return obj.method()

Is that really beneficial? All I'm seeing is myriad ways for things to
get confusing - just like in the very worst examples of
monkey-patching.

And yes, I have some experience of monkey-patching in Python,
including a situation where I couldn't just "import A; import B", I
had to first import a helper for A, then import B, and finally import
A, because there were conflicting monkey-patches. But here's the
thing: extension methods (by this pattern) would not have solved it,
because the entire *point* of the monkey-patch was to fix an
incompatibility. So it HAD to apply to a completely different module.

That's why, despite its problems, I still think that monkey-patching
is the cleaner option. It prevents objects from becoming
context-dependent.

> We don't make heavy use of monkey-patching, not because it isn't a
> useful technique, but because:
>
> - unlike Ruby, we can't extend builtins without subclassing;
>
> - we're very aware that monkey-patching is a massively powerful
>   technique with huge foot-gun potential;
>
> - and most of all, the Python community is a hell of a lot more
>   conservative than Ruby.

And the Ruby community is starting to see the risks of
monkey-patching. (There's a quiz floating around the internet - "Ruby
or Rails?" - that brings into sharp relief the incredibly far-reaching
effects of using Rails. It includes quite a few methods on builtin
objects.) So I am absolutely fine with being conservative.

We have import hooks and MacroPy. Does anyone use them in production?
I certainly don't - not because I can't, but because I won't without a
VERY good reason.

> Even basic techniques intentionally added to the language (like being
> able to attach attributes onto function objects) are often looked at as
> if they were the worst kind of obfuscated self-modifying code. Even when
> those same techniques are used in the stdlib people are still reluctant
> to use it. As a community, we're like cats: anything new and different
> scares us, even if its actually been used for 30 years.
>
> We're a risk-adverse community.
>

I'm not sure why attaching attributes to functions is frowned upon;
I'd personally make very good use of this for static variables, if
only I could dependably refer to "this_function". But risk-averse is
definitely preferable to the alternative. It means that Python is a
language that can be learned as a whole, rather than being fragmented
into "the NumPy flavour of Python" and "the Flask flavour of Python"
and so on, with their own changes to the fabric of the language.

So far, I'm not seeing anything in extension methods to make me want
to change that stance.

ChrisA
___
Python-ideas mailing list

[Python-ideas] Re: Extension methods in Python

On 2021-06-21 9:39 p.m., Steven D'Aprano wrote:

>
> Fourth step is that you go ahead and use lists as normal. Whether you 
> use getattr or dot syntax, any extension methods defined in spam.py will 
> show up, as if they were actual list methods.
>
> hasattr([], 'head')  # returns True
> list.tail  # returns the spam.tail function object (unbound method)
>
> They're not monkey-patched: other modules don't see that.
>
>

Python is a dynamic language. Maybe you're using hasattr/getattr to
forward something from A to B. If "other modules don't see that" then
this must work as if there were no extension methods in place. So you
actually wouldn't want the local load_attr override to apply to those.
If you did... well, just call the override directly. If the override was
called __opcode_load_attr_impl__ you'd just call
__opcode_load_attr_impl__ directly instead of going through getattr.
There needs to be an escape hatch for this.

Or you *could* have getattr be special (called by load_attr) and
overridable, and builtins.getattr be the escape hatch, but nobody would
like that.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/VCNS6AEZ7NDMUOD2AEKOWWGH7XLTIRGP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python



On 2021-06-21 8:57 p.m., Thomas Grainger wrote:
> It seems odd that it would be per module and not per scope?

It's unusual to import things at the scope level. Usually things get
imported at the module level, so, using module language doesn't seem
that bad.

But yes, it's per scope, but in practice it's per module because nobody
would actually use this per scope even tho they could. :p

>
> On Tue, 22 Jun 2021, 00:55 Soni L.,  > wrote:
>
>
>
> On 2021-06-21 8:42 p.m., Steven D'Aprano wrote:
> > On Mon, Jun 21, 2021 at 02:54:52PM -0300, Soni L. wrote:
> >
> > > Quite the opposite. You ask the local module (the one that the
> code was
> > > compiled in), and the module decides whether/when to ask the
> object itself.
> > >
> > > In other words, every
> > >
> > > foo.bar
> > >
> > > would be sugar for
> > >
> > > __getattr__(foo, "bar")
> > >
> > > (where __getattr__ defaults to builtins.getattr) instead of
> being sugar for
> > >
> > > (foo, "bar")
> >
> > All you've done here is push the problem further along -- how does
> > `__getattr__` (`__getattribute__`?) decide what to do?
> >
> > * Why is this extension-aware version per module, instead of a
> builtin?
> >
> > * Does that mean the caller has to write it in every module they
> want to
> >   make use of extensions?
> >
> > * Why do we need a second attribute lookup mechanism instead of
> having
> >   the existing mechanism do the work?
> >
> > * And most problematic, if we have an extension method on a
> type, the
> >   builtin getattr ought to pick it up.
> >
> >
> > By the way, per-module `__getattr__` already has a meaning, so
> this name
> > won't fly.
> >
> > https://www.python.org/dev/peps/pep-0562/
> 
> >
> >
>
> No, you got it wrong. Extension methods don't go *on* the type being
> extended. Indeed, that's how they differ from monkeypatching.
>
> The whole point of extension methods *is* to be per-module. You could
> shove it in the existing attribute lookup mechanism (aka the
> builtins.getattr) but that involves runtime reflection, whereas
> making a
> new, per-module attribute lookup mechanism specifically designed to
> support a per-module feature would be a lot better.
>
> Extension methods *do not go on the type*.
>
> And sure, let's call it __opcode_load_attr_impl__ instead. Sounds
> good?
> ___
> Python-ideas mailing list -- python-ideas@python.org
> 
> To unsubscribe send an email to python-ideas-le...@python.org
> 
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> 
> Message archived at
> 
> https://mail.python.org/archives/list/python-ideas@python.org/message/MTOP22VK2ZC3GWCQHU5RDFVIT5AAR4DW/
> 
> 
> Code of Conduct: http://python.org/psf/codeofconduct/
> 
>

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/MBHRH4FTGN7NCCHEHU3HVE4UPDDWHHHY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

2021-06-21 Thread Steven D'Aprano

On Tue, Jun 22, 2021 at 03:56:00AM +1000, Chris Angelico wrote:

> I'm actually not concerned so much with the performance as the
> confusion. What exactly does the registration apply to?

Good question.

Extension methods have four steps:

- you write a method;

- declare which class it extends;

- the caller declares that they want to use extensions;

- and they get looked up at runtime (because we can't do static 
  lookups).

The first two can go together. I might write a module "spam.py". 
Borrowing a mix of Kotlin and/or C# syntax, maybe I write:

def list.head(self, arg):
...

def list.tail(self, arg):
...

or maybe we have a decorator:

@extends(list)
def head(self, arg):
...

The third step happens at the caller site. Using the C# keyword, you 
might write this in your module "stuff.py":

uses spam

or maybe there's a way to do it with the import keyword:

# could be confused for `from spam import extensions`?
import extensions from spam

from functools import extension_methods
import spam
extension_methods.load_from(spam)

whatever it takes. Depends on how much of this needs to be baked into 
the interpreter.

Fourth step is that you go ahead and use lists as normal. Whether you 
use getattr or dot syntax, any extension methods defined in spam.py will 
show up, as if they were actual list methods.

hasattr([], 'head')  # returns True
list.tail  # returns the spam.tail function object (unbound method)

They're not monkey-patched: other modules don't see that.

> And suppose
> you have a series of extension methods that you want to make use of in
> several modules in your project, how can you refactor a bunch of
> method registration calls so you can apply them equally in multiple
> modules? We don't need an implementation yet - but we need clear
> semantics.

I put the extension modules in one library. That may not literally 
require me to put their definitions in a single .py file, I should be 
able to use a package and import extension methods from modules the same 
as any other object. But for ease of use for the caller, I probably want 
to make all my related extension methods usable from a single place.

Then you, the caller, import/use them from each of your modules where 
you want to use them:

# stuff.py
uses spam

# things.py
uses spam

And in modules where you don't want to use them, you just don't use 
them.

[...]
> True, all true, but considering that this is *not* actually part of
> the class, some of that doesn't really apply. For instance, is it
> really encapsulation? What does that word even mean when you're
> injecting methods in from the outside?

Sure it's encapsulation. We can already do this with non-builtin 
classes:

class SpammySpam:
def spam(self, arg):
...

from another_module import eggy_method

def aardvarks(self, foo, bar):
...

SpammySpam.aardvarks = aardvarks

The fact that two of those methods have source code that wasn't indented 
under the class statement is neither here nor there. Even the fact that 
eggy_method was defined in another module is irrelevant. What matters is 
that once I've put the class together, all three methods are fully 
encapsulated into the SpammySpam class, and other classes can define 
different methods with the same name.

Encapsulation is less about where you write the source code, and more 
about the fact that I can have

SpammySpam().spam

and 

Advertising().spam

without the two spam methods stomping on each other.

[...]
> And that's a very very big "if". Monkey-patching can be used for
> unittest mocking, but that won't work here. Monkey-patching can be
> used to fix bugs in someone else's code, but that only works here if
> *your* code is in a single module, or you reapply the monkey-patch in
> every module. I'm really not seeing a lot of value in the proposal.

LINQ is a pretty major part of the C# ecosystem. I think that 
proves the value of extension methods :-)

I know we're not really comparing apples with apples, Python's 
trade-offs are not the same as C#'s trade-offs. But Ruby is a dynamic 
language like Python, and they use monkey-patching all the time, proving 
the value of being able to extend classes without subclassing them.

Extension methods let us extend classes without the downsides of 
monkey-patching. Extension methods are completely opt-in while 
monkey-patching is mandatory for everyone. If we could only have one, 
extension methods would clearly be the safer choice.

We don't make heavy use of monkey-patching, not because it isn't a 
useful technique, but because:

- unlike Ruby, we can't extend builtins without subclassing;

- we're very aware that monkey-patching is a massively powerful 
  technique with huge foot-gun potential;

- and most of all, the Python community is a hell of a lot more 
  conservative than Ruby.

Even basic techniques intentionally

[Python-ideas] Re: Extension methods in Python

2021-06-21 Thread Thomas Grainger

It seems odd that it would be per module and not per scope?

On Tue, 22 Jun 2021, 00:55 Soni L.,  wrote:

>
>
> On 2021-06-21 8:42 p.m., Steven D'Aprano wrote:
> > On Mon, Jun 21, 2021 at 02:54:52PM -0300, Soni L. wrote:
> >
> > > Quite the opposite. You ask the local module (the one that the code was
> > > compiled in), and the module decides whether/when to ask the object
> itself.
> > >
> > > In other words, every
> > >
> > > foo.bar
> > >
> > > would be sugar for
> > >
> > > __getattr__(foo, "bar")
> > >
> > > (where __getattr__ defaults to builtins.getattr) instead of being
> sugar for
> > >
> > > (foo, "bar")
> >
> > All you've done here is push the problem further along -- how does
> > `__getattr__` (`__getattribute__`?) decide what to do?
> >
> > * Why is this extension-aware version per module, instead of a builtin?
> >
> > * Does that mean the caller has to write it in every module they want to
> >   make use of extensions?
> >
> > * Why do we need a second attribute lookup mechanism instead of having
> >   the existing mechanism do the work?
> >
> > * And most problematic, if we have an extension method on a type, the
> >   builtin getattr ought to pick it up.
> >
> >
> > By the way, per-module `__getattr__` already has a meaning, so this name
> > won't fly.
> >
> > https://www.python.org/dev/peps/pep-0562/
> >
> >
>
> No, you got it wrong. Extension methods don't go *on* the type being
> extended. Indeed, that's how they differ from monkeypatching.
>
> The whole point of extension methods *is* to be per-module. You could
> shove it in the existing attribute lookup mechanism (aka the
> builtins.getattr) but that involves runtime reflection, whereas making a
> new, per-module attribute lookup mechanism specifically designed to
> support a per-module feature would be a lot better.
>
> Extension methods *do not go on the type*.
>
> And sure, let's call it __opcode_load_attr_impl__ instead. Sounds good?
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/MTOP22VK2ZC3GWCQHU5RDFVIT5AAR4DW/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/OQOJ3LZLCYKCYJW2ZTHQ2OQULKDFPWVC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

On 2021-06-21 8:42 p.m., Steven D'Aprano wrote:
> On Mon, Jun 21, 2021 at 02:54:52PM -0300, Soni L. wrote:
>
> > Quite the opposite. You ask the local module (the one that the code was
> > compiled in), and the module decides whether/when to ask the object itself.
> > 
> > In other words, every
> > 
> > foo.bar
> > 
> > would be sugar for
> > 
> > __getattr__(foo, "bar")
> > 
> > (where __getattr__ defaults to builtins.getattr) instead of being sugar for
> > 
> > (foo, "bar")
>
> All you've done here is push the problem further along -- how does 
> `__getattr__` (`__getattribute__`?) decide what to do?
>
> * Why is this extension-aware version per module, instead of a builtin?
>
> * Does that mean the caller has to write it in every module they want to
>   make use of extensions?
>
> * Why do we need a second attribute lookup mechanism instead of having
>   the existing mechanism do the work?
>
> * And most problematic, if we have an extension method on a type, the 
>   builtin getattr ought to pick it up.
>
>
> By the way, per-module `__getattr__` already has a meaning, so this name 
> won't fly.
>
> https://www.python.org/dev/peps/pep-0562/
>
>

No, you got it wrong. Extension methods don't go *on* the type being
extended. Indeed, that's how they differ from monkeypatching.

The whole point of extension methods *is* to be per-module. You could
shove it in the existing attribute lookup mechanism (aka the
builtins.getattr) but that involves runtime reflection, whereas making a
new, per-module attribute lookup mechanism specifically designed to
support a per-module feature would be a lot better.

Extension methods *do not go on the type*.

And sure, let's call it __opcode_load_attr_impl__ instead. Sounds good?
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/MTOP22VK2ZC3GWCQHU5RDFVIT5AAR4DW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

2021-06-21 Thread Steven D'Aprano

On Mon, Jun 21, 2021 at 02:54:52PM -0300, Soni L. wrote:

> Quite the opposite. You ask the local module (the one that the code was
> compiled in), and the module decides whether/when to ask the object itself.
> 
> In other words, every
> 
> foo.bar
> 
> would be sugar for
> 
> __getattr__(foo, "bar")
> 
> (where __getattr__ defaults to builtins.getattr) instead of being sugar for
> 
> (foo, "bar")

All you've done here is push the problem further along -- how does 
`__getattr__` (`__getattribute__`?) decide what to do?

* Why is this extension-aware version per module, instead of a builtin?

* Does that mean the caller has to write it in every module they want to
  make use of extensions?

* Why do we need a second attribute lookup mechanism instead of having
  the existing mechanism do the work?

* And most problematic, if we have an extension method on a type, the 
  builtin getattr ought to pick it up.

By the way, per-module `__getattr__` already has a meaning, so this name 
won't fly.

https://www.python.org/dev/peps/pep-0562/

-- 
Steve
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/BK5VM4IVWW7UFQ3D3RK5LBMKQOYBNIDV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

2021-06-21 Thread André Roberge

On Mon, Jun 21, 2021 at 3:28 PM Soni L.  wrote:

>
>
>
> For exec/eval you just pass in the locals:
>
> exec(foo, globals(), locals())
>
> because this __getattribute__ is just a local like any other.
>
> As for each module, you'd import them. But not quite with "import":
>
> import extension_methods # magic module, probably provides an
> @extend(class_) e.g. @extend(list)
> import shallow_flatten
> import deep_flatten
> __getattribute__ = extension_methods.getattribute(
>   shallow_flatten.flatten, # uses __name__
>   deepflatten=deep_flatten.flatten, # name override
>   __getattribute__=__getattribute__, # optional, defaults to
> builtins.getattr
> )
>
> This would have to be done for each .py that wants to use the extension
> methods.
>

I bet you that you could already do this today with a custom import hook.
If you want to "easily" experiment with this, I would suggest having a look
at https://aroberge.github.io/ideas/docs/html/index.html
which likely has all the basic scaffolding that you would need.

André Roberge



> >
> > ChrisA
> > ___
> > Python-ideas mailing list -- python-ideas@python.org
> > To unsubscribe send an email to python-ideas-le...@python.org
> > https://mail.python.org/mailman3/lists/python-ideas.python.org/
> > Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/3DL46SIS2A7D3W4FNSTH37O6VDJJB2ZP/
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/ROZN62HQYL6X5LCJ7J5DNOC7W3DBGYHF/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/KLEBIWRLYLLXHTJT264OPMSVY3CBUG37/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

On 2021-06-21 3:01 p.m., Chris Angelico wrote:
> Thanks for clarifying. This doesn't change the problem though - it
> just changes where the issue shows up. (BTW, what you're describing is
> closer to __getattribute__ than it is to __getattr__, so if you're
> proposing this as the semantics, I strongly recommend going with that
> name.)

Oh, sorry, thought __getattribute__ was the fallback and __getattr__ the
one always called, what with getattr -> __getattr__. But yeah,
__getattribute__ then.

>
> So, here's the question - a clarification of what I asked vaguely up
> above. Suppose you have a bunch of these extension methods, and a
> large project. How are you going to register the right extension
> methods in the right modules within your project? You're binding the
> functionality to the module in which the code was compiled, which will
> make exec/eval basically unable to use them, and that means you'll
> need some way to set them in each module, or to import the setting
> from somewhere else. How do you propose doing this?

For exec/eval you just pass in the locals:

exec(foo, globals(), locals())

because this __getattribute__ is just a local like any other.

As for each module, you'd import them. But not quite with "import":

import extension_methods # magic module, probably provides an
@extend(class_) e.g. @extend(list)
import shallow_flatten
import deep_flatten
__getattribute__ = extension_methods.getattribute(
  shallow_flatten.flatten, # uses __name__
  deepflatten=deep_flatten.flatten, # name override
  __getattribute__=__getattribute__, # optional, defaults to
builtins.getattr
)

This would have to be done for each .py that wants to use the extension
methods.

>
> ChrisA
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/3DL46SIS2A7D3W4FNSTH37O6VDJJB2ZP/
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/ROZN62HQYL6X5LCJ7J5DNOC7W3DBGYHF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

2021-06-21 Thread Chris Angelico

On Tue, Jun 22, 2021 at 3:55 AM Soni L.  wrote:
>
>
>
> On 2021-06-21 12:49 p.m., Chris Angelico wrote:
> > On Tue, Jun 22, 2021 at 1:44 AM Soni L.  wrote:
> > >
> > >
> > >
> > > On 2021-06-21 12:26 p.m., Stephen J. Turnbull wrote:
> > > > Soni L. writes:
> > > >
> > > >  > The trick to extension methods is that they're only available when 
> > > > you
> > > >  > explicitly use them.
> > > >
> > > > What does "explicitly use them" mean?  How does this help avoid the
> > > > kinds of problems we know that monkey-patching causes?
> > >
> > > Monkey-patching:
> > >
> > > ```py mod1.py
> > > import foo
> > >
> > > foo.Bar.monkeymethod = ...
> > > ```
> > >
> > > ```py mod2.py
> > > import foo
> > >
> > > foo.Bar.monkeymethod = ...
> > > ```
> > >
> > > "Extension methods":
> > >
> > > ```py mod1.py
> > > import foo
> > >
> > > def __getattr__(o, attr):
> > >   if isinstance(o, foo.Bar) and attr == "monkeymethod":
> > > return ...
> > >   return getattr(o, attr)
> > > ```
> > >
> > > ```py mod2.py
> > > import foo
> > >
> > > def __getattr__(o, attr):
> > >   if isinstance(o, foo.Bar) and attr == "monkeymethod":
> > > return ...
> > >   return getattr(o, attr)
> > > ```
> > >
> > > Note how the former changes foo.Bar, whereas the latter only changes the
> > > module's own __getattr__. You can't have conflicts with the latter.
> > > (Also note that this "module's own __getattr__" doesn't provide
> > > extension methods by itself, but can be used as a mechanism to implement
> > > extension methods.)
> >
> > So what you're saying is that, in effect, every attribute lookup has
> > to first ask the object itself, and then ask the module? Which module?
> > The one that the code was compiled in? The one that is currently
> > running? Both?
> >
> > And how is this better than just using a plain ordinary function? Not
> > everything has to be a method.
>
> Quite the opposite. You ask the local module (the one that the code was
> compiled in), and the module decides whether/when to ask the object itself.
>
> In other words, every
>
> foo.bar
>
> would be sugar for
>
> __getattr__(foo, "bar")
>
> (where __getattr__ defaults to builtins.getattr) instead of being sugar for
>
> (foo, "bar")
>
> (where <> is used to indicate that it doesn't quite desugar that way -
> otherwise you'd need to recursively desugar it to
> builtins.getattr(builtins, "getattr") which uh, doesn't work.)
>

Thanks for clarifying. This doesn't change the problem though - it
just changes where the issue shows up. (BTW, what you're describing is
closer to __getattribute__ than it is to __getattr__, so if you're
proposing this as the semantics, I strongly recommend going with that
name.)

So, here's the question - a clarification of what I asked vaguely up
above. Suppose you have a bunch of these extension methods, and a
large project. How are you going to register the right extension
methods in the right modules within your project? You're binding the
functionality to the module in which the code was compiled, which will
make exec/eval basically unable to use them, and that means you'll
need some way to set them in each module, or to import the setting
from somewhere else. How do you propose doing this?

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/3DL46SIS2A7D3W4FNSTH37O6VDJJB2ZP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python




On 2021-06-21 12:49 p.m., Chris Angelico wrote:
> On Tue, Jun 22, 2021 at 1:44 AM Soni L.  wrote:
> >
> >
> >
> > On 2021-06-21 12:26 p.m., Stephen J. Turnbull wrote:
> > > Soni L. writes:
> > >
> > >  > The trick to extension methods is that they're only available when you
> > >  > explicitly use them.
> > >
> > > What does "explicitly use them" mean?  How does this help avoid the
> > > kinds of problems we know that monkey-patching causes?
> >
> > Monkey-patching:
> >
> > ```py mod1.py
> > import foo
> >
> > foo.Bar.monkeymethod = ...
> > ```
> >
> > ```py mod2.py
> > import foo
> >
> > foo.Bar.monkeymethod = ...
> > ```
> >
> > "Extension methods":
> >
> > ```py mod1.py
> > import foo
> >
> > def __getattr__(o, attr):
> >   if isinstance(o, foo.Bar) and attr == "monkeymethod":
> > return ...
> >   return getattr(o, attr)
> > ```
> >
> > ```py mod2.py
> > import foo
> >
> > def __getattr__(o, attr):
> >   if isinstance(o, foo.Bar) and attr == "monkeymethod":
> > return ...
> >   return getattr(o, attr)
> > ```
> >
> > Note how the former changes foo.Bar, whereas the latter only changes the
> > module's own __getattr__. You can't have conflicts with the latter.
> > (Also note that this "module's own __getattr__" doesn't provide
> > extension methods by itself, but can be used as a mechanism to implement
> > extension methods.)
>
> So what you're saying is that, in effect, every attribute lookup has
> to first ask the object itself, and then ask the module? Which module?
> The one that the code was compiled in? The one that is currently
> running? Both?
>
> And how is this better than just using a plain ordinary function? Not
> everything has to be a method.

Quite the opposite. You ask the local module (the one that the code was
compiled in), and the module decides whether/when to ask the object itself.

In other words, every

foo.bar

would be sugar for

__getattr__(foo, "bar")

(where __getattr__ defaults to builtins.getattr) instead of being sugar for

(foo, "bar")

(where <> is used to indicate that it doesn't quite desugar that way -
otherwise you'd need to recursively desugar it to
builtins.getattr(builtins, "getattr") which uh, doesn't work.)

>
> ChrisA
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-ideas@python.org/message/EMA2EBZLIR7DLNJAHWHURATJ4WVBOEUJ/
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/AGYFBB6XJ4V7ASID54PNGFZK74ZD4HWA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

2021-06-21 Thread Chris Angelico

On Tue, Jun 22, 2021 at 3:43 AM Steven D'Aprano  wrote:
>
> On Tue, Jun 22, 2021 at 01:49:56AM +1000, Chris Angelico wrote:
>
> > So what you're saying is that, in effect, every attribute lookup has
> > to first ask the object itself, and then ask the module? Which module?
> > The one that the code was compiled in? The one that is currently
> > running? Both?
>
> Mu.
>
> https://en.wikipedia.org/wiki/Mu_(negative)#%22Unasking%22_the_question
>
> We don't have an implementation yet, so it is too early to worry about
> precisely where you look up the extension methods, except that it is
> opt-in (so by default, there's no additional cost involved) and there
> must be *some* sort of registry *somewhere* that handles the mapping of
> extension methods to classes.
>
> We certainly don't want this to slow down *every method call*, but if it
> only slowed down method lookups a little bit when you actually used the
> feature, that might be acceptable. We already make use of lots of
> features which are slow as continental drift compared to C, because they
> add power to the language and are *fast enough*.
>
> E.g. name lookups are resolved at runtime, not compile-time; dynamic
> attribute lookups using gettattribute and getattr dunders; virtual
> subclasses; generic functions (functools.singledispatch); descriptors.

I'm actually not concerned so much with the performance as the
confusion. What exactly does the registration apply to? And suppose
you have a series of extension methods that you want to make use of in
several modules in your project, how can you refactor a bunch of
method registration calls so you can apply them equally in multiple
modules? We don't need an implementation yet - but we need clear
semantics.

> > And how is this better than just using a plain ordinary function? Not
> > everything has to be a method.
>
> You get method syntax, obj.method, which is nice but not essential.
>
> When a method is called from an instance, you know that the first
> parameter `self` has got to be the correct type, no type-checking is
> required. That's good. And the same would apply to extension methods.
>
> You get bound methods as first class values, which is useful.
>
> You get inheritance, which is powerful.
>
> And you get encapsulation, which is important.

True, all true, but considering that this is *not* actually part of
the class, some of that doesn't really apply. For instance, is it
really encapsulation? What does that word even mean when you're
injecting methods in from the outside?

> I think this is a Blub moment. We don't think it's useful because we
> have functions, and we're not Java, so "not everything needs to be a
> method". Sure, but methods are useful, and they do bring benefits that
> top-level functions don't have. (And vice versa of course.)
>
> We have staticmethod that allows us to write a "function" (-ish) but get
> the benefits of inheritance, encapsulation, and method syntax. This
> would be similar.
>
> We acknowledge that there are benefits to monkey-patching. But we can't
> monkey-patch builtins and we are (rightly) suspicious of those who use
> monkey-patching in production. And this is good. But this would give us
> the benefits of monkey-patching without the disadvantages.
>
> *If* we can agree on semantics and come up with a reasonable efficient
> implementation that doesn't slow down every method call.
>

And that's a very very big "if". Monkey-patching can be used for
unittest mocking, but that won't work here. Monkey-patching can be
used to fix bugs in someone else's code, but that only works here if
*your* code is in a single module, or you reapply the monkey-patch in
every module. I'm really not seeing a lot of value in the proposal.

Let's completely ignore the performance cost for the moment and just
try to figure out semantics, with it being actually useful and not
unwieldy.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/HGZ73H5YT3WSD6B26SWOMVYC6W6POGM2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

2021-06-21 Thread Steven D'Aprano

On Tue, Jun 22, 2021 at 01:49:56AM +1000, Chris Angelico wrote:

> So what you're saying is that, in effect, every attribute lookup has
> to first ask the object itself, and then ask the module? Which module?
> The one that the code was compiled in? The one that is currently
> running? Both?

Mu.

https://en.wikipedia.org/wiki/Mu_(negative)#%22Unasking%22_the_question

We don't have an implementation yet, so it is too early to worry about 
precisely where you look up the extension methods, except that it is 
opt-in (so by default, there's no additional cost involved) and there 
must be *some* sort of registry *somewhere* that handles the mapping of 
extension methods to classes.

We certainly don't want this to slow down *every method call*, but if it 
only slowed down method lookups a little bit when you actually used the 
feature, that might be acceptable. We already make use of lots of 
features which are slow as continental drift compared to C, because they 
add power to the language and are *fast enough*.

E.g. name lookups are resolved at runtime, not compile-time; dynamic 
attribute lookups using gettattribute and getattr dunders; virtual 
subclasses; generic functions (functools.singledispatch); descriptors.

> And how is this better than just using a plain ordinary function? Not
> everything has to be a method.

You get method syntax, obj.method, which is nice but not essential.

When a method is called from an instance, you know that the first 
parameter `self` has got to be the correct type, no type-checking is 
required. That's good. And the same would apply to extension methods.

You get bound methods as first class values, which is useful.

You get inheritance, which is powerful.

And you get encapsulation, which is important.

I think this is a Blub moment. We don't think it's useful because we 
have functions, and we're not Java, so "not everything needs to be a 
method". Sure, but methods are useful, and they do bring benefits that 
top-level functions don't have. (And vice versa of course.)

We have staticmethod that allows us to write a "function" (-ish) but get 
the benefits of inheritance, encapsulation, and method syntax. This 
would be similar.

We acknowledge that there are benefits to monkey-patching. But we can't 
monkey-patch builtins and we are (rightly) suspicious of those who use 
monkey-patching in production. And this is good. But this would give us 
the benefits of monkey-patching without the disadvantages.

*If* we can agree on semantics and come up with a reasonable efficient 
implementation that doesn't slow down every method call.

-- 
Steve
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/QCEC764ILFOUDWO5HXVKTTJSA6XIT6KI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python

2021-06-21 Thread Chris Angelico

On Tue, Jun 22, 2021 at 1:44 AM Soni L.  wrote:
>
>
>
> On 2021-06-21 12:26 p.m., Stephen J. Turnbull wrote:
> > Soni L. writes:
> >
> >  > The trick to extension methods is that they're only available when you
> >  > explicitly use them.
> >
> > What does "explicitly use them" mean?  How does this help avoid the
> > kinds of problems we know that monkey-patching causes?
>
> Monkey-patching:
>
> ```py mod1.py
> import foo
>
> foo.Bar.monkeymethod = ...
> ```
>
> ```py mod2.py
> import foo
>
> foo.Bar.monkeymethod = ...
> ```
>
> "Extension methods":
>
> ```py mod1.py
> import foo
>
> def __getattr__(o, attr):
>   if isinstance(o, foo.Bar) and attr == "monkeymethod":
> return ...
>   return getattr(o, attr)
> ```
>
> ```py mod2.py
> import foo
>
> def __getattr__(o, attr):
>   if isinstance(o, foo.Bar) and attr == "monkeymethod":
> return ...
>   return getattr(o, attr)
> ```
>
> Note how the former changes foo.Bar, whereas the latter only changes the
> module's own __getattr__. You can't have conflicts with the latter.
> (Also note that this "module's own __getattr__" doesn't provide
> extension methods by itself, but can be used as a mechanism to implement
> extension methods.)

So what you're saying is that, in effect, every attribute lookup has
to first ask the object itself, and then ask the module? Which module?
The one that the code was compiled in? The one that is currently
running? Both?

And how is this better than just using a plain ordinary function? Not
everything has to be a method.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/EMA2EBZLIR7DLNJAHWHURATJ4WVBOEUJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Extension methods in Python