[Python-ideas] Re: Add a "partial with placeholders" function to the functools module

[Dominik Vilsmeier]

An extension of `partial` itself would indeed be a clean solution (+ backwards 
compatibility), my concern was just that when most people are still using it in 
the "traditional" way (for example partializing via keyword), that the 
additional checks for `SKIP` introduce an unnecessary overhead for those cases. 
However by checking if `args` contain any placeholder in `__new__` and storing 
that information in a separate variable it should be possible to keep it to a 
minimum.

Regarding third-party implementations one aspect to consider is compatibility 
with `partial` itself, i.e. what should happen when further `partial`'izing a 
previously defined placeholder function (it would need to be a placeholder 
function again, otherwise the `Ellipsis` or `SKIP` doesn't make much sense).
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/VNMZY5XBVR6UTOP7VMIF4XTRMNZPWPSH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a "partial with placeholders" function to the functools module

[Dominik Vilsmeier]

Indeed it is important to not only consider the potential use cases for such a 
placeholder function, but to consider the cases that go beyond of what a lambda 
can do. A lambda is always a good option if it is used as a "single-serving" 
function, i.e. one that is only relevant locally where it is coded (and not 
stored somewhere). If on the other hand the function is to be reused or users 
are expected to interact with it, more clarity is needed (e.g. a descriptive 
`repr`). Of course in such cases a developer could always go with a proper 
function definition and even supply it with a doc string; it's just that for 
simple cases this feels a bit heavy.
In the end it would provide a concise solution for cases where (a) partializing 
via keyword is not possible (positional-only parameters now being official 
might give rise to new use cases as well) and (b) the function is not 
"single-service", i.e. it is retained in some way (where PEP 8 says "do not 
assign a lambda").
As an example say there's a 2D array (semantics like numpy) and we want to 
convert a flat index to 2D-index (numpy contains such functionality but let's 
say we're doing something more lightweight):

a = array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
convert_flat_index = placeholder(divmod, ..., a.shape[1])
assert a[convert_flat_index(4)] == 5
assert a[convert_flat_index(8)] == 9

The other option would be a full-fledged function definition (which seems a bit 
heavy):

def generate_index_converter(n):
def convert_flat_index(x):
return divmod(x, n)
return convert_flat_index
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/XMHDFQGN5KLUFP7KKO3CDMXEBAXS542L/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a "partial with placeholders" function to the functools module

[David Mertz]

On Sat, Jul 27, 2019 at 9:28 PM Steven D'Aprano  wrote:

> If you're going to add support for a placeholder, there is no need to
> create a new kind of partial function?
>

Exactly! This is just an enhanced functools.partial.  An enhancement that
has already been done in several 3rd party libraries, in fact.  I like the
ellipsis, which most 3rd parties use, but it could be configurable what the
sentinel is.  Maybe a default brand new `functools.SKIP` object, but with
the docs explaining the way to use the ellipsis for those 99% of users for
whom that wouldn't break anything.

Someone posted an example implementation that used a configurable
placeholder like this.


> > Sure we could also use a `lambda` instead
> I believe that partial is faster than lambda.
>

I don't really care about faster.  I think the "partial with placeholder"
is simply nicer to read for many cases.  Sure this is a toy, but:

>>> lastname = 'Mertz'
>>> greet_david = partial(print, ..., 'David', lastname)  # In Python 3.9
>>> greet_david('Verwelkoming')
Verwelkoming David Mertz

vs.

>>> greet_david = lambda greeting: print(greeting, "David", lastname)
>>> greet_david('Cześć')
Cześć David Mertz

OK, that's a bad example because we are using it for the side effect too,
but I think it illustrates how the more flexible partial() would look
better than lambda.


-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/XG27FN6BSHNJVTS5YRLEZBRICRZLGT43/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a "partial with placeholders" function to the functools module

[Andrew Barnert via Python-ideas]

On Jul 27, 2019, at 12:47, Anders Hovmöller  wrote:
> 
> 
>> On 27 Jul 2019, at 21:10, Andrew Barnert  wrote:
>> 
>> Many of the compelling examples for PEP 570 are good examples here. The 
>> following are all impossible intentionally, and for different reasons:
>> 
>>   skipper = partial(range, step=n)
>>   powbase = partial(pow, mod=base)
>>   clscheck = partial(isinstance, class_or_tuple=cls)
> 
> Those seem like great cases for changing the standard library to use normal 
> parameters! range and pow are both nicer with keyword arguments imo.

Look at the signatures for range and isinstance: range takes an optional 
argument on the left (or, if you prefer, it’s an overload of two different 
signatures), which can’t even be expressed in Python syntax; isinstance takes a 
class or a tuple of classes in the same parameter, so there’s no good name for 
it. There are a few other functions just like range, and a bunch like 
isinstance, and some other classes of functions with different kinds of weird 
signatures (although pow isn’t one of them).

You could argue that these are just bad designs, but better designs weren’t 
possible when they were added to Python. For example, anything designed to take 
1 or many classes since 2.2 would just use *classes instead of a single 
argument, but isinstance-style functions predate 2.2, so there was no *args. 
Changing those existing functions would break a lot of existing code. And if 
that wasn’t acceptable for 2.2 or even 3.0, it’s probably never going to be 
acceptable.

>> Also, even for cases like the OP’s where there’s no semantics reason the 
>> argument couldn’t have a keyword, there may still be other reasons. When 
>> argclinic was added in PEP 436, it was specifically argued that it shouldn’t 
>> be used as an opportunity to phase out positional-only params, in part 
>> because for many functions the performance cost of keyword processing 
>> outweighs the uncommon potential use of the keywords; IIRC, the OP’s 
>> specific method was even one of the core examples. And, as PEP 570 points 
>> out, METH_FASTCALL makes that potentially true for pure Python functions as 
>> well, to the extent that some other stdlib functions might actually want to 
>> lose their keyword args.
> 
> Well that sounds pretty terrible to me. I’ve tried to write calls with 
> keywords of many many functions in the standard library because it just isn’t 
> really readable with positional :(

I think many of the functions in the stdlib do get it wrong, for legacy reasons 
(nobody thought about the tradeoff consciously until the argclinic discussion, 
and also it was a pain to add keywords in C before argclinic—and many of the 
functions you use most often date back even farther, to before keywords 
existing at all), but there is an actual tradeoff, so it shouldn’t be 100% of 
functions take keywords. The two PEPs both make the case for that.

And nobody’s going to sweep through the stdlib evaluating every function. If 
you’ve got a group of functions that particularly annoys you, and can make the 
case that the arg parsing performance isn’t important for that function and the 
keywords are useful, you can file it on b.p.o. (ideally with a patch) and I’m 
sure someone will take a look.

Meanwhile, there are going to be functions that take positional arguments, 
whether for semantic reasons, for performance reasons, or for pure legacy 
issues. As long as any of those cases exist, the OP’s proposal makes sense, at 
least to me.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/VGA5FYQ4AIMINQ27Y5QUTYWXRFFKMXDW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a "partial with placeholders" function to the functools module

[Steven D'Aprano]

On Sat, Jul 27, 2019 at 12:47:39PM -, Dominik Vilsmeier wrote:

> I propose adding a function to `functools` that works with 
> placeholders and thus offers even greater flexibility. The Ellipsis 
> literal `...` seems a intuitive choice for that task. When eventually 
> calling such a "partial placeholder" object, it would fill in 
> placeholders from the left and add remaining `args` to the right. In 
> terms of implementation this can be realized as a subclass of 
> `partial` itself.

If you're going to add support for a placeholder, there is no need to 
create a new kind of partial function?

Support for a placeholder is completely backwards compatible, if you use 
a brand new special sentinel that doesn't yet exist.

# May want to give it a nicer repr, but this is the 
# minimal version that will work.
SKIP = object()

There's only one public function, ``partial``, regardless of whether the 
caller uses the sentinel or not:


def function(a, b, c, d): pass

# Existing use is unchanged:
from functools import partial
partial(function, 1, 2)  # Fill in a, b.


# New functionality is triggered by the use of the sentinel:
from functools import partial, SKIP
partial(function, SKIP, 1, SKIP, 2)  # Fill in b, d.


If you don't like the use of a named sentinel, we could still use 
Ellipsis, but that will break anyone's partial functions that happen to 
already use Ellipsis as a value. That will probably require a 
warning in 3.9 and the new functionality only gets added in 3.10.


> Sure we could also use a `lambda` instead

I believe that partial is faster than lambda.


[steve@ando cpython]$ ./python -m timeit \
-s "from functools import partial" \
-s "f = partial(lambda x,y: x+y, 1)" \
"f(100)"
10 loops, best of 5: 1.98 usec per loop


[steve@ando cpython]$ ./python -m timeit \
-s "g = lambda x,y: x+y" \
-s "f = lambda a: g(1, a)" \
"f(100)"
10 loops, best of 5: 2.44 usec per loop


To avoid the name lookup of "g", I tried this, but it was even slower:

[steve@ando cpython]$ ./python -m timeit \
-s "f = lambda a: (lambda x,y: x+y)(1, a)" \
"f(100)"
10 loops, best of 5: 3.4 usec per loop


(Not surprising, as we've changed a name lookup into creating a new 
function object.)

Obviously this example is simple enough that we don't need partial at 
all, but its just an illustration demonstrating that partial seems to 
have lower overhead than normal function objects.


-- 
Steven
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/3IEMJSDGPACW2OBPP4OPUCAM7Q77EE7N/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a "partial with placeholders" function to the functools module

[Guido van Rossum]

Before we go too far down this route, let's consider whether these can't be
solved with lambda rather than introducing new special cases to partial().

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him/his **(why is my pronoun here?)*

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/5Z7QB7RAWODXPNPF774YL4BNXOQRJU5G/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a "partial with placeholders" function to the functools module

[MRAB]

On 2019-07-27 20:10, Andrew Barnert via Python-ideas wrote:

On Jul 27, 2019, at 08:04, Anders Hovmöller  wrote:



On 27 Jul 2019, at 14:47, Dominik Vilsmeier  wrote:

currently, regarding positional arguments, `partial` gives us the option to 
partialize functions from the left. There's been some interest about 
partializing functions from the right instead (e.g. [SO post, 9k views, 39 
upvotes](https://stackoverflow.com/q/7811247/3767239)), especially w.r.t. the 
various `str` methods.


Do you have other examples? That (and most likely similar) examples are just that the standard library contains methods and functions that could be fixed to accept keyword arguments. This would be less confusing and more coherent. 


Many of the compelling examples for PEP 570 are good examples here. The 
following are all impossible intentionally, and for different reasons:

 skipper = partial(range, step=n)
 powbase = partial(pow, mod=base)
 clscheck = partial(isinstance, class_or_tuple=cls)

Plus, there’s also one very general example: anything that semantically has to 
take *args can’t be changed to use keywords:

 partial(format, 2=x)
 partial(map, 1=x, 2=y)
 partial(executor.submit, 1=arg)

And similarly for itertools.product, min/max, and everything that acts as a 
proxy like submit.


[snip]

I was thinking that 'partial' is like a function definition, so why not 
have a variation on the function definition:


def powbase(x, y) is pow(x, y, base)
def clscheck(obj) is isinstance(obj, cls)

It would capture the current reference of any name that's not passed via 
the parameter list, e.g. base and cls in the above examples.

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/PIVGTG455RCAKLKSZFY5PMZK7SQJZIMW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a "partial with placeholders" function to the functools module

[Anders Hovmöller]

> On 27 Jul 2019, at 21:10, Andrew Barnert  wrote:
> 
> On Jul 27, 2019, at 08:04, Anders Hovmöller  wrote:
>> 
>>> On 27 Jul 2019, at 14:47, Dominik Vilsmeier  
>>> wrote:
>>> 
>>> currently, regarding positional arguments, `partial` gives us the option to 
>>> partialize functions from the left. There's been some interest about 
>>> partializing functions from the right instead (e.g. [SO post, 9k views, 39 
>>> upvotes](https://stackoverflow.com/q/7811247/3767239)), especially w.r.t. 
>>> the various `str` methods.
>> 
>> Do you have other examples? That (and most likely similar) examples are just 
>> that the standard library contains methods and functions that could be fixed 
>> to accept keyword arguments. This would be less confusing and more coherent. 
> 
> Many of the compelling examples for PEP 570 are good examples here. The 
> following are all impossible intentionally, and for different reasons:
> 
>skipper = partial(range, step=n)
>powbase = partial(pow, mod=base)
>clscheck = partial(isinstance, class_or_tuple=cls)

Those seem like great cases for changing the standard library to use normal 
parameters! range and pow are both nicer with keyword arguments imo.

> Plus, there’s also one very general example: anything that semantically has 
> to take *args can’t be changed to use keywords:
> 
>partial(format, 2=x)
>partial(map, 1=x, 2=y)
>partial(executor.submit, 1=arg)

Ok now THAT is a compelling argument! 

> And similarly for itertools.product, min/max, and everything that acts as a 
> proxy like submit.
> 
> Also, even for cases like the OP’s where there’s no semantics reason the 
> argument couldn’t have a keyword, there may still be other reasons. When 
> argclinic was added in PEP 436, it was specifically argued that it shouldn’t 
> be used as an opportunity to phase out positional-only params, in part 
> because for many functions the performance cost of keyword processing 
> outweighs the uncommon potential use of the keywords; IIRC, the OP’s specific 
> method was even one of the core examples. And, as PEP 570 points out, 
> METH_FASTCALL makes that potentially true for pure Python functions as well, 
> to the extent that some other stdlib functions might actually want to lose 
> their keyword args.

Well that sounds pretty terrible to me. I’ve tried to write calls with keywords 
of many many functions in the standard library because it just isn’t really 
readable with positional :(

/ Anders
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/24W72RAQFPWSL7U7GMNMH6JLI3GTVYFK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a "partial with placeholders" function to the functools module

[Andrew Barnert via Python-ideas]

On Jul 27, 2019, at 08:04, Anders Hovmöller  wrote:
> 
>> On 27 Jul 2019, at 14:47, Dominik Vilsmeier  wrote:
>> 
>> currently, regarding positional arguments, `partial` gives us the option to 
>> partialize functions from the left. There's been some interest about 
>> partializing functions from the right instead (e.g. [SO post, 9k views, 39 
>> upvotes](https://stackoverflow.com/q/7811247/3767239)), especially w.r.t. 
>> the various `str` methods.
> 
> Do you have other examples? That (and most likely similar) examples are just 
> that the standard library contains methods and functions that could be fixed 
> to accept keyword arguments. This would be less confusing and more coherent. 

Many of the compelling examples for PEP 570 are good examples here. The 
following are all impossible intentionally, and for different reasons:

skipper = partial(range, step=n)
powbase = partial(pow, mod=base)
clscheck = partial(isinstance, class_or_tuple=cls)

Plus, there’s also one very general example: anything that semantically has to 
take *args can’t be changed to use keywords:

partial(format, 2=x)
partial(map, 1=x, 2=y)
partial(executor.submit, 1=arg)

And similarly for itertools.product, min/max, and everything that acts as a 
proxy like submit.

Also, even for cases like the OP’s where there’s no semantics reason the 
argument couldn’t have a keyword, there may still be other reasons. When 
argclinic was added in PEP 436, it was specifically argued that it shouldn’t be 
used as an opportunity to phase out positional-only params, in part because for 
many functions the performance cost of keyword processing outweighs the 
uncommon potential use of the keywords; IIRC, the OP’s specific method was even 
one of the core examples. And, as PEP 570 points out, METH_FASTCALL makes that 
potentially true for pure Python functions as well, to the extent that some 
other stdlib functions might actually want to lose their keyword args.

All that being said, this proposal seems like something that could easily be 
put on PyPI to see how much uptake it gets, instead of put immediately into the 
stdlib. I think people will find it useful. But maybe, say, funcy/toolz/etc. 
will borrow the idea and it’ll turn out that almost everyone who wants it 
already wants one of those libs anyway. Or maybe the bikeshedding potential 
will be higher than expected, and a variety of different modules that all 
handle the placeholders differently will compete, and it’ll turn out that 
everyone loves some different design that allows both single-arg and multi-arg 
placeholders 
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/AAA32XZ3O7HNESRDQ437OS35MPAFUPTT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a "partial with placeholders" function to the functools module

[Anders Hovmöller]

> On 27 Jul 2019, at 14:47, Dominik Vilsmeier  wrote:
> 
> Hello,
> 
> currently, regarding positional arguments, `partial` gives us the option to 
> partialize functions from the left. There's been some interest about 
> partializing functions from the right instead (e.g. [SO post, 9k views, 39 
> upvotes](https://stackoverflow.com/q/7811247/3767239)), especially w.r.t. the 
> various `str` methods.

Do you have other examples? That (and most likely similar) examples are just 
that the standard library contains methods and functions that could be fixed to 
accept keyword arguments. This would be less confusing and more coherent. 

/ Anders 
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/ZRAGJJF7TKRYNLHXBL7ZW6MZZTKFA4YN/
Code of Conduct: http://python.org/psf/codeofconduct/