[Python-ideas] Re: PEP 671 proof-of-concept implementation
I’m with Steven. On Fri, Oct 29, 2021 at 06:22 Chris Angelico wrote: > On Fri, Oct 29, 2021 at 11:52 PM Steven D'Aprano > wrote: > > > Except that that's still backward-incompatible, since None is a very > > > common value. > > > > How is it backwards incompatible? Any tool that looks at __defaults__ > > finds *exactly* what was there before: a tuple of default values, not a > > tuple of tuples (desc, value) or (value,) as in your implementation. > > It's fundamentally impossible to make this change without some measure > of backward incompatibility. Is it such an advantage to be almost the > same? It means that tools will be correct often enough that people > might not even notice a problem, and then they subtly give a > completely false value. With my scheme, they *always* give a clear and > obviously wrong value, and it's easy to see what has to be done. > > Python doesn't, as a general rule, opt for things that encourage > subtle data-dependent bugs. > > > With your implementation, it will always lie. Always. Every single time, > > no exceptions: it will report that the default value is a tuple > > (desc, value), which is wrong. > > Yes, it will clearly need to be updated for the new version. Instead > of being almost correct, and then flat-out lying in situations where > it should learn a better way. > > > With mine, it will be correct nearly always, especially if we use > > NotImplemented as the sentinel instead of None. And the cases that it > > gets wrong will only be the ones that use late-binding. It will never > > get an early-bound default wrong. > > I disagree that "correct nearly always" is a good thing. > > > > which is basically the same as I have, > > > only all in a single attribute. > > > > Right. And by combining them into a single attribute, you break > > backwards compatibility. I think unnecessarily, at the cost of more > > complexity. > > What if, in the future, a third type of optional argument is added - > such as "leave it unbound, no default given"? How would your scheme > handle this? Mine handles it just fine: give a new kind of value in > __defaults__. (Maybe None would work for that.) > > > Remember that __defaults__ is writable. What happens if somebody sticks > > a non-tuple into the __defaults__? Or a tuple with more than two items? > > > > func.__defaults__ = ((desc, value), (descr, value), 999, (1, 2, 3)) > > Writing to it goes through a checker already. You already can't write > func.__defaults__ = "foo". > > > So under your scheme, the interpreter cannot trust that the defaults are > > tuples that can be interpreted as (desc, value). > > Technically that's true in my current implementation, because I > haven't written the proper checks, but the interpreter is in full > control of what goes into that attribute. > > > Right, but you said that the early bound defaults **currently** have a > > None there. They don't. The current status quo of early bound defaults > > is that they are set to the actual default value, not a tuple with None > > in it. Obviously you know that. So that's why I'm asking, what have > > I misunderstood? > > You've misunderstood what I meant by "currently". Ignore it. That was > just one of the open issues. > > > > even do up their own from scratch, I would welcome it. It's much > > > easier to poke holes in an implementation than to actually write one > > > that is perfect. > > > > I've suggested an implementation that, I think, will be less complex and > > backwards compatible. I don't know if it will be faster. I expect in the > > common case of early binding, it will be, but what do I know about C? > > I asked for someone to WRITE an implementation, not suggest one. :) > It's easy to poke holes in someone else's plans. Much harder to > actually write something. Go ahead and put some code behind your > words, and then we can test them both. > > > > And fundamentally, there WILL be behavioural changes > > > here, so I'm not hugely bothered by the fact that the inspect module > > > needs to change for this. > > > > It's not just the inspect module. __defaults__ is public[1]. Anyone and > > everyone can read it and write it. Your implementation is breaking > > backwards compatibility, and I believe you don't need to. > > > > [1] Whether it is *officially* public or not, it is de facto public. > > > > What does "de facto public" mean, and does the language guarantee that > its format will never change? > > And, once again, you're offering something that is often correct and > sometimes a flat-out lie, where I'm offering something that adds a > clear and obvious structure. If you naively assume that everything in > __defaults__ is a two-element tuple (which is the case for early-bound > defaults), you'll get an obvious IndexError when you hit a late-bound > default, so even a basic adjustment will still be safe. By your > method, unless something is aware of late defaults, it will subtly get > things wrong. > > ChrisA >
[Python-ideas] Re: `is in`/`not is in` operators
On 2021-10-25 5:32 a.m., Jeremiah Vivian wrote: For quick checking if a `Movement` object is inside of an iterable. It seems the core of your problem is that you took the mechanism that's supposed to tell you if two objects are identical for another purpose, and now are complaining that you don't have a way to tell if two objects are identical. The simple and obvious solution is not to add a way to do __contains__ checks that bypass __eq__, it's to take out your __eq__ method (you seem to be satisfied with an identity check, but if not, you can give it a proper equality check instead), and to replace your misguided `Movement() in container` checks with a simple utility method: def has_movement(container): ... return any(isinstance(item, Movement) for item in container) ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/CKKXHSSMB74VYFD6EM5IA6A4L3GZT2VW/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: PEP 671 proof-of-concept implementation
On Fri, Oct 29, 2021 at 11:52 PM Steven D'Aprano wrote: > > Except that that's still backward-incompatible, since None is a very > > common value. > > How is it backwards incompatible? Any tool that looks at __defaults__ > finds *exactly* what was there before: a tuple of default values, not a > tuple of tuples (desc, value) or (value,) as in your implementation. It's fundamentally impossible to make this change without some measure of backward incompatibility. Is it such an advantage to be almost the same? It means that tools will be correct often enough that people might not even notice a problem, and then they subtly give a completely false value. With my scheme, they *always* give a clear and obviously wrong value, and it's easy to see what has to be done. Python doesn't, as a general rule, opt for things that encourage subtle data-dependent bugs. > With your implementation, it will always lie. Always. Every single time, > no exceptions: it will report that the default value is a tuple > (desc, value), which is wrong. Yes, it will clearly need to be updated for the new version. Instead of being almost correct, and then flat-out lying in situations where it should learn a better way. > With mine, it will be correct nearly always, especially if we use > NotImplemented as the sentinel instead of None. And the cases that it > gets wrong will only be the ones that use late-binding. It will never > get an early-bound default wrong. I disagree that "correct nearly always" is a good thing. > > which is basically the same as I have, > > only all in a single attribute. > > Right. And by combining them into a single attribute, you break > backwards compatibility. I think unnecessarily, at the cost of more > complexity. What if, in the future, a third type of optional argument is added - such as "leave it unbound, no default given"? How would your scheme handle this? Mine handles it just fine: give a new kind of value in __defaults__. (Maybe None would work for that.) > Remember that __defaults__ is writable. What happens if somebody sticks > a non-tuple into the __defaults__? Or a tuple with more than two items? > > func.__defaults__ = ((desc, value), (descr, value), 999, (1, 2, 3)) Writing to it goes through a checker already. You already can't write func.__defaults__ = "foo". > So under your scheme, the interpreter cannot trust that the defaults are > tuples that can be interpreted as (desc, value). Technically that's true in my current implementation, because I haven't written the proper checks, but the interpreter is in full control of what goes into that attribute. > Right, but you said that the early bound defaults **currently** have a > None there. They don't. The current status quo of early bound defaults > is that they are set to the actual default value, not a tuple with None > in it. Obviously you know that. So that's why I'm asking, what have > I misunderstood? You've misunderstood what I meant by "currently". Ignore it. That was just one of the open issues. > > even do up their own from scratch, I would welcome it. It's much > > easier to poke holes in an implementation than to actually write one > > that is perfect. > > I've suggested an implementation that, I think, will be less complex and > backwards compatible. I don't know if it will be faster. I expect in the > common case of early binding, it will be, but what do I know about C? I asked for someone to WRITE an implementation, not suggest one. :) It's easy to poke holes in someone else's plans. Much harder to actually write something. Go ahead and put some code behind your words, and then we can test them both. > > And fundamentally, there WILL be behavioural changes > > here, so I'm not hugely bothered by the fact that the inspect module > > needs to change for this. > > It's not just the inspect module. __defaults__ is public[1]. Anyone and > everyone can read it and write it. Your implementation is breaking > backwards compatibility, and I believe you don't need to. > > [1] Whether it is *officially* public or not, it is de facto public. > What does "de facto public" mean, and does the language guarantee that its format will never change? And, once again, you're offering something that is often correct and sometimes a flat-out lie, where I'm offering something that adds a clear and obvious structure. If you naively assume that everything in __defaults__ is a two-element tuple (which is the case for early-bound defaults), you'll get an obvious IndexError when you hit a late-bound default, so even a basic adjustment will still be safe. By your method, unless something is aware of late defaults, it will subtly get things wrong. ChrisA ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at
[Python-ideas] Re: PEP 671 proof-of-concept implementation
On Fri, Oct 29, 2021 at 10:22:43PM +1100, Chris Angelico wrote: > > Pardon me if this has already been discussed, but wouldn't it be better > > to leave defaults and kwdefaults alone, and add a new pair of attributes > > for late bound defaults? `__late_defaults__` and `__late_kwdefaults__`. > > The trouble with that is that positional arguments could have any > combination of early and late defaults. That's not a problem. > If the late ones are in a > separate attribute, there'd need to be some sort of synchronization > between them. It's not like they are mutable attributes that are constantly changing after the function is defined. (The values *inside* __defaults__ may or may not be mutable, but that's neither here nor there.) > (It would work for kwdefaults, but they only apply to > kwonly args - positional-or-keyword args go into __defaults__.) > > > Otherwise its a backwards-incompatable change to the internals of the > > function object, and one which is not (so far as I can tell) necessary. > > > > Obviously you need a way to indicate that a value in __defaults__ should > > be skipped. Here's just a sketch. Given: > > > > def func(a='alpha', b='beta', @c=expression, d=None) > > > > where only c is late bound, you could have: > > > > __defaults__ = ('alpha', 'beta', None, None) > > __late_defaults__ = (None, None, , None) > > > > The None values in __defaults__ mean to look in the __late_defaults__ > > tuple. If the appropriate value there is also None, return it, otherwise > > the parameter is late-bound. Evaluate it and return the result. > > Except that that's still backward-incompatible, since None is a very > common value. How is it backwards incompatible? Any tool that looks at __defaults__ finds *exactly* what was there before: a tuple of default values, not a tuple of tuples (desc, value) or (value,) as in your implementation. For functions that don't have any late-bound defaults, just set the `__late_defaults__` attribute to None and nothing changes. If the function defaults would be `(None, 1, 2, 3, 4)` today, they will remain `(None, 1, 2, 3, 4)` tomorrow, and the interpretation will be exactly the same. Only in functions that actually use late defaults, and set the `__late_defaults__` attribute to a non-None value, will see any difference. And even then, the difference only applies to early-bound defaults that match the sentinel. Suppose some introspection tool that knows nothing of late-defaults inspects the function. Here's the function again: def func(a='alpha', b='beta', @c=expression, d=None) For parameters a and b, nothing has changed as far as the tool is concerned. It will look in __defaults__ and see the strings 'alpha' and 'beta'. For parameter d, it will look at the value in `__defaults__[3]`, and see None, and *correctly* report that the default was None, so again, nothing has changed. It is only for parameter c that the tool will get it wrong. But then, what else could it do? It knows nothing about late-bound defaults. It's either going to fail, or lie. There is no other option. With your implementation, it will always lie. Always. Every single time, no exceptions: it will report that the default value is a tuple (desc, value), which is wrong. With mine, it will be correct nearly always, especially if we use NotImplemented as the sentinel instead of None. And the cases that it gets wrong will only be the ones that use late-binding. It will never get an early-bound default wrong. There is nothing better that we can do with an introspection tool that doesn't know about late defaults, except break it by removing `__defaults__` altogether. > So this form of synchronization wouldn't work; in fact, > *by definition*, any object can be in __defaults__, so there's no > possible sentinel that can indicate that late defaults should be > checked. I just gave you two. > The only way would be to first look in late defaults, and > only then look in defaults... Other way around. I expect that early bound defaults will continue to be the most common, by far, so we prefer to look there first. 1. Look in the early defaults. If the value found is not the sentinel, use it as the default. This part is effectively that same as the status quo. 2. If it is the sentinel, look in the late defaults. 3. If the value you find in the late defaults is the same sentinel, then use it as the default. 4. If it is a code object (function?) then evaluate it, and use whatever it returns as the default. 5. If it is something else, you can treat it as an error. Similar steps for the keyword-only defaults. > which is basically the same as I have, > only all in a single attribute. Right. And by combining them into a single attribute, you break backwards compatibility. I think unnecessarily, at the cost of more complexity. I gave a step by step strategy for using a sentinel that I am confident that would work. There's a little bit
[Python-ideas] Re: PEP 671 proof-of-concept implementation
On Fri, Oct 29, 2021 at 8:11 PM Steven D'Aprano wrote: > > On Fri, Oct 29, 2021 at 07:17:05PM +1100, Chris Angelico wrote: > > > * Argument defaults (either in __defaults__ or __kwdefaults__) are now > > tuples of (desc, value) or (desc,) for early-bound and late-bound > > respectively > > * Early-bound defaults get mapped as normal. Late-bound defaults are > > left unbound at time of function call. > > Pardon me if this has already been discussed, but wouldn't it be better > to leave defaults and kwdefaults alone, and add a new pair of attributes > for late bound defaults? `__late_defaults__` and `__late_kwdefaults__`. The trouble with that is that positional arguments could have any combination of early and late defaults. If the late ones are in a separate attribute, there'd need to be some sort of synchronization between them. (It would work for kwdefaults, but they only apply to kwonly args - positional-or-keyword args go into __defaults__.) > Otherwise its a backwards-incompatable change to the internals of the > function object, and one which is not (so far as I can tell) necessary. > > Obviously you need a way to indicate that a value in __defaults__ should > be skipped. Here's just a sketch. Given: > > def func(a='alpha', b='beta', @c=expression, d=None) > > where only c is late bound, you could have: > > __defaults__ = ('alpha', 'beta', None, None) > __late_defaults__ = (None, None, , None) > > The None values in __defaults__ mean to look in the __late_defaults__ > tuple. If the appropriate value there is also None, return it, otherwise > the parameter is late-bound. Evaluate it and return the result. Except that that's still backward-incompatible, since None is a very common value. So this form of synchronization wouldn't work; in fact, *by definition*, any object can be in __defaults__, so there's no possible sentinel that can indicate that late defaults should be checked. The only way would be to first look in late defaults, and only then look in defaults... which is basically the same as I have, only all in a single attribute. > That means that param=None will be a little bit more costly to fill at > function call time than it is now, but not by much. And non-None > defaults won't have any significant extra cost (just one check to see if > they are None). > > And if you really want to keep arg=None as fast as possible, we could > use some other sentinel like NotImplemented that is much less common. Having "a bit less" backward incompatibility isn't really a solution; if we need to have any, why have all the complexity? > > So far unimplemented is the description of the argument default. My > > plan is for early-bound defaults to have None there (as they currently > > do), but late-bound ones get the source code. > > That's not what I see currently in 3.10: > > >>> def func(a=1, b=2, c="hello"): > ... pass > ... > >>> func.__defaults__ > (1, 2, 'hello') > > > What am I missing? Currently, you don't get a description, you get a value. >>> def func(a=0x10, b=16, c=0o20): pass ... >>> func.__defaults__ (16, 16, 16) That's not a big deal with early-bound, but it's kinda crucial with late-bound, since the description is the only thing you'd get. The spec I'm currently going for is that a description of None means "use the repr of the value", and that's what early-bound defaults will use, so that has the same behaviour as current 3.11, and is the way it's currently implemented in the PEP 671 branch. What needs to change is late-bound defaults. Anyway, if someone wants to make changes to the implementation, or even do up their own from scratch, I would welcome it. It's much easier to poke holes in an implementation than to actually write one that is perfect. And fundamentally, there WILL be behavioural changes here, so I'm not hugely bothered by the fact that the inspect module needs to change for this. ChrisA ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/V34GMRHEYBHCESS4G4SRICWEP7LFOHNH/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: PEP 671 proof-of-concept implementation
On Fri, Oct 29, 2021 at 11:22:29AM +0200, Gerrit Holl wrote: > On Fri, 29 Oct 2021 at 11:10, Steven D'Aprano wrote: > > Obviously you need a way to indicate that a value in __defaults__ should > > be skipped. Here's just a sketch. Given: > > > > def func(a='alpha', b='beta', @c=expression, d=None) > > > > where only c is late bound, you could have: > > > > __defaults__ = ('alpha', 'beta', None, None) > > __late_defaults__ = (None, None, , None) > > Why not define an object for that? Because then it would have to be a public, named builtin. Or at least a public, named value, even if it's not in the builtin namespace, it would still be accessible to coders just by introspecting the function object. I think NotImplemented is a good sentinel to use. Its the same size as None, and much less commonly used as a default value. So the algorithm for fitting default values to parameters might look like this pseudocode: # single pass version for parameter in parameters: if parameter is unbound: obj = get default for this parameter, or fail if obj is NotImplemented: obj = get late_bound default for this parameter, or fail if obj is not NotImplemented: obj = (eval obj in function namespace) bind obj to parameter # two pass version, that ensures all early-bound parameters # have a value before the late-bound ones are evaluated for parameter in parameters: if parameter is unbound: obj = get default for this parameter, or fail if obj is NotImplemented: continue bind obj to parameter for parameter in parameters: if parameter is unbound: obj = get late_bound default for this parameter, or fail if obj is not NotImplemented: obj = (eval obj in function namespace) bind obj to parameter > In this example I don't mean a fancy new delayed evaluation type such > as has been discussed elsewhere, but just a sentinel so Python knows > it has to look into __late_defaults__ at all (I'm probably missing > something, and I didn't read all prior emails on this, but why > wouldn't this type contain a reference to the code object directly, > negating the need for __late_defaults__...?). How would the interpreter tell the difference between a code object which is early bound and needs to be returned directly, unexecuted, and a code object which is late-bound and needs to be executed? Any code object inside the function defaults can be grabbed and used as an early default, and we'd want it to remain unevaluated. -- Steve ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/TC7JQ6YALXTYWPTTCAECSSLDX47RFGCD/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: PEP 671 proof-of-concept implementation
On Fri, 29 Oct 2021 at 11:10, Steven D'Aprano wrote: > Obviously you need a way to indicate that a value in __defaults__ should > be skipped. Here's just a sketch. Given: > > def func(a='alpha', b='beta', @c=expression, d=None) > > where only c is late bound, you could have: > > __defaults__ = ('alpha', 'beta', None, None) > __late_defaults__ = (None, None, , None) Why not define an object for that? __defaults__ = ("alpha", "beta", _delayed, None) In this example I don't mean a fancy new delayed evaluation type such as has been discussed elsewhere, but just a sentinel so Python knows it has to look into __late_defaults__ at all (I'm probably missing something, and I didn't read all prior emails on this, but why wouldn't this type contain a reference to the code object directly, negating the need for __late_defaults__...?). Regards, Gerrit. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/G4R5X7G5MFYOPJOCUWASVRLR2O7MCUL6/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: PEP 671 proof-of-concept implementation
On Fri, Oct 29, 2021 at 07:17:05PM +1100, Chris Angelico wrote: > * Argument defaults (either in __defaults__ or __kwdefaults__) are now > tuples of (desc, value) or (desc,) for early-bound and late-bound > respectively > * Early-bound defaults get mapped as normal. Late-bound defaults are > left unbound at time of function call. Pardon me if this has already been discussed, but wouldn't it be better to leave defaults and kwdefaults alone, and add a new pair of attributes for late bound defaults? `__late_defaults__` and `__late_kwdefaults__`. Otherwise its a backwards-incompatable change to the internals of the function object, and one which is not (so far as I can tell) necessary. Obviously you need a way to indicate that a value in __defaults__ should be skipped. Here's just a sketch. Given: def func(a='alpha', b='beta', @c=expression, d=None) where only c is late bound, you could have: __defaults__ = ('alpha', 'beta', None, None) __late_defaults__ = (None, None, , None) The None values in __defaults__ mean to look in the __late_defaults__ tuple. If the appropriate value there is also None, return it, otherwise the parameter is late-bound. Evaluate it and return the result. That means that param=None will be a little bit more costly to fill at function call time than it is now, but not by much. And non-None defaults won't have any significant extra cost (just one check to see if they are None). And if you really want to keep arg=None as fast as possible, we could use some other sentinel like NotImplemented that is much less common. The advantage is that code that inspects function objects but doesn't know anything about late-binding will still work, except that it will report late-bound parameters as if they were set to None. Anyway, I care less about the implementation and more about not breaking backwards compatibility when it comes to inpecting function objects. > So far unimplemented is the description of the argument default. My > plan is for early-bound defaults to have None there (as they currently > do), but late-bound ones get the source code. That's not what I see currently in 3.10: >>> def func(a=1, b=2, c="hello"): ... pass ... >>> func.__defaults__ (1, 2, 'hello') What am I missing? -- Steve ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DM5W3REWVUWI7XKCN5UCA7UL27YABTOI/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] PEP 671 proof-of-concept implementation
https://github.com/Rosuav/cpython/tree/pep-671 So uhh... anyone who knows about the internals of CPython and wants to join me on this, I would *really* appreciate coauthors! The implementation ended up a lot more invasive than I originally planned. Some of that is inherent to the problem, but other parts might be able to be done more cleanly. The way I've done it: * Argument defaults (either in __defaults__ or __kwdefaults__) are now tuples of (desc, value) or (desc,) for early-bound and late-bound respectively * Early-bound defaults get mapped as normal. Late-bound defaults are left unbound at time of function call. * For each late-bound default as of the 'def' statement, a check is coded: if the local is unbound, set it based on the given expression. This means that it's possible to replace an early-bound default with a late-bound, but instead of actually evaluating the expression, it just leaves it unbound: >>> def f(x=1): print(x) ... >>> f.__defaults__ ((None, 1),) >>> f.__defaults__ = ((None,),) >>> f() Traceback (most recent call last): File "", line 1, in File "", line 1, in f UnboundLocalError: cannot access local variable 'x' where it is not associated with a value >>> I'm not entirely happy with this, but I also don't want to have too much of a performance impact in the normal case. So far unimplemented is the description of the argument default. My plan is for early-bound defaults to have None there (as they currently do), but late-bound ones get the source code. (In theory, it may be of value to retain the source code for earlies too, which would allow hex or octal integer literals to show up in help() as such, rather than showing the (decimal) repr of the resulting value.) Anyone got good pointers on how to do this, or is that likely to be impractical? Feel free to criticize my code. As you'll see from the commit messages in that branch, I have no idea what I'm doing here :) ChrisA ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/HFHO7LV4QSVFLWXJPQVSY2NPBBZACP5U/ Code of Conduct: http://python.org/psf/codeofconduct/