[Python-Dev] Re: Proto-PEP part 2: Alternate implementation proposal for "forward class" using a proxy object
So - good idea on creating the proxy. But would you really __need__ this badly that the proxy object would "become" the new class object, preserving its "id"? Just name re-binding (and the static type checkers _knowing_ the name will be re-assign to the actuall class object later) seems to be pretty straightforward, and would break too little. If one assigns the proxy to a different name in the meantime, and keep a reference to it: "consenting adults" apply: one just got a somewhat useless stub object to play along with. Ok, it would be more of a "stub" object than a "proxy" object, So, if the "proxy" is really needed - there is one type of existing proxy to classes in Python that is really transparent, and that is not easily creatable by Python code - those are the instances of "super()" - the will answer proper even to isinstance and issubclass calls, as it goes beyond what is possible by customizing __getattribute__. So, if a proxy is really needed instead of a simple name rebind, probably the existing code for "super' can be reused for them. As for: "do people keep references for the classes inside "__new__"? - Yes, all the time. The "__bind_proxy__" method you describe would again break compatibility (although in a much more reasonable way than the "__new__" split.) But then, the re-binding, if any, could take place inside "type.__new__", before it returns the newly created "cls" to the call to it made inside the custom metaclass.__new__ . Some code might break when getting back a proxied class at this point, but, up to that point, a lot of code could also break with this kind of proxies - "super()" instances work well but there are, of course, lots of corner cases. All in all, I think this is overkill for the problem at hand. As I wrote before: I stand with what Paul Moore wrote: in an ideal universe annotations for type checking should be optional. In the real world, there is a lot of pressure, surging from everywhere, for strict type-checking in all types of projects, open source or not, and I find this a very sad state of things. Breaking the language compatibility and features because it is needed for "optional" type checking is sad-squared. On Fri, Apr 22, 2022 at 10:20 PM Larry Hastings wrote: > > Here's one alternate idea for how to implement the "forward class" syntax. > > The entire point of the "forward class" statement is that it creates > the real actual class object. But what if it wasn't actually the > "real" class object? What if it was only a proxy for the real object? > > In this scenario, the syntax of "forward object" remains the same. > You define the class's bases and metaclass. But all "forward class" > does is create a simple, lightweight class proxy object. This object > has a few built-in dunder values, __name__ etc. It also allows you > to set attributes, so let's assume (for now) it calls > metaclass.__prepare__ and uses the returned "dict-like object" as > the class proxy object __dict__. > > "continue class" internally performs all the rest of the > class-creation machinery. (Everything except __prepare__, as we > already called that.) The first step is metaclass.__new__, which > returns the real class object. "continue class" takes that > object and calls a method on the class proxy object that says > "here's your real class object". From that moment on, the proxy > becomes a pass-through for the "real" class object, and nobody > ever sees a reference to the "real" class object ever again. > Every interaction with the class proxy object is passed through > to the underlying class object. __getattribute__ calls on the > proxy look up the attribute in the underlying class object. If > the object returned is a bound method object, it rebinds that > callable with the class proxy instead, so that the "self" passed > in to methods is the proxy object. Both base_cls.__init_subclass__ > and cls.__init__ see the proxy object during class creation. As far > as Python user code is concerned, the class proxy *is* the class, > in every way, important or not. > > The upside: this moves all class object creation code into "continue > class" call. We don't have to replace __new__ with two new calls. > > The downside: a dinky overhead to every interaction with a "forward > class" class object and with instances of a "forward class" class > object. > > > A huge concern: how does this interact with metaclasses implemented > in C? If you make a method call on a proxy class object, and that > calls a C function from the metaclass, we'd presumably have to pass > in the "real class object", not the proxy class object. Which means > references to the real class object could leak out somewhere, and > now we have a real-class-object vs proxy-class-object identity crisis. > Is this a real concern? > > > A possible concern: what if metaclass.__new__ keeps a reference to > the object it created? Now we have two objects with an identity > crisis. I don't know if peo
[Python-Dev] Re: Proto-PEP part 2: Alternate implementation proposal for "forward class" using a proxy object
TL;DR (literally, I will go back and read it now, but after reading the first paragraphs: _a proxy_ object yes, then dividing class creation in 2 blocks would not break things) /me goes back to text. On Fri, Apr 22, 2022 at 10:20 PM Larry Hastings wrote: > > Here's one alternate idea for how to implement the "forward class" syntax. > > The entire point of the "forward class" statement is that it creates > the real actual class object. But what if it wasn't actually the > "real" class object? What if it was only a proxy for the real object? > > In this scenario, the syntax of "forward object" remains the same. > You define the class's bases and metaclass. But all "forward class" > does is create a simple, lightweight class proxy object. This object > has a few built-in dunder values, __name__ etc. It also allows you > to set attributes, so let's assume (for now) it calls > metaclass.__prepare__ and uses the returned "dict-like object" as > the class proxy object __dict__. > > "continue class" internally performs all the rest of the > class-creation machinery. (Everything except __prepare__, as we > already called that.) The first step is metaclass.__new__, which > returns the real class object. "continue class" takes that > object and calls a method on the class proxy object that says > "here's your real class object". From that moment on, the proxy > becomes a pass-through for the "real" class object, and nobody > ever sees a reference to the "real" class object ever again. > Every interaction with the class proxy object is passed through > to the underlying class object. __getattribute__ calls on the > proxy look up the attribute in the underlying class object. If > the object returned is a bound method object, it rebinds that > callable with the class proxy instead, so that the "self" passed > in to methods is the proxy object. Both base_cls.__init_subclass__ > and cls.__init__ see the proxy object during class creation. As far > as Python user code is concerned, the class proxy *is* the class, > in every way, important or not. > > The upside: this moves all class object creation code into "continue > class" call. We don't have to replace __new__ with two new calls. > > The downside: a dinky overhead to every interaction with a "forward > class" class object and with instances of a "forward class" class > object. > > > A huge concern: how does this interact with metaclasses implemented > in C? If you make a method call on a proxy class object, and that > calls a C function from the metaclass, we'd presumably have to pass > in the "real class object", not the proxy class object. Which means > references to the real class object could leak out somewhere, and > now we have a real-class-object vs proxy-class-object identity crisis. > Is this a real concern? > > > A possible concern: what if metaclass.__new__ keeps a reference to > the object it created? Now we have two objects with an identity > crisis. I don't know if people ever do that. Fingers crossed that > they don't. Or maybe we add a new dunder method: > > @special_cased_staticmethod > metaclass.__bind_proxy__(metaclass, proxy, cls) > > This tells the metaclass "bind cls to this proxy object", so > metaclasses that care can update their database or whatever. > The default implementation uses the appropriate mechanism, > whatever it is. > > One additional probably-bad idea: in the case where it's just a > normal "class" statement, and we're not binding it to a proxy, > should we call this? > > metaclass.__bind_proxy__(metaclass, None, cls) > > The idea there being "if you register the class objects you create, > do the registration in __bind_proxy__, it's always called, and you'll > always know the canonical object in there". I'm guessing probably not, > in which case we tell metaclasses that track the class objects we > create "go ahead and track the object you return from __new__, but > be prepared to update your tracking info in case we call __bind_proxy__ > on you". > > > A small but awfully complicated wrinkle here: what do we do if the > metaclass implements __del__? Obviously, we have to call __del__ > with the "real" class object, so it can be destroyed properly. > But __del__ might resurrect that object, which means someone took a > reference to it. > > > > One final note. Given that, in this scenario, all real class creation > happens in "continue class", we could move the bases and metaclass > declaration down to the "continue class" statement. The resulting > syntax would look like: > > forward class X > > ... > > continue class X(base1, base2, metaclass=AmazingMeta, > rocket="booster") > > Is that better? worse? doesn't matter? I don't have an intuition about > it right now--I can see advantages to both sides, and no obvious > deciding factor. Certainly this syntax prevents us from calling > __prepare__ so early, so we'd have to use a real dict in the "forward > class" proxy ob
[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes
On Sat, Apr 23, 2022 at 7:10 PM Paul Moore wrote: > On Sat, 23 Apr 2022 at 22:42, Rob Cliffe via Python-Dev > wrote: > > > > UGH! > > > > I thought there was a general understanding that when typing was added > > to Python, there would be no impact, or at least minimal impact, on > > people who didn't use it. (Raises hand.) > > Now we see an(other) instance of intention creep. > > To be fair, none of this is needed unless you want to add type > annotations to your code. So it's still perfectly possible to ignore > all of this (which is what I am currently doing). > > What I am concerned about is when users of libraries I write start to > claim that I "need" to add this sort of stuff to my code, so that they > can type check their code that uses mine, and/or they can get tooltips > for my APIs in their IDEs. That's where I think the biggest issue with > a proposal like this arises - the *social* pressure on people to adopt > typing, and all the complexities it adds. But again, that's not > something that's specific to this proposal, it's inherent in the whole > question of whether people add type annotations at all. > > So I'm -1 on this proposal, but just because I fear I may be forced to > use it when I don't need to, rather than because I think it's a bad > idea per se. > > Paul > I stand with Paul here - But as a "heavy user" of metaclasses, as far as that is possible. (less in production code or "non toy projects", but a lot in understanding what goes, answering questions and teaching people on how they should best use (or skip using) a metaclass. I'd say this breaks too much of the current working of class creation for, possibly, too little. And - on a second thought: Was not this the kind of thing that PEP 563 was supposed to fix? Pep 563 is accepted now, I think one may just use a forward declaration in any annotation context: it will be lazily evaluated and the target class will have been created. So, while I understand the problem with forward references, which have been historically workaround with the usage of strings (even long before even Python 3.0, in Django's Models) I believe this is not a good approach So first part: == I'd be comfortable with the statements and blocks as they are if the "forward class" statement would not mess with the metaclass and class creation at all: just create a place-holder the type checkers could then use to find the class later on the file, or later on the code, and then use that class. Has this been thought of? Is there any reason why that would not work, better than it will put some burden on static-type checkers, as opposed to fundamentally modify the way classes are built so that any library featuring any metaclass will have to be divided in incompatible releases "before this pep" and "after this pep"? (and with the harsh inconvenience of needing to have one a part of the code which is often the most complicated 100% rewritten) Second part: = The proposal as it is does not preserve everything that is possible with the current metaclass "__new__", besides complicating things. Moreover, there _are_ similar steps on "breaking metaclass.`__new__` " that would actually be useful in customizing the way classes are created, without sacrificing compatibility (in a sense the custom "metaclass.__new__" would still be called exactly as it is today). If there is any chance this thing will move on, I'd like to detail these ideas (it would be longish, similar in size with the proto-pep text) - and maybe there is a way to reconcile compatibility, without the proposal of splitting "__new__" in two 100% incompatible methods. I will leave the actual amendments to this part of the proposal (which would affect all the inner workings of the metaclasses as exposed) to another message. But I will leave one first, huge, problem here that: ```Python def __new_forward__(metaclass, name, bases, namespace, **kwargs): def __new_continue__(metaclass, cls, **kwargs): ``` These two, as they are, do not allow for some of the things that are possible today: let's supose I want, in a custom metaclass `__new__` method inspect the namespace and creates a "__slots__" declaration based on what already _is_ on the namespace: in "__new_forward__" the namespace is filled in a non deterministic way and should not be inspected , and in "__new_continue__" it had already been processed by (what currently is) "type.__new__" and baked into the proxy-map inside the "cls". I the specific case of creating "__slots__" in a similar way "@dataclass" does, the oportunity is gone. I won't mention the suggestion in the text that "class transformers that would create slots are free to re-create the class object upon its first instantiation" is _rather_ bogus. I can imagine a metaclass "__call__" method that could do that, but then, the classes the instances would be actually using would rather become either have to be kept as an attribute in the cl
[Python-Dev] Re: Proto-PEP part 3: Closing thoughts on "forward class", etc.
On Sat, 23 Apr 2022, 11:17 am Larry Hastings, wrote: > > > Just a quick note from me on the proto-PEP and the two proposed > implementations. When I started exploring this approach, I didn't suspect > it'd require such sweeping changes to be feasible. Specifically, I didn't > think I was going to propose changing the fundamental mechanism used to > create class objects. That's an enormous change, and it makes me > uncomfortable; I suspect I won't be alone in having that reaction. > > The alternate implementation with proxy objects was borne of my reaction, > but it's worrisome too. It's a hack--though whether it's a "big" hack or a > "small" hack is debatable. Anyway, I'm specifically worried about the > underlying class object escaping the proxy and becoming visible inside > Python somehow. If that happened, we'd have two objects representing the > same "type" at runtime, a situation that could quickly become confusing. > > Also, as I hopefully made clear in the "alternate implementation" approach > using a class proxy object, I'm not 100% certain that the proxy will work > in all cases. I ran out of time to investigate it more--I wanted to post > this idea with some lead time before the 2022 Language Summit, so that > folks had time to read and digest it and discuss it before the Summit. I > have some implementation ideas--the "class proxy" class may need its own > exotic metaclass. > > Ultimately I'm posting this proto-PEP to foster discussion. I'm confident > that "forward class" / "continue class" could solve all our > forward-reference and circular-reference problems; the questions we need to > collectively answer are: > >- how should the implementation work, and >- is the cost of the implementation worth it? > > Something worth considering: whether forward references need to be *transparent* at runtime. If they can just be regular ForwardRef objects then much of the runtime complexity will go away (at the cost of some potentially confusing identity check failures between forward references and the actual class objects). ForwardRef's constructor could also potentially be enhanced to accept a "resolve" callable, and a "resolve()" method added to its public API, although the extra complexity that would bring may not be worth it. Cheers, Nick. > > > Best wishes, > > > */arry* > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/HD7YPONSPL5ZFZISKCOUWVUXMIJTQG2M/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DMITVTUIQKJW6RYVOPQXHD54VSYE7QHA/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes
On Sat, 23 Apr 2022 at 22:42, Rob Cliffe via Python-Dev wrote: > > UGH! > > I thought there was a general understanding that when typing was added > to Python, there would be no impact, or at least minimal impact, on > people who didn't use it. (Raises hand.) > Now we see an(other) instance of intention creep. To be fair, none of this is needed unless you want to add type annotations to your code. So it's still perfectly possible to ignore all of this (which is what I am currently doing). What I am concerned about is when users of libraries I write start to claim that I "need" to add this sort of stuff to my code, so that they can type check their code that uses mine, and/or they can get tooltips for my APIs in their IDEs. That's where I think the biggest issue with a proposal like this arises - the *social* pressure on people to adopt typing, and all the complexities it adds. But again, that's not something that's specific to this proposal, it's inherent in the whole question of whether people add type annotations at all. So I'm -1 on this proposal, but just because I fear I may be forced to use it when I don't need to, rather than because I think it's a bad idea per se. Paul PS To be open here, I do actually like type annotations in straightforward situations - they have located some bugs in my code for me, and tooltips in VS code *are* nice. What I don't like is not being able to stick to simple stuff and not bother annotating the complicated bits, or being pushed into over-strict annotations because it's too hard (or verbose) to express dynamic, duck-typed constraints. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MOO3FUV3XKSKEGRIAOXSIVBMGPGFXYLV/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes
UGH! I thought there was a general understanding that when typing was added to Python, there would be no impact, or at least minimal impact, on people who didn't use it. (Raises hand.) Now we see an(other) instance of intention creep. Rob Cliffe On 23/04/2022 02:13, Larry Hastings wrote: This document is a loose proto-PEP for a new "forward class" / "continue class" syntax. Keep in mind, the formatting is a mess. If I wind up submitting it as a real PEP I'll be sure to clean it up first. /arry -- PEP : Forward declaration of classes Overview Python currently has one statement to define a class, the `class` statement: ```Python class X(): # class body goes here def __init__(self, key): self.key = key ``` This single statement declares the class, including its bases and metaclass, and also defines the contents of the class in the "class body". This PEP proposes an additional syntax for declaring a class which splits this work across two statements: * The first statement is `forward class`, which declares the class and binds the class object. * The second statement is `continue class`, which defines the contents of the class in the "class body". To be clear: `forward class` creates the official, actual class object. Code that wants to take a reference to the class object may take references to the `forward class` declared class, and interact with it as normal. However, a class created by `forward class` can't be *instantiated* until after the matching `continue class` statement finishes. Defining class `X` from the previous example using this new syntax would read as follows: ``` forward class X() continue class X: # class body goes here def __init__(self, key): self.key = key ``` This PEP does not propose altering or removing the traditional `class` statement; it would continue to work as before. Rationale - Python programmers have had a minor problem with classes for years: there's no way to have early-bound circular dependencies between objects. If A depends on B, and B depends on A, there's no linear order that allows you to cleanly declare both. Most of the time, the dependencies were in late-binding code, e.g. A refers to B inside a method. So this was rarely an actual problem at runtime. When this problem did arise, in code run at definition-time, it was usually only a minor headache and could be easily worked around. But the explosion of static type analysis in Python, particularly with the `typing` module and the `mypy` tool, has made circular definition-time dependencies between classes commonplace--and much harder to solve. Here's one simple example: ```Python class A: value: B class B: value: A ``` An attribute of `B` is defined using a type annotation of `A`, and an attribute of `A` is defined using a type annotation of `B`. There's no order to these two definitions that works; either `A` isn't defined yet, or `B` isn't defined yet. Various workarounds and solutions have been proposed to solve this problem, including two PEPs: PEP 563 (automatic stringized annotations) and PEP 649 (delayed evaluation of annotations using functions). But nothing so far has been both satisfying and complete; either it is wordy and clumsy to use (manually stringizing annotations), or it added restrictions and caused massive code breakage for runtime use of annotations (PEP 563), or simply didn't solve every problem (PEP 649). This proposed `forward class` / `continue class` syntax should permit solving *every* forward-reference and circular-reference problem faced in Python, using an elegant and Pythonic new syntax. As a side benefit, `forward class` and `continue class` syntax enables rudimentary separation of "interface" from "implementation", at least for classes. A user seeking to "hide" the implementation details of their code could put their class definitions in one module, and the implementations of those classes in a different module. This new syntax is not intended to replace the traditional `class` declaration syntax in Python. If this PEP were accepted, the `class` statement would still be the preferred mechanism for creating classes in Python; `forward class` should only be used when it confers some specific benefit. Syntax -- The `forward class` statement is the same as the `class` statement, except it doesn't end with a colon and is not followed by an indented block. Without any base classes or metaclass, the `forward class` statement is as follows: ``` forward class X ``` This would declare class `X`. If `X` needs base classes or metaclass, the corresponding `forward class` statement would be as follows, rendered in a sort of "function prototype" manner: ``` forward class X(*bases, metaclass=object, **kwargs) ``` The `continue class` statement is similar to a
[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes
On 4/23/22 08:57, Eric V. Smith wrote: On 4/23/2022 9:55 AM, Jelle Zijlstra wrote: However, it doesn't solve the problem for base classes. For example, str is conceptually defined as `class str(Sequence["str"]):`. A forward reference can't make `str` defined when the bases are evaluated, because bases are resolved at the `forward class` stage. Larry's second email "Proto-PEP part 2: Alternate implementation proposal for "forward class" using a proxy object" discusses a possibility to move the bases and metaclasses to the "continue class" stage. It also has the advantage of not changing the behavior of __new__, and I think is in general easier to reason about. Let me expound on Eric's statement for a bit. Moving the base classes and metaclass to the "continue" statement might permit the self-referential "str" definition suggested above: forward class str ... continue class str(Sequence[str]): ... Though I suspect this isn't viable, or at least not today. I'm willing to bet a self-referential definition like this would result in an infinite loop when calculating the MRO. I don't have a strong sense of whether it'd be better to have the base classes and metaclass defined with the "forward" declaration or the "continue" declaration. The analogous syntax in C++ has the base classes defined with the "continue" class. Actually, that reminds me of something I should have mentioned in the proto-PEP. If the "class proxy" version is viable and desirable, and we considered moving the base classes and metaclass down to the "continue" statement, that theoretically means we could drop the "forward" and "continue" keywords entirely. I prefer them, simply because it makes it so explicit what you're reading. But the equivalent syntax in C++ doesn't bother with extra keywords for either the "forward" declaration or the "continue" declaration, and people seem to like it fine. Using that variant of the syntax, the toy example from the PEP would read as follows: class A class B: value: A class A: value: B If you're speed-reading here and wondering "wait, how is this not ambiguous?", note that in the first line of the example, the "class" statement has no colon. Also, it would never have parentheses or decorators. The forward declaration of a class would always be just "class", followed by a name, followed by a newline (or comment). //arry/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GTOJIZZXON3KND37WSATL3E7Q4KON6YO/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes
On 4/23/22 06:55, Jelle Zijlstra wrote: So to reiterate, your proposal would be to write this as: forward class B: pass class A: value: B continue class B: value: A Not quite; the "forward class" statement doesn't have a colon or a class body. This would be written as: forward class B class A: value: B continue class B: value: A While the current workaround is: class A: value: "B" class B: value: "A" In this example, with two toy classes in one file, it shouldn't be necessary to quote the annotation in B. So all you need is the quotes around the first annotation: class A: value: "B" class B: value: A I don't think I would write the "forward class" version if I had the choice. It's clunkier and requires more refactoring if I change my mind about whether the `value` attribute should exist. In this toy example, it adds an extra line. Describing that as "clunky" is a matter of opinion; I disagree and think it's fine. But the real difference is when it comes to larger codebases. If classes "A" and "B" are referenced dozens or even hundreds of times, you'd have to add quote marks around every annotation that references one (both?). Manual stringization of large codebases was sufficiently disliked as to have brought about the creation and acceptance of PEP 563. Judicious use of the "forward class" statement should obviate most (all?) the manual stringizing in these codebases. //arry/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TVCB6ITGHO5QI5GYFNMEVEZKE6K24UDB/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes
On 4/23/2022 9:55 AM, Jelle Zijlstra wrote: However, it doesn't solve the problem for base classes. For example, str is conceptually defined as `class str(Sequence["str"]):`. A forward reference can't make `str` defined when the bases are evaluated, because bases are resolved at the `forward class` stage. Larry's second email "Proto-PEP part 2: Alternate implementation proposal for "forward class" using a proxy object" discusses a possibility to move the bases and metaclasses to the "continue class" stage. It also has the advantage of not changing the behavior of __new__, and I think is in general easier to reason about. He and I have discussed this approach, but neither of have looked at in enough detail to know if the implementation is possible. Some of the concerns are noted in that email. Eric ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MODT7FKWYNDOR4D7LZRW2UOQB5OQSSJ3/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes
On 4/23/2022 3:28 AM, Tobias HT wrote: On the other hand, there's something I've been seeing around. I don't know if it was introduced by Mypy or something, but its the use of declaration files. I think they are saved as pyi files. They just have the declaration of a python object, be it class or variable. What if we just found a way of reusing that instead? As they currently exist, stub files (.pyi files) don't contain enough information. In particular, they don't have the metaclass information. This could be changed, but at that point you basically have the "forward" declaration, but it's hidden away where the interpreter can't see it. Eric ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZNZS7Q2MYGGDW5DJ4LOAQ2FOOULKQV3S/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes
This seems like it would solve a huge problem that I've often faced myself, but by introducing something that might later on cause problems I anticipate. Like some people suggested earlier, introducing new keyword in python is not a good solution, I feel it just adds to the already existing bulk and increases complexity. Also Like some else mentioned again, this solution doesn't seem pythonic. I don't know what they meant by that, but in hindsight, it does really not feel pythonic. On the other hand, there's something I've been seeing around. I don't know if it was introduced by Mypy or something, but its the use of declaration files. I think they are saved as pyi files. They just have the declaration of a python object, be it class or variable. What if we just found a way of reusing that instead? ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/IW223KRYW7TX6UTTDMMGYQQWHEGWV32O/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: [Typing-sig] Almost accepting PEP 681 – Data Class Transforms
El sáb, 23 abr 2022 a las 4:26, Petr Viktorin () escribió: > Hello, > As an initial implementation that will be improved in the future, the > specification in PEP 681 is fine. Feel free to add the decorator to > Python 3.11 at your convenience. > Thanks Petr, that's great to hear! I opened a PR at https://github.com/python/cpython/pull/91861 so we can get the new decorator in before the feature freeze. > > However, the PEP includes several worrying recommendations like: > > - we recommend that the maintainers of attrs move away from the legacy > semantics and adopt auto_attribs behaviors by default. > - We chose not to support this feature and recommend that attrs users > avoid converters. > - Attrs users should use the dataclass-standard eq and order parameter > names instead. > > These are probably meant as recommendations from typing-sig, but an > accepted PEP represents consensus of the entire Python community. A > typing PEP is not an appropriate place to make recommendations like > this, especially without reaching out to the maintainer of attrs. > As far as I know,the attrs and pydantic libraries are using the > reference implementation, but their authors weren't consulted on the PEP > itself. > > Could you either change the wording (e.g. say that the unsupported > features need bespoke type-checker functionality for proper type > checking), or work with attrs to make the same recommendations in its > documentation? > Agree that these recommendations sound overly normative. I think we should remove the recommendations and simply spell out the features that won't work with dataclass_transform(). It's up to users' own judgment what they do with that information. Erik, could you propose a change to the PEP text? > > > Happy typing, > — Petr, on behalf of the Steering Council > ___ > Typing-sig mailing list -- typing-...@python.org > To unsubscribe send an email to typing-sig-le...@python.org > https://mail.python.org/mailman3/lists/typing-sig.python.org/ > Member address: jelle.zijls...@gmail.com > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QPT3SJBXKSUVQ4DEFSZVVQBTBMG2XKML/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Python multithreading without the GIL
Hello all, I am very excited about a future multithreaded Python. I managed to postpone some rewrites in the company I work for Rust/Go, precisely because of the potential to have a Python solution in the medium term. I was wondering. Is Sam Gross' nogil merge being seriously considered by the core Python team? ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/SNSLKDHCE3J2VQHZCWFHNPDAEWGKEWN6/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes
I don't think this proposal is a good solution for the problems of static
typing users.
El vie, 22 abr 2022 a las 18:16, Larry Hastings ()
escribió:
> But the explosion of static type analysis in Python, particularly with
> the `typing` module and the `mypy` tool, has made circular definition-time
> dependencies between classes commonplace--and much harder to solve. Here's
> one simple example:
>
> ```Python
> class A:
> value: B
>
> class B:
> value: A
> ```
>
> So to reiterate, your proposal would be to write this as:
forward class B:
pass
class A:
value: B
continue class B:
value: A
While the current workaround is:
class A:
value: "B"
class B:
value: "A"
I don't think I would write the "forward class" version if I had the
choice. It's clunkier and requires more refactoring if I change my mind
about whether the `value` attribute should exist.
I'd prefer if we found some elegant way to get the natural way to write
this code to work, which is the way you wrote it in your post (with no
explicit declarations or stringifications). Carl Meyer's idea at
https://github.com/larryhastings/co_annotations/issues/2#issuecomment-1092432875
is a very promising approach that should allow us to do that.
I also agree with Mehdi2277's concern that this feature would be difficult
for static type checkers to fully implement, because class declaration and
implementation may be widely separated. To be sure, static checkers can put
in additional restrictions, like requiring declaration and implementation
to be in the same module, but it would be unfortunate if a feature designed
to help users of static typing actually makes it harder to adopt it.
-
That said, one nice thing about the proposal is that it can help with
forward references outside of annotations, a problem which neither PEP 563
nor 649 currently solves. Those include type aliases (MyAlias = Sequence["A
| B"]) and TypeVar bounds (Typevar("AT", bound="A | B")). However, it
doesn't solve the problem for base classes. For example, str is
conceptually defined as `class str(Sequence["str"]):`. A forward reference
can't make `str` defined when the bases are evaluated, because bases are
resolved at the `forward class` stage.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/F7L4SU3QIQLPOYHJF5RNJLROF4VMJTMK/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Almost accepting PEP 681 – Data Class Transforms
Hello, As an initial implementation that will be improved in the future, the specification in PEP 681 is fine. Feel free to add the decorator to Python 3.11 at your convenience. However, the PEP includes several worrying recommendations like: - we recommend that the maintainers of attrs move away from the legacy semantics and adopt auto_attribs behaviors by default. - We chose not to support this feature and recommend that attrs users avoid converters. - Attrs users should use the dataclass-standard eq and order parameter names instead. These are probably meant as recommendations from typing-sig, but an accepted PEP represents consensus of the entire Python community. A typing PEP is not an appropriate place to make recommendations like this, especially without reaching out to the maintainer of attrs. As far as I know,the attrs and pydantic libraries are using the reference implementation, but their authors weren't consulted on the PEP itself. Could you either change the wording (e.g. say that the unsupported features need bespoke type-checker functionality for proper type checking), or work with attrs to make the same recommendations in its documentation? Happy typing, — Petr, on behalf of the Steering Council ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JWMSWJ6JZJNNG3JKPBYDMQUUHZFD2FZL/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes
I should have said "numpy_forward", not "numpy.forward". I changed my mind at the last second as I was writing that email, and momentarily forgot that when you import x.y you implicitly import x. /arry On Sat, Apr 23, 2022, 01:53 Larry Hastings wrote: > > On 4/23/22 01:14, Steven D'Aprano wrote: > > On Sat, Apr 23, 2022 at 12:46:37AM -0700, Larry Hastings wrote: > > > But rather than speculate further, perhaps someone who works on one of > the static type analysis checkers will join the discussion and render an > informed opinion about how easy or hard it would be to support "forward > class" and "continue class". > > > No offense Larry, but since this proto-PEP is designed to help the > typing community (I guess...) shouldn't you have done that before > approaching Python-Dev with the proposal? > > The perfect is the enemy of the good. Like I said, I wanted to get this > out there before the Language Summit, and I just ran out of time. I think > there's also some sort of typing summit next week? I'm not really plugged > in to the static type analysis world--I don't use it in any of my projects. > > > Wouldn't that be a massively breaking change? Anyone who does: > > from numpy import ndarray > > will get the forward-declared class object instead of the fully > initialised class object, leading to all sorts of action-at-a-distance > bugs. > > I wasn't recommending The Famous numpy Project do this exact thing, it was > an abstract example using the name "numpy". I didn't think this was a real > example anyway, as I was assuming that most people who import numpy don't > do so in an "if TYPE_CHECKING:" block. > > Separating the forward class declaration from the continue class > implementation in the actual "numpy" module itself is probably not in the > cards for a while, if ever. But perhaps numpy could do this: > > import numpy.forward > if TYPE_CHECKING: > import numpy > > In this case, the "numpy" module would also internally "import > numpy.forward", and would contain the "continue class" statements for the > forward-declared classes in "numpy.forward". > > There are lots of ways to solve problems with the flexibility afforded by > the proposed "forward class" / "continue class" syntax. Perhaps in the > future you'll suggest some of them! > > > */arry* > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QHHI3D2T7MK2Y4SCRJ33BO433R4MKZYU/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes
On 4/23/22 03:10, Terry Reedy wrote: On 4/22/2022 11:16 PM, Larry Hastings wrote: So I still prefer "forward class". I don't think it's as clear as "forward class" 'forward class' for an incomplete class is not at all clear to me. It is not clear to me which part of speech you intend it to be: noun, verb, adjective, or adverb. You must have some experience with 'forward' in a different context that makes it clearer to you. It's a reference to the term "forward declaration": https://en.wikipedia.org/wiki/Forward_declaration //arry/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VEX3ONWBKIIDF2VR5OPANSKEIDFV3AP3/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes
On 4/22/2022 11:16 PM, Larry Hastings wrote: So I still prefer "forward class". I don't think it's as clear as "forward class" 'forward class' for an incomplete class is not at all clear to me. It is not clear to me which part of speech you intend it to be: noun, verb, adjective, or adverb. You must have some experience with 'forward' in a different context that makes it clearer to you. -- Terry Jan Reedy ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4BB7UPSE26RFYUMX4PZOJHHQYKCYAYAR/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes
On 4/23/22 01:14, Steven D'Aprano wrote: On Sat, Apr 23, 2022 at 12:46:37AM -0700, Larry Hastings wrote: But rather than speculate further, perhaps someone who works on one of the static type analysis checkers will join the discussion and render an informed opinion about how easy or hard it would be to support "forward class" and "continue class". No offense Larry, but since this proto-PEP is designed to help the typing community (I guess...) shouldn't you have done that before approaching Python-Dev with the proposal? The perfect is the enemy of the good. Like I said, I wanted to get this out there before the Language Summit, and I just ran out of time. I think there's also some sort of typing summit next week? I'm not really plugged in to the static type analysis world--I don't use it in any of my projects. Wouldn't that be a massively breaking change? Anyone who does: from numpy import ndarray will get the forward-declared class object instead of the fully initialised class object, leading to all sorts of action-at-a-distance bugs. I wasn't recommending The Famous numpy Project do this exact thing, it was an abstract example using the name "numpy". I didn't think this was a real example anyway, as I was assuming that most people who import numpy don't do so in an "if TYPE_CHECKING:" block. Separating the forward class declaration from the continue class implementation in the actual "numpy" module itself is probably not in the cards for a while, if ever. But perhaps numpy could do this: import numpy.forward if TYPE_CHECKING: import numpy In this case, the "numpy" module would also internally "import numpy.forward", and would contain the "continue class" statements for the forward-declared classes in "numpy.forward". There are lots of ways to solve problems with the flexibility afforded by the proposed "forward class" / "continue class" syntax. Perhaps in the future you'll suggest some of them! //arry/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/Y2PSUKYJFCVUUWBBV5Z3HNZBVRVS6SZS/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes
On 4/23/22 00:53, Steven D'Aprano wrote: It's a "forward-declared class object". It's the real class object, but it hasn't been fully initialized yet, and won't be until the "continue class" statement. The only thing that makes it not fully initialised is that it has a bozo bit dunder "__forward__" instructing the interpreter to disallow instantiation. Yes? If I take that class object created by `forward class X`, and delete the dunder, there is no difference between it and other regular classes. Am I correct? No, there are several differences. * It still has the "dict-like object" returned by metaclass.__prepare__, rather than its final dict. * Its class body hasn't been called yet, so it likely doesn't have any of its important dunder methods. * It hasn't had its BaseClass.__init_subclass__ called yet. * It hasn't had its metaclass.__init__ called yet. The "forward-declared class object" is in a not-yet-fully initialized state, and is not ready for use as a class. From my perspective, the "__forward__" attribute is an internal implementation detail, and something that user code should strictly leave alone. But if it's considered too dangerous to expose to users, we could hide it in the class object and not expose it to users. I'm not convinced that's the right call; I think the Consenting Adults rule still applies. Python lets you do crazy things like assigning to __class__, and resurrecting objects from inside their __del__; manually removing __forward__ seems like it falls into the same category. It's not recommended, and we might go so far as to say doing that results in undefined behavior. But Python shouldn't stand in your way if you really think you need to do it for some reason. //arry/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2XDQGNBYQMN6M7Y5JNZA26JX5XS5AYIF/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes
On Sat, Apr 23, 2022 at 12:46:37AM -0700, Larry Hastings wrote: > But rather than speculate further, perhaps someone who works on one of > the static type analysis checkers will join the discussion and render an > informed opinion about how easy or hard it would be to support "forward > class" and "continue class". No offense Larry, but since this proto-PEP is designed to help the typing community (I guess...) shouldn't you have done that before approaching Python-Dev with the proposal? I might have missed it, but I don't think I've seen this discussed on the typing-sig mailing list. Or have I misunderstood? If typing is not driving this, what is? > >One other edge case here is how would you forward declare an annotation > >for a type that like this, > > > >if TYPE_CHECKING: > > import numpy > > > >def f(x: numpy.ndarray) -> None: > > ... > > > >forward declaring ndarray here would not make numpy.ndarray available. > > In this case, adding forward declarations for classes in "numpy" would > be up to the "numpy" module. One approach might look more like this: > >import numpy # contains forward declarations >if TYPE_CHECKING: > import numpy.impl Wouldn't that be a massively breaking change? Anyone who does: from numpy import ndarray will get the forward-declared class object instead of the fully initialised class object, leading to all sorts of action-at-a-distance bugs. from numpy import ndarry a = ndarray([1, 2]) # Fails, because its a FDCO import some_random_module # which happens to import numpy.impl # which finishes the initialisation of the class object a = ndarray([1, 2]) # and this now magically works! -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5NPG3AG2GYDY7S3RBZD3XTUQDD7WQEQU/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes
On Fri, Apr 22, 2022 at 10:09:33PM -0700, Larry Hastings wrote: [Larry] > >>To be clear: `forward class` creates the official, actual class object. > >>Code that wants to take a reference to the class object may take > >>references > >>to the `forward class` declared class, and interact with it as normal. > >>However, a class created by `forward class` can't be *instantiated* > >>until after the matching `continue class` statement finishes. [Steve (me)] > >Since the "forward class" is a real class, > > It's a "forward-declared class object". It's the real class object, but > it hasn't been fully initialized yet, and won't be until the "continue > class" statement. The only thing that makes it not fully initialised is that it has a bozo bit dunder "__forward__" instructing the interpreter to disallow instantiation. Yes? If I take that class object created by `forward class X`, and delete the dunder, there is no difference between it and other regular classes. Am I correct? So your reasoning is circular: you give it a dunder marking it as a "forward-declared class object" to prevent it from being instantiated, but the only reason it can't be instantiated is that it has the dunder. I won't respond to the rest of your post until you have clarified the above, in case I have misunderstood. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/B7BZDPTWQTZ67PO7AB2BPOBJRDKAFCDZ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes
On 4/22/22 23:41, Mehdi2277 wrote: My main question for this approach is how would this work with type checkers? It would be new syntax for Python, so type checkers would have to understand it. Is there any restriction that forward class's continuation must appear in same module? No. If it's allowed that a forward class may be continued in a different module I do not see how type checker like mypy/pyright could handle that. Classes are generally viewed as closed and fully defined within type checker. Monkey patching at runtime later is not supported. If it became official Python syntax, I suspect they'd figure out a way to support it. They might require that the expression used in the "continue class" statement map to the original "forward class" declaration, e.g. they might stipulate that they don't support this: forward class X random_name = X continue class random_name: ... But rather than speculate further, perhaps someone who works on one of the static type analysis checkers will join the discussion and render an informed opinion about how easy or hard it would be to support "forward class" and "continue class". One other edge case here is how would you forward declare an annotation for a type that like this, if TYPE_CHECKING: import numpy def f(x: numpy.ndarray) -> None: ... forward declaring ndarray here would not make numpy.ndarray available. In this case, adding forward declarations for classes in "numpy" would be up to the "numpy" module. One approach might look more like this: import numpy # contains forward declarations if TYPE_CHECKING: import numpy.impl Though numpy presumably couldn't do this while they still supported older versions of Python. That's one downside of using new syntax--you can't use it until you stop support for old versions of Python that predate it. Would you forward declare modules? Is that allowed? I haven't proposed any syntax for forward-declaring modules, only classes. I'm confused in general how if TYPE_CHECKING issue is handled by this approach. Usually class being imported in those blocks is defined normally (without continue class) somewhere else. My proposal should mate well with "if TYPE_CHECKING". You would define your forward classes in a module that does get imported, but leave the continue classes in a separate module that is only imported "if TYPE_CHECKING", as per my example with "numpy" above. //arry/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/A7MW3B7JYM2LT4VUCAE7OUHJSYEX76G7/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Proto-PEP part 1: Forward declaration of classes
On Sat, Apr 23, 2022 at 06:41:23AM -, Mehdi2277 wrote: > My main question for this approach is how would this work with type > checkers? Is there any restriction that forward class's continuation > must appear in same module? If it's allowed that a forward class may > be continued in a different module I do not see how type checker like > mypy/pyright could handle that. Larry said that the name that follows `continue class` is an expression, so that something like this is allowed: import mymodule continue class mymodule.X: def method(self): pass so yes, you can continue classes in other modules. He said that could be used as a very primitive form of separation of interface and implementation, by putting the `forward class` in one module and the `continue` in another. -- Steve ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/N3YIF2FJARSE73M6VJX6UNU776FS6QC4/ Code of Conduct: http://python.org/psf/codeofconduct/
