TL;DR: We need improved documentation of the way meta-classes behave for generic classes, and possibly reconsider the way "__setattr__" and "__getattribute__" behave for such classes.
I am using meta-programming pretty heavily in one of my projects. It took me a while to figure out the dance between meta-classes and generic classes in Python 3.6.0. I couldn't find good documentation for any of this (if anyone has a good link, please share...), but with a liberal use of "print" I managed to reverse engineer how this works. The behavior isn't intuitive but I can understand the motivation (basically, "type annotations shall not change the behavior of the program"). For the uninitiated: * It turns out that there are two kinds of instances of generic classes: the "unspecialized" class (basically ignoring type parameters), and "specialized" classes (created when you write "Foo[Bar]", which know the type parameters, "Bar" in this case). * This means the meta-class "__new__" method is called sometimes to create the unspecialized class, and sometimes to create a specialized one - in the latter case, it is called with different arguments... * No object is actually an instance of the specialized class; that is, the "__class__" of an instance of "Foo[Bar]" is actually the unspecialized "Foo" (which means you can't get the type parameters by looking at an instance of a generic class). So far, so good, sort of. I implemented my meta-classes to detect whether they are creating a "specialized" or "unspecialized" class and behave accordingly. However, these meta-classes stopped working when switching to Python 3.6.1. The reason is that in Python 3.6.1, a "__setattr__" implementation was added to "GenericMeta", which redirects the setting of an attribute of a specialized class instance to set the attribute of the unspecialized class instance instead. This causes code such as the following (inside the meta-class) to behave in a mighty confusing way: if is-not-specialized: cls._my_attribute = False else: # Is specialized: cls._my_attribute = True assert cls._my_attribute # Fails! As you can imagine, this caused us some wailing and gnashing of teeth, until we figured out (1) that this was the problem and (2) why it was happening. Looking into the source code in "typing.py", I see that I am not the only one who had this problem. Specifically, the implementers of the "abc" module had the exact same problem. Their solution was simple: the "GenericMeta.__setattr__" code explicitly tests whether the attribute name starts with "_abc_", in which case it maintains the old behavior. Obviously, I should not patch the standard library typing.py to preserve "_my_attribute". My current workaround is to derive from GenericMeta, define my own "__setattr__", which preserves the old behavior for "_my_attribute", and use that instead of the standard GenericMeta everywhere. My code now works in both 3.6.0 and 3.6.1. However, I think the following points are worth fixing and/or discussion: * This is a breaking change, but it isn't listed in https://www.python.org/downloads/release/python-361/ - it should probably be listed there. * In general it would be good to have some documentation on the way that meta-classes and generic classes interact with each other, as part of the standard library documentation (apologies if it is there and I missed it... link?) * I'm not convinced the new behavior is a better default. I don't recall seeing a discussion about making this change, possibly I missed it (link?) * There is a legitimate need for the old behavior (normal per-instance attributes). For example, it is needed by the "abc" module (as well as my project). So, some mechanism should be recommended (in the documentation) for people who need the old behavior. * Separating between "really per instance" attributes and "forwarded to the unspecialized instance" attributes based on their prefix seems to violate "explicit is better than implicit". For example, it would have been explicit to say "cls.__unspecialized__.attribute" (other explicit mechanisms are possible). * Perhaps the whole notion of specialized vs. unspecialized class instances needs to be made more explicit in the GenericMeta API... * Finally and IMVHO most importantly, it is *very* confusing to override "__setattr__" and not override "__getattribute__" to match. This gives rise to code like "cls._foo = True; assert cls._foo" failing. This feels wrong.... And presumably fixing the implementation so that "__getattribute__" forwards the same set of attributes to the "unspecialized" instance wouldn't break any code... Other than code that already broken due to the new functionality, that is. -- https://mail.python.org/mailman/listinfo/python-list