[Tutor] question about metaclasses
hi pyguruscan you please tell me why we need metaclasses and how to use themthanks a lotAnil Do you Yahoo!? Everyone is raving about the all-new Yahoo! Mail Beta.___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
[Tutor] question about metaclasses
Hi, In another thread on this list I was reminded of types.SimpleNamespace. This is nice, but I wanted to create a bag class with constants that are read-only. My main question is about example #3 below (example #2 just illustrates my thought process). Is this a use case to a metaclass? Or can I do it some other way (maybe a class decorator?). I would like to create a metaclass that converts any non-special attributes (=not starting with '_') into properties, if needed. That way I can specify my bag class in a very clean way: I only specify the metaclass, and I list the attributes as normal attrbutes, because the metaclass will convert them into properties. Why does following the line (in #3) not convert the normal attribute into a property? setattr(cls, attr, property(lambda self: obj)) # incorrect! # 1- # nice, but I want the constants to be read-only from types import SimpleNamespace const = SimpleNamespace(YES=1, NO=0, DUNNO=9) const.YES = 0 print(const) # 2- # works, but I wonder if there's a builtin way class Const(object): """Adding attributes is ok, modifying them is not""" YES = property(lambda self: 1) NO = property(lambda self: 0) DUNNO = property(lambda self: 42) #THROWS_ERROR = 666 def __new__(cls): for attr in dir(cls): if attr.startswith('_'): continue elif not isinstance(getattr(cls, attr), property): raise ValueError("Only properties allowed") return super().__new__(cls) def __repr__(self): kv = ["%s=%s" % (attr, getattr(self, attr)) for \ attr in sorted(dir(Const)) if not attr.startswith('_')] return "ReadOnlyNamespace(" + ", ".join(kv) + ")" c = Const() print(repr(c)) #c.YES = 42 # raises AttributeError (desired behavior) print(c.YES) # 3- class Meta(type): def __new__(cls, name, bases, attrs): for attr, obj in attrs.items(): if attr.startswith('_'): continue elif not isinstance(obj, property): import pdb;pdb.set_trace() #setattr(cls, attr, property(lambda self: obj)) # incorrect! raise ValueError("Only properties allowed") return super().__new__(cls, name, bases, attrs) class MyReadOnlyConst(metaclass=Meta): __metaclass__ = Meta YES = property(lambda self: 1) NO = property(lambda self: 0) DUNNO = property(lambda self: 42) THROWS_ERROR = 666 c2 = MyReadOnlyConst() print(c2.THROWS_ERROR) #c2.THROWS_ERROR = 777 #print(c2.THROWS_ERROR) Thank you in advance and sorry about the large amount of code! Albert-Jan ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] question about metaclasses
anil maran wrote: > hi pygurus > can you please tell me why we need metaclasses and how to use them Hmm...metaclasses are an advanced topic, first exposure to them usually causes one's brain to explode. Fortunately the condition is only temporary :-) Basically a metaclass is the type of a class, or the type of a type. Think about it this way - every object has a type. The type of 1 is int, the type of 'a' is str. In [16]: type(1) Out[16]: In [17]: type('a') Out[17]: Note that is just the printed representation of the type int: In [19]: type(1) == int Out[19]: True In [20]: print int But int and str are themselves objects - what is their type? In [18]: type(int) Out[18]: In [21]: type(str) Out[21]: Why might you care? In general, it is the type of an object that determines its behaviour. The behaviour of an int is determined by the int type. What determines the behaviour of a class? Its type! So if you want to customize the behaviour of a class, you create a custom metatype for the class. That is a very brief introduction. Here are some relatively introductory articles. You can find more examples by searching the Python Cookbook and comp.lang.python for "metaclass". Don't expect to understand this the first time. http://www-128.ibm.com/developerworks/linux/library/l-pymeta.html http://www-128.ibm.com/developerworks/linux/library/l-pymeta2/ Here is Guido's brief explanation: http://www.python.org/download/releases/2.2.3/descrintro/#metaclasses Kent ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] question about metaclasses
On Wed, Jan 10, 2018 at 04:08:04PM +, Albert-Jan Roskam wrote: > In another thread on this list I was reminded of > types.SimpleNamespace. This is nice, but I wanted to create a bag > class with constants that are read-only. If you expect to specify the names of the constants ahead of time, the best solution is (I think) a namedtuple. from collections import namedtuple Bag = namedtuple('Bag', 'yes no dunno') a = Bag(yes=1, no=0, dunno=42) b = Bag(yes='okay', no='no way', dunno='not a clue') ought to do what you want. Don't make the mistake of doing this: from collections import namedtuple a = namedtuple('Bag', 'yes no dunno')(yes=1, no=0, dunno=42) b = namedtuple('Bag', 'yes no dunno')(yes='okay', no='no way', dunno='not a clue') because that's quite wasteful of memory: each of a and b belong to a separate hidden class, and classes are rather largish objects. If you expect to be able to add new items on the fly, but have them read-only once set, that's a different story. -- Steve ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] question about metaclasses
On Wed, Jan 10, 2018 at 10:08 AM, Albert-Jan Roskam wrote: > Hi, > > > In another thread on this list I was reminded of types.SimpleNamespace. This > is nice, but I wanted to create a bag class with constants that are > read-only. My main question is about example #3 below (example #2 just > illustrates my thought process). Is this a use case to a metaclass? Or can I > do it some other way (maybe a class decorator?). I would like to create a > metaclass that converts any non-special attributes (=not starting with '_') > into properties, if needed. That way I can specify my bag class in a very > clean way: I only specify the metaclass, and I list the attributes as normal > attrbutes, because the metaclass will convert them into properties. You appear to be reimplementing Enum. > Why does following the line (in #3) not convert the normal attribute into a > property? > > setattr(cls, attr, property(lambda self: obj)) # incorrect! Because `cls` is `Meta`, not `MyReadOnlyConst`; `__new__` is implicitly a classmethod and `MyReadOnlyConst` doesn't actually exist yet. When `MyReadOnlyConst` is created by `type.__new__` it will be filled with the contents of `attrs`, so instead of `setattr` you want `attrs[attr] = property(...)`. But once you're past the learning exercise that this is, just use enum.Enum or collections.namedtuple :) -- Zach ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] question about metaclasses
Albert-Jan Roskam wrote: > Why does following the line (in #3) > # 3- > class Meta(type): > def __new__(cls, name, bases, attrs): > for attr, obj in attrs.items(): > if attr.startswith('_'): > continue > elif not isinstance(obj, property): > import pdb;pdb.set_trace() > #setattr(cls, attr, property(lambda self: obj)) # > #incorrect! > raise ValueError("Only properties allowed") > return super().__new__(cls, name, bases, attrs) > > class MyReadOnlyConst(metaclass=Meta): > __metaclass__ = Meta > YES = property(lambda self: 1) > NO = property(lambda self: 0) > DUNNO = property(lambda self: 42) > THROWS_ERROR = 666 > > > c2 = MyReadOnlyConst() > print(c2.THROWS_ERROR) > #c2.THROWS_ERROR = 777 > #print(c2.THROWS_ERROR) > not convert the normal attribute into > a property? > > setattr(cls, attr, property(lambda self: obj)) # incorrect! cls is Meta itself, not MyReadOnlyConst (which is an instance of Meta). When the code in Meta.__new__() executes MyReadOnlyConst does not yet exist, but future attributes are already there, in the form of the attrs dict. Thus to convert the integer value into a read-only property you can manipulate that dict (or the return value of super().__new__()): class Meta(type): def __new__(cls, name, bases, attrs): for attr, obj in attrs.items(): if attr.startswith('_'): continue elif not isinstance(obj, property): attrs[attr] = property(lambda self, obj=obj: obj) return super().__new__(cls, name, bases, attrs) class MyReadOnlyConst(metaclass=Meta): YES = property(lambda self: 1) NO = property(lambda self: 0) DUNNO = property(lambda self: 42) THROWS_ERROR = 666 c = MyReadOnlyConst() try: c.THROWS_ERROR = 42 except AttributeError: pass else: assert False assert c.THROWS_ERROR == 666 PS: If you don't remember why the obj=obj is necessary: Python uses late binding; without that trick all lambda functions would return the value bound to the obj name when the for loop has completed. A simplified example: >>> fs = [lambda: x for x in "abc"] >>> fs[0](), fs[1](), fs[2]() ('c', 'c', 'c') >>> fs = [lambda x=x: x for x in "abc"] >>> fs[0](), fs[1](), fs[2]() ('a', 'b', 'c') ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] question about metaclasses
On Wed, Jan 10, 2018 at 07:29:58PM +0100, Peter Otten wrote: [...] > elif not isinstance(obj, property): > attrs[attr] = property(lambda self, obj=obj: obj) > PS: If you don't remember why the obj=obj is necessary: > Python uses late binding; without that trick all lambda functions would > return the value bound to the obj name when the for loop has completed. This is true, but I think your terminology is misleading. For default values to function parameters, Python uses *early* binding, not late binding: the default value is computed once at the time the function is created, not each time it is needed. So in this case, each of those property objects use a function that sets the default value of obj to the current value of obj at the time that the property is created. Without the obj=obj parameter, Python creates a *closure*. A closure is a computer-science term for something like a snap shot of the environment where the function was created. If we had written this instead: attrs[attr] = property(lambda self: obj) the name "obj" doesn't refer to a local variable of the lambda function. Nor does it refer to a global variable, or a builtin function. It refers to a "non-local variable": it belongs to the function that surrounds the lambda function, not the lambda itself. And so Python would create a *closure* for the lambda function so that when it eventually gets called, it knows where to find the value of obj. And *that* process, of looking up the value of obj from a closure, uses late binding: all the lambda functions will refer to the same environment, which means they will also see the same value for obj. Namely the last value obj received when the outer function (the one that the closure refers back to) completed. Here's another example to show the difference. Rather than use lambda, I'm going to use regular "def" to prove that this has nothing to do with lambda itself, the rules apply every time you create a function. Start with the closure version: # --- cut here %< --- def factory_closure(): # Create many new functions, each of which refer to i in its # enclosing scope. functions = [] for i in range(5): def f(): return i # << this i is a NONLOCAL variable functions.append(f) return functions functions = factory_closure() # All the closures refer to the same thing. for f in functions: print(f.__closure__) # And the functions all see the same value for i print([f() for f in functions]) # --- cut here %< --- And here is a version which avoids the closure issue by using the function parameter default value trick: # --- cut here %< --- def factory_no_closure(): # Create many new functions, each of which refer to i using # a parameter default value. functions = [] for i in range(5): def f(i=i): return i # << this i is a LOCAL variable functions.append(f) return functions functions = factory_no_closure() # None of the functions need a closure. for g in functions: print(g.__closure__) # And the functions all see different values for i print([g() for g in functions]) # --- cut here %< --- In practice, this is generally only an issue when single invocation of a factory function creates two or more functions at once, and that generally means inside a loop: def factory(): for i in something: create function referring to i If your factory only returns one function at a time, like this: def factory(i): create function referring to i for n in something: factory(n) then each function still uses a closure, but they are *different* closures because each one is created on a different invocation of the factory. That's another way to avoid this "early/late binding" problem. -- Steve ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] question about metaclasses
Steven D'Aprano wrote: > On Wed, Jan 10, 2018 at 07:29:58PM +0100, Peter Otten wrote: > > [...] >> elif not isinstance(obj, property): >> attrs[attr] = property(lambda self, obj=obj: obj) > >> PS: If you don't remember why the obj=obj is necessary: >> Python uses late binding; without that trick all lambda functions would >> return the value bound to the obj name when the for loop has completed. > > This is true, but I think your terminology is misleading. For default > values to function parameters, Python uses *early* binding, not late > binding: the default value is computed once at the time the function is > created, not each time it is needed. You are right; I should have stated clearly where I was talking about the closure. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] question about metaclasses
On Jan 10, 2018 18:57, Steven D'Aprano wrote: > > On Wed, Jan 10, 2018 at 04:08:04PM +, Albert-Jan Roskam wrote: > > > In another thread on this list I was reminded of > > types.SimpleNamespace. This is nice, but I wanted to create a bag > > class with constants that are read-only. > > If you expect to specify the names of the constants ahead of time, the > best solution is (I think) a namedtuple. Aaah *slaps forehead*, for some reason I didn't think about this, though I use namedtuples quite often. Using a metaclass for the very first time was great fun though :-) > from collections import namedtuple > Bag = namedtuple('Bag', 'yes no dunno') > a = Bag(yes=1, no=0, dunno=42) > b = Bag(yes='okay', no='no way', dunno='not a clue') > > ought to do what you want. > > Don't make the mistake of doing this: > > from collections import namedtuple > a = namedtuple('Bag', 'yes no dunno')(yes=1, no=0, dunno=42) > b = namedtuple('Bag', 'yes no dunno')(yes='okay', no='no way', dunno='not a > clue') But if I do: Bag = namedtuple('Bag', 'yes no dunno') ... and then I create hundreds of Bag instances, this doesn't have a large memory footprint, right? (Because of __slots__) Or is a regular tuple still (much) less wasteful? > because that's quite wasteful of memory: each of a and b belong to a > separate hidden class, and classes are rather largish objects. > > > If you expect to be able to add new items on the fly, but have them > read-only once set, that's a different story. > > > -- > Steve > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] question about metaclasses
On Jan 10, 2018 19:32, Peter Otten <__pete...@web.de> wrote: > > Albert-Jan Roskam wrote: > > > Why does following the line (in #3) > > > # 3- > > class Meta(type): > > def __new__(cls, name, bases, attrs): > > for attr, obj in attrs.items(): > > if attr.startswith('_'): > > continue > > elif not isinstance(obj, property): > > import pdb;pdb.set_trace() > > #setattr(cls, attr, property(lambda self: obj)) # > > #incorrect! > > raise ValueError("Only properties allowed") > > return super().__new__(cls, name, bases, attrs) > > > > class MyReadOnlyConst(metaclass=Meta): > > __metaclass__ = Meta > > YES = property(lambda self: 1) > > NO = property(lambda self: 0) > > DUNNO = property(lambda self: 42) > > THROWS_ERROR = 666 > > > > > > c2 = MyReadOnlyConst() > > print(c2.THROWS_ERROR) > > #c2.THROWS_ERROR = 777 > > #print(c2.THROWS_ERROR) > > > not convert the normal attribute int > a property? > > > > setattr(cls, attr, property(lambda self: obj)) # incorrect! > > cls is Meta itself, not MyReadOnlyConst (which is an instance of Meta). > When the code in Meta.__new__() executes MyReadOnlyConst does not yet exist, > but future attributes are already there, in the form of the attrs dict. > Thus to convert the integer value into a read-only property you can > manipulate that dict (or the return value of super().__new__()): > > class Meta(type): > def __new__(cls, name, bases, attrs): > for attr, obj in attrs.items(): > if attr.startswith('_'): > continue > elif not isinstance(obj, property): > attrs[attr] = property(lambda self, obj=obj: obj) > > return super().__new__(cls, name, bases, attrs) > > class MyReadOnlyConst(metaclass=Meta): > YES = property(lambda self: 1) > NO = property(lambda self: 0) > DUNNO = property(lambda self: 42) > THROWS_ERROR = 666 > > c = MyReadOnlyConst() > try: > c.THROWS_ERROR = 42 > except AttributeError: > pass > else: > assert False > assert c.THROWS_ERROR == 666 Thanks all for your replies! Awesome, this is exactly what I want. I think I'll also override __setattr__ so that each newly added attribute is automatically converted into a property Is a metaclass the best/preferred/only way of doing this? Or is a class decorator an alternative route? Is the following analogy for doing stuff when a class is created ('born') correct? Metaclass --> prenatal surgery __new__ --> perinatal surgery Class decorator --> postnatal surgery > PS: If you don't remember why the obj=obj is necessary: > Python uses late binding; without that trick all lambda functions would > return the value bound to the obj name when the for loop has completed. > A simplified example: > > >>> fs = [lambda: x for x in "abc"] > >>> fs[0](), fs[1](), fs[2]() > ('c', 'c', 'c') > >>> fs = [lambda x=x: x for x in "abc"] > >>> fs[0](), fs[1](), fs[2]() > ('a', 'b', 'c') > > > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] question about metaclasses
On Thu, Jan 18, 2018 at 05:31:24PM +, Albert-Jan Roskam wrote: > > Don't make the mistake of doing this: > > > > from collections import namedtuple > > a = namedtuple('Bag', 'yes no dunno')(yes=1, no=0, dunno=42) > > b = namedtuple('Bag', 'yes no dunno')(yes='okay', no='no way', dunno='not a > > clue') > > But if I do: > Bag = namedtuple('Bag', 'yes no dunno') > ... and then I create hundreds of Bag instances, this doesn't have a > large memory footprint, right? (Because of __slots__) Or is a regular > tuple still (much) less wasteful? Correct. namedtuple instances are *nearly* as compact as regular tuples. Making many instances of the one named tuple class is efficient; making many named tuple classes, with one instance each, is slow and wasteful of memory. -- Steve ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] question about metaclasses
On Thu, Jan 18, 2018 at 05:14:43PM +, Albert-Jan Roskam wrote: > Is a metaclass the best/preferred/only way of doing this? Or is a > class decorator an alternative route? I haven't thought deeply about this, but I suspect a class decorator should do the job too. The general advice is to use the simplest thing that does the job. The deeper you have to go into Python's scary internals, the better reason you should have: - ordinary Python code is simpler than... - code using decorators, which is simpler than... - classes using custom descriptors, which is simpler than... - classes with __init_subclass__, which is simpler than... - code with metaclasses. So in general you should pick the first technique (starting at the top) which solves your problem in a satisfactory manner. The __init_subclass__ method is new to Python 3.6: https://docs.python.org/3/reference/datamodel.html#object.__init_subclass__ Remember the excellent advice: Debugging is harder than programming, so if you write the trickiest and most clever code you are capable of, by definition you won't be able to debug it. Having said that, I applaud you for investigating metaclasses! > Is the following analogy for doing stuff when a class is created ('born') > correct? > Metaclass --> prenatal surgery > __new__ --> perinatal surgery > Class decorator --> postnatal surgery The last one is certainly correct: a class decorator gets called with the fully created class object, and then has the opportunity to modify it as needed. It isn't clear to me what you mean by __new__. Do you mean the *class* __new__ method or the *metaclass* __new__ method? If I write this: class MetaC(type): def __new__(meta, *args): # pretend this does something useful class C(metaclass=MetaC): def __new__(cls, *args): ... then MetaC.__new__ and C.__new__ are called at very different times. MetaC.__new__ is called as part of the process of creating C in the first place; so at the beginning of MetaC.__new__, C does not exist. C.__new__ on the other hand doesn't get called when creating C, it gets called when creating instances of C. Think of __new__ as the twin of __init__, with the following differences: * __new__ is called before the instance ("self") exists, and gets to customise the creation of the instance; * __init__ is called after the instance is created, and gets to modify self provided self is mutable. So writing an __init__ method for (say) a string subclass is usually a waste of time, since the string is already set and cannot be changed. This does not work: class UpperString(str): # make strings uppercase def __init__(self): self = self.upper() But this does: class UpperString(str): def __new__(cls, arg): arg = str(arg).upper() return super().__new__(cls, arg) Metaclasses are both extremely powerful *and* mind-blowing (back in Python 1.5, they were known as "the Killer Joke") because they can customise (nearly?) any part of the class operation. So you can think of a metaclass as all of the following: - prenatal surgery (metaclass.__new__); - postnatal surgery (metaclass.__init__); - cybernetic implants (metaclass.__getattribute__, __getattr__ etc); - mind-meld or telepathy between otherwise unrelated classes (define methods in the metaclass as a kind of alternative to inheritence); - gene therapy (all of the above?); - (I can't think of a good analogy for this one): metaclass.__call__ lets you customized what happens when you call your class, allowing you to manipulate the arguments or the returned instance. For example: class Meta(type): def __call__(self, *args): print("called by", self) instance = super().__call__(*args) instance.foo = 1 return instance prints a message and adds a "foo" attribute to every instance of every class that belongs to it. Oh, and just to make it more fun... metaclasses don't have to be a class! See if you can predict what this will do: def meta(*args): print("Hi there, I'm your metaclass for today, how can I blow your mind?") return 42 class A(metaclass=meta): pass See also The Killer Joke: https://www.python.org/doc/essays/metaclasses/ That was written back in Python 1.5 days, so a lot of the information in it is now obsolete. But the basic techniques should more or less work today. -- Steve ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor