[Tutor] question about metaclasses
hi pyguruscan you please tell me why we need metaclasses and how to use themthanks a lotAnil Do you Yahoo!? Everyone is raving about the all-new Yahoo! Mail Beta.___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
[Tutor] question about metaclasses
Hi,
In another thread on this list I was reminded of types.SimpleNamespace. This is
nice, but I wanted to create a bag class with constants that are read-only. My
main question is about example #3 below (example #2 just illustrates my thought
process). Is this a use case to a metaclass? Or can I do it some other way
(maybe a class decorator?). I would like to create a metaclass that converts
any non-special attributes (=not starting with '_') into properties, if needed.
That way I can specify my bag class in a very clean way: I only specify the
metaclass, and I list the attributes as normal attrbutes, because the metaclass
will convert them into properties.
Why does following the line (in #3) not convert the normal attribute into a
property?
setattr(cls, attr, property(lambda self: obj)) # incorrect!
# 1-
# nice, but I want the constants to be read-only
from types import SimpleNamespace
const = SimpleNamespace(YES=1, NO=0, DUNNO=9)
const.YES = 0
print(const)
# 2-
# works, but I wonder if there's a builtin way
class Const(object):
"""Adding attributes is ok, modifying them is not"""
YES = property(lambda self: 1)
NO = property(lambda self: 0)
DUNNO = property(lambda self: 42)
#THROWS_ERROR = 666
def __new__(cls):
for attr in dir(cls):
if attr.startswith('_'):
continue
elif not isinstance(getattr(cls, attr), property):
raise ValueError("Only properties allowed")
return super().__new__(cls)
def __repr__(self):
kv = ["%s=%s" % (attr, getattr(self, attr)) for \
attr in sorted(dir(Const)) if not attr.startswith('_')]
return "ReadOnlyNamespace(" + ", ".join(kv) + ")"
c = Const()
print(repr(c))
#c.YES = 42 # raises AttributeError (desired behavior)
print(c.YES)
# 3-
class Meta(type):
def __new__(cls, name, bases, attrs):
for attr, obj in attrs.items():
if attr.startswith('_'):
continue
elif not isinstance(obj, property):
import pdb;pdb.set_trace()
#setattr(cls, attr, property(lambda self: obj)) # incorrect!
raise ValueError("Only properties allowed")
return super().__new__(cls, name, bases, attrs)
class MyReadOnlyConst(metaclass=Meta):
__metaclass__ = Meta
YES = property(lambda self: 1)
NO = property(lambda self: 0)
DUNNO = property(lambda self: 42)
THROWS_ERROR = 666
c2 = MyReadOnlyConst()
print(c2.THROWS_ERROR)
#c2.THROWS_ERROR = 777
#print(c2.THROWS_ERROR)
Thank you in advance and sorry about the large amount of code!
Albert-Jan
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] question about metaclasses
anil maran wrote:
> hi pygurus
> can you please tell me why we need metaclasses and how to use them
Hmm...metaclasses are an advanced topic, first exposure to them usually
causes one's brain to explode. Fortunately the condition is only
temporary :-)
Basically a metaclass is the type of a class, or the type of a type.
Think about it this way - every object has a type. The type of 1 is int,
the type of 'a' is str.
In [16]: type(1)
Out[16]:
In [17]: type('a')
Out[17]:
Note that is just the printed representation of the type int:
In [19]: type(1) == int
Out[19]: True
In [20]: print int
But int and str are themselves objects - what is their type?
In [18]: type(int)
Out[18]:
In [21]: type(str)
Out[21]:
Why might you care? In general, it is the type of an object that
determines its behaviour. The behaviour of an int is determined by the
int type. What determines the behaviour of a class? Its type! So if you
want to customize the behaviour of a class, you create a custom metatype
for the class.
That is a very brief introduction. Here are some relatively introductory
articles. You can find more examples by searching the Python Cookbook
and comp.lang.python for "metaclass". Don't expect to understand this
the first time.
http://www-128.ibm.com/developerworks/linux/library/l-pymeta.html
http://www-128.ibm.com/developerworks/linux/library/l-pymeta2/
Here is Guido's brief explanation:
http://www.python.org/download/releases/2.2.3/descrintro/#metaclasses
Kent
___
Tutor maillist - Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] question about metaclasses
On Wed, Jan 10, 2018 at 04:08:04PM +, Albert-Jan Roskam wrote:
> In another thread on this list I was reminded of
> types.SimpleNamespace. This is nice, but I wanted to create a bag
> class with constants that are read-only.
If you expect to specify the names of the constants ahead of time, the
best solution is (I think) a namedtuple.
from collections import namedtuple
Bag = namedtuple('Bag', 'yes no dunno')
a = Bag(yes=1, no=0, dunno=42)
b = Bag(yes='okay', no='no way', dunno='not a clue')
ought to do what you want.
Don't make the mistake of doing this:
from collections import namedtuple
a = namedtuple('Bag', 'yes no dunno')(yes=1, no=0, dunno=42)
b = namedtuple('Bag', 'yes no dunno')(yes='okay', no='no way', dunno='not a
clue')
because that's quite wasteful of memory: each of a and b belong to a
separate hidden class, and classes are rather largish objects.
If you expect to be able to add new items on the fly, but have them
read-only once set, that's a different story.
--
Steve
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] question about metaclasses
On Wed, Jan 10, 2018 at 10:08 AM, Albert-Jan Roskam wrote: > Hi, > > > In another thread on this list I was reminded of types.SimpleNamespace. This > is nice, but I wanted to create a bag class with constants that are > read-only. My main question is about example #3 below (example #2 just > illustrates my thought process). Is this a use case to a metaclass? Or can I > do it some other way (maybe a class decorator?). I would like to create a > metaclass that converts any non-special attributes (=not starting with '_') > into properties, if needed. That way I can specify my bag class in a very > clean way: I only specify the metaclass, and I list the attributes as normal > attrbutes, because the metaclass will convert them into properties. You appear to be reimplementing Enum. > Why does following the line (in #3) not convert the normal attribute into a > property? > > setattr(cls, attr, property(lambda self: obj)) # incorrect! Because `cls` is `Meta`, not `MyReadOnlyConst`; `__new__` is implicitly a classmethod and `MyReadOnlyConst` doesn't actually exist yet. When `MyReadOnlyConst` is created by `type.__new__` it will be filled with the contents of `attrs`, so instead of `setattr` you want `attrs[attr] = property(...)`. But once you're past the learning exercise that this is, just use enum.Enum or collections.namedtuple :) -- Zach ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] question about metaclasses
Albert-Jan Roskam wrote:
> Why does following the line (in #3)
> # 3-
> class Meta(type):
> def __new__(cls, name, bases, attrs):
> for attr, obj in attrs.items():
> if attr.startswith('_'):
> continue
> elif not isinstance(obj, property):
> import pdb;pdb.set_trace()
> #setattr(cls, attr, property(lambda self: obj)) #
> #incorrect!
> raise ValueError("Only properties allowed")
> return super().__new__(cls, name, bases, attrs)
>
> class MyReadOnlyConst(metaclass=Meta):
> __metaclass__ = Meta
> YES = property(lambda self: 1)
> NO = property(lambda self: 0)
> DUNNO = property(lambda self: 42)
> THROWS_ERROR = 666
>
>
> c2 = MyReadOnlyConst()
> print(c2.THROWS_ERROR)
> #c2.THROWS_ERROR = 777
> #print(c2.THROWS_ERROR)
> not convert the normal attribute into
> a property?
>
> setattr(cls, attr, property(lambda self: obj)) # incorrect!
cls is Meta itself, not MyReadOnlyConst (which is an instance of Meta).
When the code in Meta.__new__() executes MyReadOnlyConst does not yet exist,
but future attributes are already there, in the form of the attrs dict.
Thus to convert the integer value into a read-only property you can
manipulate that dict (or the return value of super().__new__()):
class Meta(type):
def __new__(cls, name, bases, attrs):
for attr, obj in attrs.items():
if attr.startswith('_'):
continue
elif not isinstance(obj, property):
attrs[attr] = property(lambda self, obj=obj: obj)
return super().__new__(cls, name, bases, attrs)
class MyReadOnlyConst(metaclass=Meta):
YES = property(lambda self: 1)
NO = property(lambda self: 0)
DUNNO = property(lambda self: 42)
THROWS_ERROR = 666
c = MyReadOnlyConst()
try:
c.THROWS_ERROR = 42
except AttributeError:
pass
else:
assert False
assert c.THROWS_ERROR == 666
PS: If you don't remember why the obj=obj is necessary:
Python uses late binding; without that trick all lambda functions would
return the value bound to the obj name when the for loop has completed.
A simplified example:
>>> fs = [lambda: x for x in "abc"]
>>> fs[0](), fs[1](), fs[2]()
('c', 'c', 'c')
>>> fs = [lambda x=x: x for x in "abc"]
>>> fs[0](), fs[1](), fs[2]()
('a', 'b', 'c')
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] question about metaclasses
On Wed, Jan 10, 2018 at 07:29:58PM +0100, Peter Otten wrote: [...] > elif not isinstance(obj, property): > attrs[attr] = property(lambda self, obj=obj: obj) > PS: If you don't remember why the obj=obj is necessary: > Python uses late binding; without that trick all lambda functions would > return the value bound to the obj name when the for loop has completed. This is true, but I think your terminology is misleading. For default values to function parameters, Python uses *early* binding, not late binding: the default value is computed once at the time the function is created, not each time it is needed. So in this case, each of those property objects use a function that sets the default value of obj to the current value of obj at the time that the property is created. Without the obj=obj parameter, Python creates a *closure*. A closure is a computer-science term for something like a snap shot of the environment where the function was created. If we had written this instead: attrs[attr] = property(lambda self: obj) the name "obj" doesn't refer to a local variable of the lambda function. Nor does it refer to a global variable, or a builtin function. It refers to a "non-local variable": it belongs to the function that surrounds the lambda function, not the lambda itself. And so Python would create a *closure* for the lambda function so that when it eventually gets called, it knows where to find the value of obj. And *that* process, of looking up the value of obj from a closure, uses late binding: all the lambda functions will refer to the same environment, which means they will also see the same value for obj. Namely the last value obj received when the outer function (the one that the closure refers back to) completed. Here's another example to show the difference. Rather than use lambda, I'm going to use regular "def" to prove that this has nothing to do with lambda itself, the rules apply every time you create a function. Start with the closure version: # --- cut here %< --- def factory_closure(): # Create many new functions, each of which refer to i in its # enclosing scope. functions = [] for i in range(5): def f(): return i # << this i is a NONLOCAL variable functions.append(f) return functions functions = factory_closure() # All the closures refer to the same thing. for f in functions: print(f.__closure__) # And the functions all see the same value for i print([f() for f in functions]) # --- cut here %< --- And here is a version which avoids the closure issue by using the function parameter default value trick: # --- cut here %< --- def factory_no_closure(): # Create many new functions, each of which refer to i using # a parameter default value. functions = [] for i in range(5): def f(i=i): return i # << this i is a LOCAL variable functions.append(f) return functions functions = factory_no_closure() # None of the functions need a closure. for g in functions: print(g.__closure__) # And the functions all see different values for i print([g() for g in functions]) # --- cut here %< --- In practice, this is generally only an issue when single invocation of a factory function creates two or more functions at once, and that generally means inside a loop: def factory(): for i in something: create function referring to i If your factory only returns one function at a time, like this: def factory(i): create function referring to i for n in something: factory(n) then each function still uses a closure, but they are *different* closures because each one is created on a different invocation of the factory. That's another way to avoid this "early/late binding" problem. -- Steve ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] question about metaclasses
Steven D'Aprano wrote: > On Wed, Jan 10, 2018 at 07:29:58PM +0100, Peter Otten wrote: > > [...] >> elif not isinstance(obj, property): >> attrs[attr] = property(lambda self, obj=obj: obj) > >> PS: If you don't remember why the obj=obj is necessary: >> Python uses late binding; without that trick all lambda functions would >> return the value bound to the obj name when the for loop has completed. > > This is true, but I think your terminology is misleading. For default > values to function parameters, Python uses *early* binding, not late > binding: the default value is computed once at the time the function is > created, not each time it is needed. You are right; I should have stated clearly where I was talking about the closure. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] question about metaclasses
On Jan 10, 2018 18:57, Steven D'Aprano wrote:
>
> On Wed, Jan 10, 2018 at 04:08:04PM +, Albert-Jan Roskam wrote:
>
> > In another thread on this list I was reminded of
> > types.SimpleNamespace. This is nice, but I wanted to create a bag
> > class with constants that are read-only.
>
> If you expect to specify the names of the constants ahead of time, the
> best solution is (I think) a namedtuple.
Aaah *slaps forehead*, for some reason I didn't think about this, though I use
namedtuples quite often. Using a metaclass for the very first time was great
fun though :-)
> from collections import namedtuple
> Bag = namedtuple('Bag', 'yes no dunno')
> a = Bag(yes=1, no=0, dunno=42)
> b = Bag(yes='okay', no='no way', dunno='not a clue')
>
> ought to do what you want.
>
> Don't make the mistake of doing this:
>
> from collections import namedtuple
> a = namedtuple('Bag', 'yes no dunno')(yes=1, no=0, dunno=42)
> b = namedtuple('Bag', 'yes no dunno')(yes='okay', no='no way', dunno='not a
> clue')
But if I do:
Bag = namedtuple('Bag', 'yes no dunno')
... and then I create hundreds of Bag instances, this doesn't have a large
memory footprint, right? (Because of __slots__) Or is a regular tuple still
(much) less wasteful?
> because that's quite wasteful of memory: each of a and b belong to a
> separate hidden class, and classes are rather largish objects.
>
>
> If you expect to be able to add new items on the fly, but have them
> read-only once set, that's a different story.
>
>
> --
> Steve
> ___
> Tutor maillist - Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] question about metaclasses
On Jan 10, 2018 19:32, Peter Otten <__pete...@web.de> wrote:
>
> Albert-Jan Roskam wrote:
>
> > Why does following the line (in #3)
>
> > # 3-
> > class Meta(type):
> > def __new__(cls, name, bases, attrs):
> > for attr, obj in attrs.items():
> > if attr.startswith('_'):
> > continue
> > elif not isinstance(obj, property):
> > import pdb;pdb.set_trace()
> > #setattr(cls, attr, property(lambda self: obj)) #
> > #incorrect!
> > raise ValueError("Only properties allowed")
> > return super().__new__(cls, name, bases, attrs)
> >
> > class MyReadOnlyConst(metaclass=Meta):
> > __metaclass__ = Meta
> > YES = property(lambda self: 1)
> > NO = property(lambda self: 0)
> > DUNNO = property(lambda self: 42)
> > THROWS_ERROR = 666
> >
> >
> > c2 = MyReadOnlyConst()
> > print(c2.THROWS_ERROR)
> > #c2.THROWS_ERROR = 777
> > #print(c2.THROWS_ERROR)
>
> > not convert the normal attribute int > a property?
> >
> > setattr(cls, attr, property(lambda self: obj)) # incorrect!
>
> cls is Meta itself, not MyReadOnlyConst (which is an instance of Meta).
> When the code in Meta.__new__() executes MyReadOnlyConst does not yet exist,
> but future attributes are already there, in the form of the attrs dict.
> Thus to convert the integer value into a read-only property you can
> manipulate that dict (or the return value of super().__new__()):
>
> class Meta(type):
> def __new__(cls, name, bases, attrs):
> for attr, obj in attrs.items():
> if attr.startswith('_'):
> continue
> elif not isinstance(obj, property):
> attrs[attr] = property(lambda self, obj=obj: obj)
>
> return super().__new__(cls, name, bases, attrs)
>
> class MyReadOnlyConst(metaclass=Meta):
> YES = property(lambda self: 1)
> NO = property(lambda self: 0)
> DUNNO = property(lambda self: 42)
> THROWS_ERROR = 666
>
> c = MyReadOnlyConst()
> try:
> c.THROWS_ERROR = 42
> except AttributeError:
> pass
> else:
> assert False
> assert c.THROWS_ERROR == 666
Thanks all for your replies!
Awesome, this is exactly what I want. I think I'll also override __setattr__
so that each newly added attribute is automatically converted into a property
Is a metaclass the best/preferred/only way of doing this? Or is a class
decorator an alternative route?
Is the following analogy for doing stuff when a class is created ('born')
correct?
Metaclass --> prenatal surgery
__new__ --> perinatal surgery
Class decorator --> postnatal surgery
> PS: If you don't remember why the obj=obj is necessary:
> Python uses late binding; without that trick all lambda functions would
> return the value bound to the obj name when the for loop has completed.
> A simplified example:
>
> >>> fs = [lambda: x for x in "abc"]
> >>> fs[0](), fs[1](), fs[2]()
> ('c', 'c', 'c')
> >>> fs = [lambda x=x: x for x in "abc"]
> >>> fs[0](), fs[1](), fs[2]()
> ('a', 'b', 'c')
>
>
> ___
> Tutor maillist - Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] question about metaclasses
On Thu, Jan 18, 2018 at 05:31:24PM +, Albert-Jan Roskam wrote:
> > Don't make the mistake of doing this:
> >
> > from collections import namedtuple
> > a = namedtuple('Bag', 'yes no dunno')(yes=1, no=0, dunno=42)
> > b = namedtuple('Bag', 'yes no dunno')(yes='okay', no='no way', dunno='not a
> > clue')
>
> But if I do:
> Bag = namedtuple('Bag', 'yes no dunno')
> ... and then I create hundreds of Bag instances, this doesn't have a
> large memory footprint, right? (Because of __slots__) Or is a regular
> tuple still (much) less wasteful?
Correct.
namedtuple instances are *nearly* as compact as regular tuples. Making
many instances of the one named tuple class is efficient; making many
named tuple classes, with one instance each, is slow and wasteful of
memory.
--
Steve
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] question about metaclasses
On Thu, Jan 18, 2018 at 05:14:43PM +, Albert-Jan Roskam wrote:
> Is a metaclass the best/preferred/only way of doing this? Or is a
> class decorator an alternative route?
I haven't thought deeply about this, but I suspect a class decorator
should do the job too.
The general advice is to use the simplest thing that does the job. The
deeper you have to go into Python's scary internals, the better reason
you should have:
- ordinary Python code is simpler than...
- code using decorators, which is simpler than...
- classes using custom descriptors, which is simpler than...
- classes with __init_subclass__, which is simpler than...
- code with metaclasses.
So in general you should pick the first technique (starting at the top)
which solves your problem in a satisfactory manner.
The __init_subclass__ method is new to Python 3.6:
https://docs.python.org/3/reference/datamodel.html#object.__init_subclass__
Remember the excellent advice:
Debugging is harder than programming, so if you write the trickiest and
most clever code you are capable of, by definition you won't be able to
debug it.
Having said that, I applaud you for investigating metaclasses!
> Is the following analogy for doing stuff when a class is created ('born')
> correct?
> Metaclass --> prenatal surgery
> __new__ --> perinatal surgery
> Class decorator --> postnatal surgery
The last one is certainly correct: a class decorator gets called with
the fully created class object, and then has the opportunity to modify
it as needed.
It isn't clear to me what you mean by __new__. Do you mean the *class*
__new__ method or the *metaclass* __new__ method?
If I write this:
class MetaC(type):
def __new__(meta, *args):
# pretend this does something useful
class C(metaclass=MetaC):
def __new__(cls, *args):
...
then MetaC.__new__ and C.__new__ are called at very different times.
MetaC.__new__ is called as part of the process of creating C in the
first place; so at the beginning of MetaC.__new__, C does not exist.
C.__new__ on the other hand doesn't get called when creating C, it gets
called when creating instances of C. Think of __new__ as the twin of
__init__, with the following differences:
* __new__ is called before the instance ("self") exists,
and gets to customise the creation of the instance;
* __init__ is called after the instance is created, and
gets to modify self provided self is mutable.
So writing an __init__ method for (say) a string subclass is usually a
waste of time, since the string is already set and cannot be changed.
This does not work:
class UpperString(str):
# make strings uppercase
def __init__(self):
self = self.upper()
But this does:
class UpperString(str):
def __new__(cls, arg):
arg = str(arg).upper()
return super().__new__(cls, arg)
Metaclasses are both extremely powerful *and* mind-blowing (back in
Python 1.5, they were known as "the Killer Joke") because they can
customise (nearly?) any part of the class operation. So you can think of
a metaclass as all of the following:
- prenatal surgery (metaclass.__new__);
- postnatal surgery (metaclass.__init__);
- cybernetic implants (metaclass.__getattribute__, __getattr__ etc);
- mind-meld or telepathy between otherwise unrelated classes
(define methods in the metaclass as a kind of alternative to
inheritence);
- gene therapy (all of the above?);
- (I can't think of a good analogy for this one): metaclass.__call__
lets you customized what happens when you call your class, allowing
you to manipulate the arguments or the returned instance.
For example:
class Meta(type):
def __call__(self, *args):
print("called by", self)
instance = super().__call__(*args)
instance.foo = 1
return instance
prints a message and adds a "foo" attribute to every instance of every
class that belongs to it.
Oh, and just to make it more fun... metaclasses don't have to be a
class! See if you can predict what this will do:
def meta(*args):
print("Hi there, I'm your metaclass for today, how can I blow your mind?")
return 42
class A(metaclass=meta):
pass
See also The Killer Joke:
https://www.python.org/doc/essays/metaclasses/
That was written back in Python 1.5 days, so a lot of the information in
it is now obsolete. But the basic techniques should more or less work
today.
--
Steve
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
