Re: [Tutor] set locals
On 12/18/2013 09:45 PM, Alan Gauld wrote: On 18/12/13 17:45, Mark Lawrence wrote: Can I be so bold as to ask how discussing metaclasses and __setattr__ on a tutor mailing list is going to help the newbie who's having problems with their "hello world" program? It won't, but the tutor list is also for experienced programmers new to Python, so it might help there. Although I do grant it's a tad more exotic (I nearly said erotic!) than our usual fare. And I'd probably expect to see it on the main Python list. But IMO it is within the group's remit - just! But there is the equally valid counterpoint that overly exotic discussion here can turn away the real newbies who are our bread and butter audience. You are right. Actually, I preferred to susbscribe to tutor (1) because there's far too much traffic on the main python list (2) because other people's questions help and exploring python further (or remember forgotten points) (3) I can at times help myself, which is also enjoyable. Thus, in fact, in practice I consider tutor as if it were a mutual help list rather than dedicated to novice (python) programmers (but I do know this is what it is intended to be). So, sorry for the occasionnal e(x|r)otic stuff; and if you ask to stop it, I won't protest. (You may even ask for it off-list, Alan, if you prefer not to pollute the list more with meta-threads.) Denis ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] set locals
On Wed, Dec 18, 2013 at 6:16 AM, spir wrote: > On 12/18/2013 12:07 PM, eryksun wrote: >> >> You need __setattr__ from the metaclass: >> >> >>> class C: pass >> ... >> >>> type(C).__setattr__(C, "baz", "BAZ") >> >>> C.baz >> 'BAZ' > > Oh, that makes sense: so, __setattr__ on a class is for its instances, > right? (thus, to set attrs on a class itself, we need ___setattr__ from its > own class, if I understand rightly?) Note that vars(type)['__dict__'] is a data descriptor, so the lookup gives it precedence. There actually is a vars(C)['__dict__'], the descriptor meant for instances of C, but the lookup stops before it gets there. On the other hand, vars(type)['__setattr__'] is a non-data descriptor (i.e. no __set__ method), so the lookup for C.__setattr__ skips it to bind and return vars(object)['__setattr__'] instead. That's akin to shadowing a method with instance data. For example: >>> c = C() >>> c.__setattr__ = 'spam' >>> vars(c) {'__setattr__': 'spam'} >>> c.__setattr__ 'spam' Here are 2 reasons why object.__setattr__ is wrong for a CPython type object: type.__setattr__ includes a check to prevent setting and deleting attributes of built-in types. type.__setattr__ updates the affected slot (C struct field) in the type and its subclasses. This also invalidates the method cache for the affected types. For example, if you just modify a type's dict entry for '__len__', this doesn't update the sq_length and mp_length slots in the PyHeapTypeObject, so built-in len() called on an instance will raise a TypeError. Hence also the reason that modifying the dict directly is disallowed by the __dict__ proxy. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] set locals
On Wed, Dec 18, 2013 at 05:45:02PM +, Mark Lawrence wrote: > Can I be so bold as to ask how discussing metaclasses and __setattr__ on > a tutor mailing list is going to help the newbie who's having problems > with their "hello world" program? It's not just newbies who need tutoring. Sometimes I've learned things here too. -- Steven ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] set locals
On 18/12/13 17:45, Mark Lawrence wrote: Can I be so bold as to ask how discussing metaclasses and __setattr__ on a tutor mailing list is going to help the newbie who's having problems with their "hello world" program? It won't, but the tutor list is also for experienced programmers new to Python, so it might help there. Although I do grant it's a tad more exotic (I nearly said erotic!) than our usual fare. And I'd probably expect to see it on the main Python list. But IMO it is within the group's remit - just! But there is the equally valid counterpoint that overly exotic discussion here can turn away the real newbies who are our bread and butter audience. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] set locals
On 18/12/2013 11:16, spir wrote: On 12/18/2013 12:07 PM, eryksun wrote: On Wed, Dec 18, 2013 at 5:40 AM, spir wrote: C.__setattr__(C, "baz", "BAZ") which fails, for any reason, with TypeError: can't apply this __setattr__ to type object You need __setattr__ from the metaclass: >>> class C: pass ... >>> type(C).__setattr__(C, "baz", "BAZ") >>> C.baz 'BAZ' Oh, that makes sense: so, __setattr__ on a class is for its instances, right? (thus, to set attrs on a class itself, we need ___setattr__ from its own class, if I understand rightly?) You can bind the method like this: >>> C_setattr = type(C).__setattr__.__get__(C) >>> C_setattr('foo', 'bar') >>> C.foo 'bar' But just use built-in setattr(), really. Really, yes! Denis Can I be so bold as to ask how discussing metaclasses and __setattr__ on a tutor mailing list is going to help the newbie who's having problems with their "hello world" program? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] set locals
On 12/18/2013 11:51 AM, eryksun wrote: On Tue, Dec 17, 2013 at 10:52 AM, spir wrote: is it at all possible to set new vars (or any symbol) into an existing scope (typically locals())? scope[name] = value raises by me an error like: TypeError: 'mappingproxy' object does not support item assignment I guess 'mappingproxy' is the implementation name of a scope (here, local), and I thought scopes were just dicts; so what is the issue? Do you see an alternative? For a CPython function, built-in locals() creates/updates the frame's f_locals mapping against the fast locals by calling the C function PyFrame_FastToLocals. The reverse update can be done (but why?) with the help of an extension module or ctypes to call PyFrame_LocalsToFast. For example: import sys from ctypes import * pythonapi.PyFrame_LocalsToFast.argtypes = [py_object, c_int] def f(x, clear=0): if not clear: locals()['x'] += 1 else: del locals()['x'] frame = sys._getframe() pythonapi.PyFrame_LocalsToFast(frame, clear) return x >>> f(1) 2 If cleared this way (akin to `del x`), the fast local for x is set to NULL (unbound), so trying to return it raises an UnboundLocalError: >>> f(1, clear=1) Traceback (most recent call last): File "", line 1, in File "", line 9, in f UnboundLocalError: local variable 'x' referenced before assignment There's no facilitated way to add new fast locals. The memory used for the frame's stack, fast locals, and closure cells is allocated when the frame is instantiated, based on attributes of the compiled code object. On the other hand, a class body is unoptimized (intentionally), so it uses the frame's f_locals mapping (that's a dict, unless you're using the PEP 3115 __prepare__ hook). The populated mapping gets passed to the metaclass __new__ (e.g. type.__new__), which copies it to a new dict for the class. All right, I don't understand the details (not knowing anything about CPython's implentation internals), but the general scheme is clear enough, thank you! You can use locals() to your heart's content in a class body: class Cls: locals()['x'] = 'spam' >>> Cls.x 'spam' All right. So, when a user defines a grammar (parser) inside a class (as I like to do myself) they could call my utility naming func directly from _inside_ the class def. class some_parser: # no capital, it's a parser, not an actual class digit = Range("09") # lots of other pattern defs Pattern.name(locals()) I'll try it to see if setting inside a class's locals works fine... works! So, apparently, I don't need to special-case classes anymore, do I? I still ask for your advice because, since I don't get all details, despite my quick trial I'm not 100% sure there aren't cases where it would not work. I just need to register copies of patterns in the given scope/namespace in case they already have a non-None .name attribute. A class's __dict__ attribute is a descriptor defined by the metaclass. In CPython, for example, this descriptor needs to know the offset into the PyTypeObject where the dict is stored. The code it calls is simple enough to include here: static PyObject * type_dict(PyTypeObject *type, void *context) { if (type->tp_dict == NULL) { Py_INCREF(Py_None); return Py_None; } return PyDictProxy_New(type->tp_dict); } If you aren't familiar with descriptors, read the following: http://docs.python.org/3/howto/descriptor.html And try the following: class Cls(object, metaclass=type): pass type_dict_descr = vars(type)['__dict__'] type_dict_descr.__get__(Cls, type) Calling the __get__ method eventually makes its way to the above C function type_dict, which creates the mappingproxy by calling PyDictProxy_New. The proxy wraps the type's dict to keep you from hacking it directly. The functions to do so aren't even defined (i.e. mp_ass_subscript and sq_ass_item). So the abstract C API function PyObject_SetItem just raises a TypeError. If you need to set attributes dynamically, use built-in setattr(). I have been familiar with Python metaclasses some years ago (used them to simulate toy languages in python itself as interpretor! ;-) including one prototype-based à la Lua, Self, Io or JS) but this did not give me any precise clue about the other side of the business, in particular descriptors. I remember however having stepped on them once or twice, but at the time I knew too few about language implementations in C to be able to get anything. I'll have another look at them following your links, thank you very much, "eryksun"! denis ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] set locals
On 12/18/2013 12:07 PM, eryksun wrote: On Wed, Dec 18, 2013 at 5:40 AM, spir wrote: C.__setattr__(C, "baz", "BAZ") which fails, for any reason, with TypeError: can't apply this __setattr__ to type object You need __setattr__ from the metaclass: >>> class C: pass ... >>> type(C).__setattr__(C, "baz", "BAZ") >>> C.baz 'BAZ' Oh, that makes sense: so, __setattr__ on a class is for its instances, right? (thus, to set attrs on a class itself, we need ___setattr__ from its own class, if I understand rightly?) You can bind the method like this: >>> C_setattr = type(C).__setattr__.__get__(C) >>> C_setattr('foo', 'bar') >>> C.foo 'bar' But just use built-in setattr(), really. Really, yes! Denis ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] set locals
On Wed, Dec 18, 2013 at 5:40 AM, spir wrote: > C.__setattr__(C, "baz", "BAZ") > which fails, for any reason, with > TypeError: can't apply this __setattr__ to type object You need __setattr__ from the metaclass: >>> class C: pass ... >>> type(C).__setattr__(C, "baz", "BAZ") >>> C.baz 'BAZ' You can bind the method like this: >>> C_setattr = type(C).__setattr__.__get__(C) >>> C_setattr('foo', 'bar') >>> C.foo 'bar' But just use built-in setattr(), really. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] set locals
On Tue, Dec 17, 2013 at 10:52 AM, spir wrote: > is it at all possible to set new vars (or any symbol) into an existing scope > (typically locals())? > > scope[name] = value > raises by me an error like: > TypeError: 'mappingproxy' object does not support item assignment > > I guess 'mappingproxy' is the implementation name of a scope (here, local), > and I thought scopes were just dicts; so what is the issue? Do you see an > alternative? For a CPython function, built-in locals() creates/updates the frame's f_locals mapping against the fast locals by calling the C function PyFrame_FastToLocals. The reverse update can be done (but why?) with the help of an extension module or ctypes to call PyFrame_LocalsToFast. For example: import sys from ctypes import * pythonapi.PyFrame_LocalsToFast.argtypes = [py_object, c_int] def f(x, clear=0): if not clear: locals()['x'] += 1 else: del locals()['x'] frame = sys._getframe() pythonapi.PyFrame_LocalsToFast(frame, clear) return x >>> f(1) 2 If cleared this way (akin to `del x`), the fast local for x is set to NULL (unbound), so trying to return it raises an UnboundLocalError: >>> f(1, clear=1) Traceback (most recent call last): File "", line 1, in File "", line 9, in f UnboundLocalError: local variable 'x' referenced before assignment There's no facilitated way to add new fast locals. The memory used for the frame's stack, fast locals, and closure cells is allocated when the frame is instantiated, based on attributes of the compiled code object. On the other hand, a class body is unoptimized (intentionally), so it uses the frame's f_locals mapping (that's a dict, unless you're using the PEP 3115 __prepare__ hook). The populated mapping gets passed to the metaclass __new__ (e.g. type.__new__), which copies it to a new dict for the class. You can use locals() to your heart's content in a class body: class Cls: locals()['x'] = 'spam' >>> Cls.x 'spam' A class's __dict__ attribute is a descriptor defined by the metaclass. In CPython, for example, this descriptor needs to know the offset into the PyTypeObject where the dict is stored. The code it calls is simple enough to include here: static PyObject * type_dict(PyTypeObject *type, void *context) { if (type->tp_dict == NULL) { Py_INCREF(Py_None); return Py_None; } return PyDictProxy_New(type->tp_dict); } If you aren't familiar with descriptors, read the following: http://docs.python.org/3/howto/descriptor.html And try the following: class Cls(object, metaclass=type): pass type_dict_descr = vars(type)['__dict__'] type_dict_descr.__get__(Cls, type) Calling the __get__ method eventually makes its way to the above C function type_dict, which creates the mappingproxy by calling PyDictProxy_New. The proxy wraps the type's dict to keep you from hacking it directly. The functions to do so aren't even defined (i.e. mp_ass_subscript and sq_ass_item). So the abstract C API function PyObject_SetItem just raises a TypeError. If you need to set attributes dynamically, use built-in setattr(). ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] set locals
On 12/18/2013 10:02 AM, Peter Otten wrote: spir wrote: [...] Like Steven I have no idea how you produced the mappingproxy. Are you trying to use a class as a namespace (in Python 3.3)? class A: pass ... A.__dict__["foo"] = "bar" Traceback (most recent call last): File "", line 1, in TypeError: 'mappingproxy' object does not support item assignment I have no idea neither, for me it's just a term in an error message. Here is a minimal reduction of the context, that produces similar outcome (py 3.3): EDIT: you are right about class dicts, see below. def set_var(scope, id, value): scope[id] = value set_var(locals(), "foo", 'FOO') assert(foo == 'FOO')# OK bar = 'bar' set_var(globals(), "bar", 'BAR') assert(bar == 'BAR')# OK class C: pass set_var(C.__dict__, "baz", 'BAZ') # ==> TypeError: 'mappingproxy' object does not support item assignment assert(C.baz == 'BAZ') # statement never reached C.baz = "trial to define symbol first" set_var(C.__dict__, "baz", 'BAZ') # ==> TypeError: 'mappingproxy' object does not support item assignment assert(C.baz == 'BAZ') # statement never reached It is effectively the case that I like to use classes as namespaces (to define grammars / parsers): they are highly practicle, one can even have computation in their def! They're kind of sub-modules, in fact; free scopes. So, it was actually so that the error happened in such a case; and thus "mappingproxy" seems to be an implementation term for a class's dict, or so. This does explain why we cannot modify them (as we can do it in plain python code, but symbol names must be constant). But I have no solution for my issue: there is no way to write: c.id = value with id beeing a variable. Or is there? As fat as I know I really need to do c.__dict__[id_var] = value EDIT: works!!! I tried C.__setattr__(C, "baz", "BAZ") which fails, for any reason, with TypeError: can't apply this __setattr__ to type object But the direct way works fine, for any reason: setattr(C, "baz", "BAZ") Thus we can modify set_var like eg: def set_var(scope, id, value): if isinstance(scope, type): setattr(scope, id, value) else: scope[id] = value Which then enables: class C: pass set_var(C, "baz", 'BAZ') assert(C.baz == 'BAZ') # OK === For the record, here in (the code of) my final tool func: def name (scope): # name all defined pattern in given scope # Note: if scope is a class, there is a distinction between the scope # and the actual dict we traverse (c.__dict__): for any reason, we # cannot set entries in a class's dict directly, but must use setattr. dic = scope.__dict__ if isinstance(scope, type) else scope # all patterns in scope (in dic, in fact): names_pats = ((name, pat) for (name, pat) in dic.items() \ if isinstance(pat, Pattern)) for (name, pat) in names_pats: # If the original pat is already named, leave it as is and # instead create a copy with the new name: if pat.name: new_pat = copy(pat) new_pat.name = name # give it its name if isinstance(scope, type): # case scope is a class setattr(scope, name, new_pat) else: # case scope is module or local scope[name] = new_pat # Else, this is a new pattern, which just demands a name: else: pat.name = name # Special case of recursive pattern, # also name its actual matching pattern: if isinstance(pat, Recurs): pat.pattern.name = name Issue solved ;-). Thank you vey much, Denis ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] set locals
spir wrote: > Hello, > > is it at all possible to set new vars (or any symbol) into an existing > scope (typically locals())? locals() normally contains a copy of the current namespace as a dict. Setting items is possible but only alters the dict and has no effect on the original namespace: >>> def f(): ... x = 1 ... namespace = locals() ... print(namespace) ... namespace["x"] = 2 ... print(namespace) ... print(x) ... >>> f() {'x': 1} {'x': 2} 1 On the module level the local is identical with the global namespace, and you can define variables with >>> x Traceback (most recent call last): File "", line 1, in NameError: name 'x' is not defined >>> locals()["x"] = 42 >>> x 42 In Python 2 you can introduce local variables with exec: >>> def f(): ... exec 'foo = "bar"' ... print foo ... >>> f() bar In Python 3 that is no longer possible: >>> def f(): ... exec('foo = "bar"') ... print(locals()) ... print(eval("foo")) ... print(foo) ... >>> f() {'foo': 'bar'} bar Traceback (most recent call last): File "", line 1, in File "", line 5, in f NameError: name 'foo' is not defined > scope[name] = value > raises by me an error like: > TypeError: 'mappingproxy' object does not support item assignment > > I guess 'mappingproxy' is the implementation name of a scope (here, > local), and I thought scopes were just dicts; so what is the issue? Do you > see an alternative? Like Steven I have no idea how you produced the mappingproxy. Are you trying to use a class as a namespace (in Python 3.3)? >>> class A: pass ... >>> A.__dict__["foo"] = "bar" Traceback (most recent call last): File "", line 1, in TypeError: 'mappingproxy' object does not support item assignment ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] set locals
On Tue, Dec 17, 2013 at 04:52:25PM +0100, spir wrote: > Hello, > > is it at all possible to set new vars (or any symbol) into an existing > scope (typically locals())? In general, no. The only time that writing to locals() is guaranteed to work is when you are in the top-level scope and locals returns the same dict as globals(). > scope[name] = value > raises by me an error like: > TypeError: 'mappingproxy' object does not support item assignment > > I guess 'mappingproxy' is the implementation name of a scope (here, local), I cannot reproduce locals() returning a mappingproxy. What version of Python are you using, and how did you generate scope? -- Steven ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor