Re: Overriding iadd for dictionary like objects
In article <7f82416a-53be-41b3-9503-1492454cc...@upsg2000gro.googlegroups.com>, RunThePun wrote: >On Sep 1, 3:00=A0am, a...@pythoncraft.com (Aahz) wrote: >> In article .com>, >> RunThePun =A0 wrote: >>>On Aug 30, 10:33=3DA0pm, a...@pythoncraft.com (Aahz) wrote: In article ups=3D >>>.com>, RunThePun =3DA0 wrote: >> >I made a DictMixin where the keys are filenames and the values are the >file contents. It was very simple and easy to do thanks to DictMixin. >> >For example this code writes "abc" in a file named "temp.txt" and >prints the contents of the file named "swallow", these files are >looked up/created/deleted in the directory "spam": d =3D3D3D FilesDict('spam') d['temp.txt'] =3D3D3D 'abc' print(d['swallow']) >> >My problem arose when I wanted to append a string to a file which >using open(..., 'ab') would have been miles more efficient because I >wouldn't have to read the entire file (__getitem__) and then write the >entire file back (__setitem__). The files are expected to be as big as >600 KB which will be appended 30 bytes at a time about 3 times a >second. Performance-wise the system would probably work without open >(..., 'ab') but it would be a real thrashing so the current solution >uses a method "AddTo" as Robert suggested, sacrificing the neat >getitem/setitem syntax. >> You can do mostly what you want, I think, by having __setitem__() convert string values into FileProxy() objects that have an appropriat= >e __iadd__() method. =3DA0That brings a whole new set of problems, of co= >urse. >> >>>I'm guessing you meant __getitem__, which is what Jan Kaliszewski >>>suggested, but as you noted, would be a bit cumbersome in this case. >> >> Actually, what I meant was __setitem__. =A0The idea is that you create th= >e >> proxy item when you add the data to the dict (wrapping the original >> data), and the proxy has an __iadd__ method, which would allow you to do >> the file append. > >But you do mean that __getitem__ would return a wrapped object as >well, right? Otherwise I don't see how the iadd would be relevant >because: > d['a'] +=3D 3 >is equivalent to: > d.__setitem__('a', d.__getitem__('a').__iadd__(3)) The __getitem__ returns the proxy object created by the first __setitem__ when you did d['a'] = 'something' The __iadd__ of the proxy object returns self, so it's the same object, which means that the second __setitem__ (called by +=) can check to see that you already have a proxy object and it does not need to rewrap it. If this still doesn't make sense, I suggest you go ahead and try the experiment to prove that it does work. ;-) -- Aahz (a...@pythoncraft.com) <*> http://www.pythoncraft.com/ "Look, it's your affair if you want to play with five people, but don't go calling it doubles." --John Cleese anticipates Usenet -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
On Sep 1, 3:00 am, a...@pythoncraft.com (Aahz) wrote: > In article > , > > > > > > RunThePun wrote: > >On Aug 30, 10:33=A0pm, a...@pythoncraft.com (Aahz) wrote: > >> In article >.com>, > >> RunThePun =A0 wrote: > > >>>I made a DictMixin where the keys are filenames and the values are the > >>>file contents. It was very simple and easy to do thanks to DictMixin. > > >>>For example this code writes "abc" in a file named "temp.txt" and > >>>prints the contents of the file named "swallow", these files are > >>>looked up/created/deleted in the directory "spam": > >> d =3D3D FilesDict('spam') > >> d['temp.txt'] =3D3D 'abc' > >> print(d['swallow']) > > >>>My problem arose when I wanted to append a string to a file which > >>>using open(..., 'ab') would have been miles more efficient because I > >>>wouldn't have to read the entire file (__getitem__) and then write the > >>>entire file back (__setitem__). The files are expected to be as big as > >>>600 KB which will be appended 30 bytes at a time about 3 times a > >>>second. Performance-wise the system would probably work without open > >>>(..., 'ab') but it would be a real thrashing so the current solution > >>>uses a method "AddTo" as Robert suggested, sacrificing the neat > >>>getitem/setitem syntax. > > >> You can do mostly what you want, I think, by having __setitem__() > >> convert string values into FileProxy() objects that have an appropriate > >> __iadd__() method. =A0That brings a whole new set of problems, of course. > > >I'm guessing you meant __getitem__, which is what Jan Kaliszewski > >suggested, but as you noted, would be a bit cumbersome in this case. > > Actually, what I meant was __setitem__. The idea is that you create the > proxy item when you add the data to the dict (wrapping the original > data), and the proxy has an __iadd__ method, which would allow you to do > the file append. > -- > Aahz (a...@pythoncraft.com) <*> http://www.pythoncraft.com/ > > "I support family values -- Addams family values" --www.nancybuttons.com But you do mean that __getitem__ would return a wrapped object as well, right? Otherwise I don't see how the iadd would be relevant because: d['a'] += 3 is equivalent to: d.__setitem__('a', d.__getitem__('a').__iadd__(3)) -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
In article , RunThePun wrote: >On Aug 30, 10:33=A0pm, a...@pythoncraft.com (Aahz) wrote: >> In article .com>, >> RunThePun =A0 wrote: >>> >>>I made a DictMixin where the keys are filenames and the values are the >>>file contents. It was very simple and easy to do thanks to DictMixin. >>> >>>For example this code writes "abc" in a file named "temp.txt" and >>>prints the contents of the file named "swallow", these files are >>>looked up/created/deleted in the directory "spam": >> d =3D3D FilesDict('spam') >> d['temp.txt'] =3D3D 'abc' >> print(d['swallow']) >>> >>>My problem arose when I wanted to append a string to a file which >>>using open(..., 'ab') would have been miles more efficient because I >>>wouldn't have to read the entire file (__getitem__) and then write the >>>entire file back (__setitem__). The files are expected to be as big as >>>600 KB which will be appended 30 bytes at a time about 3 times a >>>second. Performance-wise the system would probably work without open >>>(..., 'ab') but it would be a real thrashing so the current solution >>>uses a method "AddTo" as Robert suggested, sacrificing the neat >>>getitem/setitem syntax. >> >> You can do mostly what you want, I think, by having __setitem__() >> convert string values into FileProxy() objects that have an appropriate >> __iadd__() method. =A0That brings a whole new set of problems, of course. > >I'm guessing you meant __getitem__, which is what Jan Kaliszewski >suggested, but as you noted, would be a bit cumbersome in this case. Actually, what I meant was __setitem__. The idea is that you create the proxy item when you add the data to the dict (wrapping the original data), and the proxy has an __iadd__ method, which would allow you to do the file append. -- Aahz (a...@pythoncraft.com) <*> http://www.pythoncraft.com/ "I support family values -- Addams family values" --www.nancybuttons.com -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
On Aug 30, 10:33 pm, a...@pythoncraft.com (Aahz) wrote: > In article > , > > > > > > RunThePun wrote: > > >I made a DictMixin where the keys are filenames and the values are the > >file contents. It was very simple and easy to do thanks to DictMixin. > > >For example this code writes "abc" in a file named "temp.txt" and > >prints the contents of the file named "swallow", these files are > >looked up/created/deleted in the directory "spam": > d =3D FilesDict('spam') > d['temp.txt'] =3D 'abc' > print(d['swallow']) > > >My problem arose when I wanted to append a string to a file which > >using open(..., 'ab') would have been miles more efficient because I > >wouldn't have to read the entire file (__getitem__) and then write the > >entire file back (__setitem__). The files are expected to be as big as > >600 KB which will be appended 30 bytes at a time about 3 times a > >second. Performance-wise the system would probably work without open > >(..., 'ab') but it would be a real thrashing so the current solution > >uses a method "AddTo" as Robert suggested, sacrificing the neat > >getitem/setitem syntax. > > You can do mostly what you want, I think, by having __setitem__() > convert string values into FileProxy() objects that have an appropriate > __iadd__() method. That brings a whole new set of problems, of course. > -- > Aahz (a...@pythoncraft.com) <*> http://www.pythoncraft.com/ > > "I support family values -- Addams family values" --www.nancybuttons.com I'm guessing you meant __getitem__, which is what Jan Kaliszewski suggested, but as you noted, would be a bit cumbersome in this case. -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
In article , RunThePun wrote: > >I made a DictMixin where the keys are filenames and the values are the >file contents. It was very simple and easy to do thanks to DictMixin. > >For example this code writes "abc" in a file named "temp.txt" and >prints the contents of the file named "swallow", these files are >looked up/created/deleted in the directory "spam": d =3D FilesDict('spam') d['temp.txt'] =3D 'abc' print(d['swallow']) > >My problem arose when I wanted to append a string to a file which >using open(..., 'ab') would have been miles more efficient because I >wouldn't have to read the entire file (__getitem__) and then write the >entire file back (__setitem__). The files are expected to be as big as >600 KB which will be appended 30 bytes at a time about 3 times a >second. Performance-wise the system would probably work without open >(..., 'ab') but it would be a real thrashing so the current solution >uses a method "AddTo" as Robert suggested, sacrificing the neat >getitem/setitem syntax. You can do mostly what you want, I think, by having __setitem__() convert string values into FileProxy() objects that have an appropriate __iadd__() method. That brings a whole new set of problems, of course. -- Aahz (a...@pythoncraft.com) <*> http://www.pythoncraft.com/ "I support family values -- Addams family values" --www.nancybuttons.com -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
PS. Sorry for sending 2 posts -- the latter is the correct one. Cheers, *j -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
On Aug 29, 1:58 pm, Carl Banks wrote: > On Aug 28, 10:37 pm, Joshua Judson Rosen wrote: > > > > > > > Carl Banks writes: > > > > On Aug 28, 2:42 pm, Terry Reedy wrote: > > > > > Carl Banks wrote: > > > > > I don't think it needs a syntax for that, but I'm not so sure a method > > > > > to modify a value in place with a single key lookup wouldn't > > > > > occasioanally be useful. > > > > > Augmented assignment does that. > > > > Internally uses two lookups, one for getting, and one for setting. > > > > I think this is an unavoidable given Python's semantics. Look at > > > the traceback: > > > > >>> def x(): > > > ... d['a'] += 1 > > > ... > > > >>> dis.dis(x) > > > 2 0 LOAD_GLOBAL 0 (d) > > > 3 LOAD_CONST 1 ('a') > > > 6 DUP_TOPX 2 > > > 9 BINARY_SUBSCR > > > OK, there's one lookup, but... > > > > 10 LOAD_CONST 2 (1) > > > 13 INPLACE_ADD > > > 14 ROT_THREE > > > 15 STORE_SUBSCR > > > 16 LOAD_CONST 0 (None) > > > 19 RETURN_VALUE > > > ... I don't see anything in there that retrieves the value a second time > > STORE_SUBSCR has to look up the position in the hash table to store > the value, hence the second lookup. > > > > > > As a workaround, if lookups are expensive, > > > > > But they are not. Because (C)Python is heavily based on dict name lookup > > > > for builtins and global names and attributes, as well as overt dict > > > > lookup, must effort has gone into optimizing dict lookup. > > > > The actual lookup algorithm Python dicts use is well-optimized, yes, > > > but the dict could contain keys that have expensive comparison and > > > hash-code calculation, in which case lookup is going to be slow. > > > I'll like the originator correct me if I've made a mistake, but I read > > "lookup" as actually meaning "lookup", not "value-comparison". > > This has nothing to do with value comparison. I was talking about key > comparison, which happens when looking up a position in a hash table. > I was the first person to use the word "lookup" in this thread and I > specifically meant hash-table position lookup. > > > At least in part because the question, as it was posed, specifically > > related to a wrapper-class (providing a mapping ("dict like") interface) > > around a database of some sort other than Python's dict class per se. > > > How do the details of Python's native dict-type's internal (hashtable) > > algorithm matter when they're explicitly /not/ being used? > > Well it doesn't apply specifically to the OP's problem. I changed the > topic a bit by making it specific to dicts. Is that ok with you? Was > that not allowed? > > The OP can add a method like apply_to_value to his own class, but one > can't do that for dicts. Ergo why something like apply_to_value() > would be useful enough in rare circumstances where lookup is very slow > to merit a moments consideration before being rejected. > > (If dict did have a method like that, the OP would at least know which > method to override.) > > Carl Banks First of all I'd like to say thanks for this discussion, you guys are awesome. I probably should have explained my problem better to begin with and I apologize for that. So now I'll start from the top: I made a DictMixin where the keys are filenames and the values are the file contents. It was very simple and easy to do thanks to DictMixin. For example this code writes "abc" in a file named "temp.txt" and prints the contents of the file named "swallow", these files are looked up/created/deleted in the directory "spam": >>> d = FilesDict('spam') >>> d['temp.txt'] = 'abc' >>> print(d['swallow']) This was very convenient for me because I wanted to use a simple DB which could be read and edited by shell scripts and non-pythonistas, without the heavy ORM. Also, if in the future an online full featured DB would be needed, I could easily convert the DictMixin methods. So up to here I had a good solution. My problem arose when I wanted to append a string to a file which using open(..., 'ab') would have been miles more efficient because I wouldn't have to read the entire file (__getitem__) and then write the entire file back (__setitem__). The files are expected to be as big as 600 KB which will be appended 30 bytes at a time about 3 times a second. Performance-wise the system would probably work without open (..., 'ab') but it would be a real thrashing so the current solution uses a method "AddTo" as Robert suggested, sacrificing the neat getitem/setitem syntax. Just so I don't leave out any information, I actually also made a DirectoryDict for having multiple 'tables'. In this DictMixin, keys are directory names and values are FilesDict instances. So to write "European or African" to the file "/root/tmp/velocity" one would be just: >>> d = DirectoryDict("/root") >>> d["tmp"]["velocity"]
Re: Overriding iadd for dictionary like objects
On Aug 28, 10:37 pm, Joshua Judson Rosen wrote: > Carl Banks writes: > > > On Aug 28, 2:42 pm, Terry Reedy wrote: > > > > Carl Banks wrote: > > > > I don't think it needs a syntax for that, but I'm not so sure a method > > > > to modify a value in place with a single key lookup wouldn't > > > > occasioanally be useful. > > > > Augmented assignment does that. > > > Internally uses two lookups, one for getting, and one for setting. > > > I think this is an unavoidable given Python's semantics. Look at > > the traceback: > > > >>> def x(): > > ... d['a'] += 1 > > ... > > >>> dis.dis(x) > > 2 0 LOAD_GLOBAL 0 (d) > > 3 LOAD_CONST 1 ('a') > > 6 DUP_TOPX 2 > > 9 BINARY_SUBSCR > > OK, there's one lookup, but... > > > 10 LOAD_CONST 2 (1) > > 13 INPLACE_ADD > > 14 ROT_THREE > > 15 STORE_SUBSCR > > 16 LOAD_CONST 0 (None) > > 19 RETURN_VALUE > > ... I don't see anything in there that retrieves the value a second time STORE_SUBSCR has to look up the position in the hash table to store the value, hence the second lookup. > > > > As a workaround, if lookups are expensive, > > > > But they are not. Because (C)Python is heavily based on dict name lookup > > > for builtins and global names and attributes, as well as overt dict > > > lookup, must effort has gone into optimizing dict lookup. > > > The actual lookup algorithm Python dicts use is well-optimized, yes, > > but the dict could contain keys that have expensive comparison and > > hash-code calculation, in which case lookup is going to be slow. > > I'll like the originator correct me if I've made a mistake, but I read > "lookup" as actually meaning "lookup", not "value-comparison". This has nothing to do with value comparison. I was talking about key comparison, which happens when looking up a position in a hash table. I was the first person to use the word "lookup" in this thread and I specifically meant hash-table position lookup. > At least in part because the question, as it was posed, specifically > related to a wrapper-class (providing a mapping ("dict like") interface) > around a database of some sort other than Python's dict class per se. > > How do the details of Python's native dict-type's internal (hashtable) > algorithm matter when they're explicitly /not/ being used? Well it doesn't apply specifically to the OP's problem. I changed the topic a bit by making it specific to dicts. Is that ok with you? Was that not allowed? The OP can add a method like apply_to_value to his own class, but one can't do that for dicts. Ergo why something like apply_to_value() would be useful enough in rare circumstances where lookup is very slow to merit a moments consideration before being rejected. (If dict did have a method like that, the OP would at least know which method to override.) Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
On Sat, 29 Aug 2009 01:23:59 -0400, Joshua Judson Rosen wrote: > Robert Kern writes: >> >> On 2009-08-28 16:42 PM, Terry Reedy wrote: >> > Carl Banks wrote: >> > >> > > I don't think it needs a syntax for that, but I'm not so sure a >> > > method to modify a value in place with a single key lookup wouldn't >> > > occasioanally be useful. >> > >> > Augmented assignment does that. >> >> No, it uses one __getitem__ and one __setitem__ thus two key lookups. > > Apparently you're defining "key lookup" some other way than as `what > __getitem__ does'. > > What exactly does "key lookup" mean to you? > > I've always understood it as `retrieving the value associated with a > key', which obviously isn't required for assignment--otherwise it > wouldn't be possible to add new keys to a mapping. When you retrieve a value from a dictionary using __getitem__, e.g.: dict["K"] the dict has to search the hash table for the record with key "K". This a key lookup. (I use the term "search", but of course for hash tables this is usually very fast. For Python dicts, you can assume it will usually be a constant time, independent of the key or the size of the dict.) When you store a value in a dictionary using __setitem__, e.g.: dict["K"] = 42 the dict has to search the hash table for the correct place to store a record with key "K". It obviously can't place the record in some arbitrary place, it has to be in the correct place for future lookups to find it. This is also a key lookup. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
Carl Banks writes: > > On Aug 28, 2:42 pm, Terry Reedy wrote: > > > Carl Banks wrote: > > > I don't think it needs a syntax for that, but I'm not so sure a method > > > to modify a value in place with a single key lookup wouldn't > > > occasioanally be useful. > > > > Augmented assignment does that. > > Internally uses two lookups, one for getting, and one for setting. > > I think this is an unavoidable given Python's semantics. Look at > the traceback: > > > >>> def x(): > ... d['a'] += 1 > ... > >>> dis.dis(x) > 2 0 LOAD_GLOBAL 0 (d) > 3 LOAD_CONST 1 ('a') > 6 DUP_TOPX 2 > 9 BINARY_SUBSCR OK, there's one lookup, but... > 10 LOAD_CONST 2 (1) > 13 INPLACE_ADD > 14 ROT_THREE > 15 STORE_SUBSCR > 16 LOAD_CONST 0 (None) > 19 RETURN_VALUE ... I don't see anything in there that retrieves the value a second time > > > As a workaround, if lookups are expensive, > > > > But they are not. Because (C)Python is heavily based on dict name lookup > > for builtins and global names and attributes, as well as overt dict > > lookup, must effort has gone into optimizing dict lookup. > > The actual lookup algorithm Python dicts use is well-optimized, yes, > but the dict could contain keys that have expensive comparison and > hash-code calculation, in which case lookup is going to be slow. I'll like the originator correct me if I've made a mistake, but I read "lookup" as actually meaning "lookup", not "value-comparison". At least in part because the question, as it was posed, specifically related to a wrapper-class (providing a mapping ("dict like") interface) around a database of some sort other than Python's dict class per se. How do the details of Python's native dict-type's internal (hashtable) algorithm matter when they're explicitly /not/ being used? -- Don't be afraid to ask (Lf.((Lx.xx) (Lr.f(rr. -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
Robert Kern writes: > > On 2009-08-28 16:42 PM, Terry Reedy wrote: > > Carl Banks wrote: > > > > > I don't think it needs a syntax for that, but I'm not so sure a > > > method to modify a value in place with a single key lookup > > > wouldn't occasioanally be useful. > > > > Augmented assignment does that. > > No, it uses one __getitem__ and one __setitem__ thus two key lookups. Apparently you're defining "key lookup" some other way than as `what __getitem__ does'. What exactly does "key lookup" mean to you? I've always understood it as `retrieving the value associated with a key', which obviously isn't required for assignment--otherwise it wouldn't be possible to add new keys to a mapping. -- Don't be afraid to ask (Lf.((Lx.xx) (Lr.f(rr. -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
On Aug 28, 2:42 pm, Terry Reedy wrote: > Carl Banks wrote: > > I don't think it needs a syntax for that, but I'm not so sure a method > > to modify a value in place with a single key lookup wouldn't > > occasioanally be useful. > > Augmented assignment does that. Internally uses two lookups, one for getting, and one for setting. I think this is an unavoidable given Python's semantics. Look at the traceback: >>> def x(): ... d['a'] += 1 ... >>> dis.dis(x) 2 0 LOAD_GLOBAL 0 (d) 3 LOAD_CONST 1 ('a') 6 DUP_TOPX 2 9 BINARY_SUBSCR 10 LOAD_CONST 2 (1) 13 INPLACE_ADD 14 ROT_THREE 15 STORE_SUBSCR 16 LOAD_CONST 0 (None) 19 RETURN_VALUE > > As a workaround, if lookups are expensive, > > But they are not. Because (C)Python is heavily based on dict name lookup > for builtins and global names and attributes, as well as overt dict > lookup, must effort has gone into optimizing dict lookup. The actual lookup algorithm Python dicts use is well-optimized, yes, but the dict could contain keys that have expensive comparison and hash-code calculation, in which case lookup is going to be slow. Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
On 2009-08-28 16:42 PM, Terry Reedy wrote: Carl Banks wrote: I don't think it needs a syntax for that, but I'm not so sure a method to modify a value in place with a single key lookup wouldn't occasioanally be useful. Augmented assignment does that. No, it uses one __getitem__ and one __setitem__ thus two key lookups. For instance: def increment(value): return value+1 d = { 'a': 1 } d.apply_to_value('a',increment) print d and d['a'] would be 2. The difference is that only one lookup occurs. Like this? >>> d={'a': 2} >>> d['a'] += 2 >>> d['a'] 4 This does not cover all replacements, but certainly the most common. Take look farther up in the thread for the actual point at issue. The OP knows that augmented assignment works for the common cases. He wants to override the behavior on the container to handle some uncommon cases, and this is not possible because Python breaks up the operation into three orthogonal actions. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
Carl Banks wrote: I don't think it needs a syntax for that, but I'm not so sure a method to modify a value in place with a single key lookup wouldn't occasioanally be useful. Augmented assignment does that. For instance: def increment(value): return value+1 d = { 'a': 1 } d.apply_to_value('a',increment) print d and d['a'] would be 2. The difference is that only one lookup occurs. Like this? >>> d={'a': 2} >>> d['a'] += 2 >>> d['a'] 4 This does not cover all replacements, but certainly the most common. [snip] As a workaround, if lookups are expensive, But they are not. Because (C)Python is heavily based on dict name lookup for builtins and global names and attributes, as well as overt dict lookup, must effort has gone into optimizing dict lookup. > you can add something even slower ;-). In particular, a method lookup + method call, as you suggest above. One can always avoid calculating the key object twice if that is expensive. Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
On Aug 26, 11:49 pm, RunThePun wrote: > Anybody have any more ideas? I think python should/could havev a > syntax for overriding this behaviour, i mean, obviously the complexity > of supporting all operators with the getitem syntax could introduce > alot of clutter. But maybe there's an elegant solution out there... I don't think it needs a syntax for that, but I'm not so sure a method to modify a value in place with a single key lookup wouldn't occasioanally be useful. For instance: def increment(value): return value+1 d = { 'a': 1 } d.apply_to_value('a',increment) print d and d['a'] would be 2. The difference is that only one lookup occurs. (The dictionary would be locked to modifications while the callback function is called, same as if it were being iterated over.) Thing is, there's no way to get that ability except subclass dict in C; can't be done from Python. Which is why even though it'd be a pretty rare need it at least deserves a bit of consideration before being tabled. As a workaround, if lookups are expensive, you can add an indirection to the dict. For example, you could let each value in the dict be a singleton list which is permanently bound to the key. Then you only have look up the key once to modify the value. d = { 'a': [1] } ston = d['a'] ston[0] += 3 and now d['a'][0] is 4, and there was only one dict lookup. Downside is that you have the [0] to carry around, but that is the price of indirection. A wrapper class might help. Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
In article <21e57363-4e92-41cb-9907-5aef96ad0...@o15g2000yqm.googlegroups.com>, RunThePun wrote: > >Anybody have any more ideas? I think python should/could havev a >syntax for overriding this behaviour, i mean, obviously the complexity >of supporting all operators with the getitem syntax could introduce >alot of clutter. But maybe there's an elegant solution out there... Any solution you create must support this: >>> L = [3] >>> L[0] += 7 >>> L [10] -- Aahz (a...@pythoncraft.com) <*> http://www.pythoncraft.com/ "I support family values -- Addams family values" --www.nancybuttons.com -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
On 2009-08-27 01:49 AM, RunThePun wrote: Anybody have any more ideas? I think python should/could havev a syntax for overriding this behaviour, i mean, obviously the complexity of supporting all operators with the getitem syntax could introduce alot of clutter. But maybe there's an elegant solution out there... I would recommend just adding a method to MyDict that does exactly what you want. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
On Aug 27, 6:58 am, Robert Kern wrote: > On 2009-08-26 20:00 PM, Jan Kaliszewski wrote: > > > > > > > 27-08-2009 o 00:48:33 Robert Kern wrote: > > >> On 2009-08-26 17:16 PM, RunThePun wrote: > >>> I'd like to build a database wrapper using DictMixin and allow items > >>> to be appended by my own code. The problem is += is always understood > >>> as setitem and getitem plainly. > > >>> d = MyDict() > >>> d['a'] = 1 > > >>> # this is the problem code that's I'd like to override. It's always > >>> setitem('a', getitem('a') + 3) > >>> d['a'] += 3 > >>> # i wanted to do something like my own 'appenditem' function which for > >>> example could be useful if getitem is an expensive operation which can > >>> be avoided. > > >>> I hope that was clear enough of a request, it's really late at night > >>> here... > > >> I'm sorry, this is just part of the syntax of Python. You cannot > >> override it. > > > Though > > d['a'] = 3 > > is equivalent to: > > d.__setitem__('a', 3) > > > The > > d['a'] += 3 > > *is not* equivalent to: > > d.__setitem__('a', d.__getitem__('a') + 3) > > *but is* equivalent to: > > d.__getitem__('a').__iadd__(3) > > > Then you can override __getitem__() of MyDict in such a way that it > > returns prepared (wrapped) object with overriden __iadd__() as you > > want to. > > You could, but then you will almost certainly run into problems using the > wrapped object in places that really expect the true object. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > that is made terrible by our own mad attempt to interpret it as though it > had > an underlying truth." > -- Umberto Eco Exactly my problem Robert. I'm actually going to be using this dict for storing and retrieving strings so wrapping str and replacing the iadd would cause alot of craziness. Anybody have any more ideas? I think python should/could havev a syntax for overriding this behaviour, i mean, obviously the complexity of supporting all operators with the getitem syntax could introduce alot of clutter. But maybe there's an elegant solution out there... ---RP -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
On 2009-08-26 20:00 PM, Jan Kaliszewski wrote: 27-08-2009 o 00:48:33 Robert Kern wrote: On 2009-08-26 17:16 PM, RunThePun wrote: I'd like to build a database wrapper using DictMixin and allow items to be appended by my own code. The problem is += is always understood as setitem and getitem plainly. d = MyDict() d['a'] = 1 # this is the problem code that's I'd like to override. It's always setitem('a', getitem('a') + 3) d['a'] += 3 # i wanted to do something like my own 'appenditem' function which for example could be useful if getitem is an expensive operation which can be avoided. I hope that was clear enough of a request, it's really late at night here... I'm sorry, this is just part of the syntax of Python. You cannot override it. Though d['a'] = 3 is equivalent to: d.__setitem__('a', 3) The d['a'] += 3 *is not* equivalent to: d.__setitem__('a', d.__getitem__('a') + 3) *but is* equivalent to: d.__getitem__('a').__iadd__(3) Then you can override __getitem__() of MyDict in such a way that it returns prepared (wrapped) object with overriden __iadd__() as you want to. You could, but then you will almost certainly run into problems using the wrapped object in places that really expect the true object. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
27-08-2009 o 00:48:33 Robert Kern wrote: On 2009-08-26 17:16 PM, RunThePun wrote: I'd like to build a database wrapper using DictMixin and allow items to be appended by my own code. The problem is += is always understood as setitem and getitem plainly. d = MyDict() d['a'] = 1 # this is the problem code that's I'd like to override. It's always setitem('a', getitem('a') + 3) d['a'] += 3 # i wanted to do something like my own 'appenditem' function which for example could be useful if getitem is an expensive operation which can be avoided. I hope that was clear enough of a request, it's really late at night here... I'm sorry, this is just part of the syntax of Python. You cannot override it. Though d['a'] = 3 is equivalent to: d.__setitem__('a', 3) The d['a'] += 3 *is not* equivalent to: d.__setitem__('a', d.__getitem__('a') + 3) *but is* equivalent to: d.__setitem__('a', d.__getitem__('a').__iadd__(3)) Then you can override __getitem__() of MyDict in such a way that it returns prepared (wrapped) object with overriden __iadd__() as you want to. How could I now it: 1 import collections 2 import functools 3 import itertools 4 5 6 def verbose_func(func): 7 'Function decorator that makes a function "verbose"' 8 9 @functools.wraps(func, assigned=('__name__', '__doc__')) 10 def func_wrapper(*args, **kwargs): 11 iargs = (map(str, args)) 12 ikwargs = ('{0}={1}'.format(key, value) 13for key, value in kwargs.items()) 14 func_args = ', '.join(itertools.chain(iargs, ikwargs)) 15 print('{0}({1})'.format(func.__name__, func_args)) 16 return func(*args, **kwargs) 17 18 return func_wrapper 19 20 21 def verbose_cls(base): 22 'Class decorator that makes callable attributes "verbose"' 23 24 quiet = ('__new__', '__repr__', '__str__') 25 26 def cls_wrapper(cls): 27 for name in vars(base): 28 attr = getattr(cls, name) 29 if isinstance(attr, collections.Callable) and name not in quiet: 30 setattr(cls, name, verbose_func(attr)) 31 return cls 32 33 return cls_wrapper 34 35 36 @verbose_cls(dict) 37 class VerboseDict(dict): 38 pass 39 40 41 @verbose_cls(int) 42 class MyInt(int): 43 44 @verbose_func 45 def __iadd__(self, other): 46 int.__add__(self, other) # can do something more interesting 47 48 49 if __name__ == '__main__': 50 d = VerboseDict() 51 52 print("d['a'] = 3") 53 d['a'] = MyInt(3) 54 55 print("d['a'] += 3") 56 d['a'] += MyInt(3) *j -- Jan Kaliszewski (zuo) -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
27-08-2009 o 00:48:33 Robert Kern wrote: On 2009-08-26 17:16 PM, RunThePun wrote: I'd like to build a database wrapper using DictMixin and allow items to be appended by my own code. The problem is += is always understood as setitem and getitem plainly. d = MyDict() d['a'] = 1 # this is the problem code that's I'd like to override. It's always setitem('a', getitem('a') + 3) d['a'] += 3 # i wanted to do something like my own 'appenditem' function which for example could be useful if getitem is an expensive operation which can be avoided. I hope that was clear enough of a request, it's really late at night here... I'm sorry, this is just part of the syntax of Python. You cannot override it. Though d['a'] = 3 is equivalent to: d.__setitem__('a', 3) The d['a'] += 3 *is not* equivalent to: d.__setitem__('a', d.__getitem__('a') + 3) *but is* equivalent to: d.__getitem__('a').__iadd__(3) Then you can override __getitem__() of MyDict in such a way that it returns prepared (wrapped) object with overriden __iadd__() as you want to. How could I now it: 1 import collections 2 import functools 3 import itertools 4 5 6 def verbose_func(func): 7 'Function decorator that makes a function "verbose"' 8 9 @functools.wraps(func, assigned=('__name__', '__doc__')) 10 def func_wrapper(*args, **kwargs): 11 iargs = (map(str, args)) 12 ikwargs = ('{0}={1}'.format(key, value) 13for key, value in kwargs.items()) 14 func_args = ', '.join(itertools.chain(iargs, ikwargs)) 15 print('{0}({1})'.format(func.__name__, func_args)) 16 return func(*args, **kwargs) 17 18 return func_wrapper 19 20 21 def verbose_cls(base): 22 'Class decorator that makes callable attributes "verbose"' 23 24 quiet = ('__new__', '__repr__', '__str__') 25 26 def cls_wrapper(cls): 27 for name in vars(base): 28 attr = getattr(cls, name) 29 if isinstance(attr, collections.Callable) and name not in quiet: 30 setattr(cls, name, verbose_func(attr)) 31 return cls 32 33 return cls_wrapper 34 35 36 @verbose_cls(dict) 37 class VerboseDict(dict): 38 pass 39 40 41 @verbose_cls(int) 42 class MyInt(int): 43 44 @verbose_func 45 def __iadd__(self, other): 46 int.__add__(self, other) # can do something more interesting 47 48 49 if __name__ == '__main__': 50 d = VerboseDict() 51 52 print("d['a'] = 3") 53 d['a'] = MyInt(3) 54 55 print("d['a'] += 3") 56 d['a'] += MyInt(3) *j -- Jan Kaliszewski (zuo) -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
I haven't tested it, but did you encounter a problem defining __iadd__ in the class definition? See: http://docs.python.org/reference/datamodel.html#object.__iadd__ Cheers, Ching-Yun "Xavier" Ho, Technical Artist Contact Information Mobile: (+61) 04 3335 4748 Skype ID: SpaXe85 Email: cont...@xavierho.com Website: http://xavierho.com/ On Thu, Aug 27, 2009 at 8:48 AM, Robert Kern wrote: > On 2009-08-26 17:16 PM, RunThePun wrote: > >> I'd like to build a database wrapper using DictMixin and allow items >> to be appended by my own code. The problem is += is always understood >> as setitem and getitem plainly. >> >> d = MyDict() >> d['a'] = 1 >> >> # this is the problem code that's I'd like to override. It's always >> setitem('a', getitem('a') + 3) >> d['a'] += 3 >> # i wanted to do something like my own 'appenditem' function which for >> example could be useful if getitem is an expensive operation which can >> be avoided. >> >> I hope that was clear enough of a request, it's really late at night >> here... >> > > I'm sorry, this is just part of the syntax of Python. You cannot override > it. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma > that is made terrible by our own mad attempt to interpret it as though it > had > an underlying truth." > -- Umberto Eco > > > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
Re: Overriding iadd for dictionary like objects
On 2009-08-26 17:16 PM, RunThePun wrote: I'd like to build a database wrapper using DictMixin and allow items to be appended by my own code. The problem is += is always understood as setitem and getitem plainly. d = MyDict() d['a'] = 1 # this is the problem code that's I'd like to override. It's always setitem('a', getitem('a') + 3) d['a'] += 3 # i wanted to do something like my own 'appenditem' function which for example could be useful if getitem is an expensive operation which can be avoided. I hope that was clear enough of a request, it's really late at night here... I'm sorry, this is just part of the syntax of Python. You cannot override it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Overriding iadd for dictionary like objects
I'd like to build a database wrapper using DictMixin and allow items to be appended by my own code. The problem is += is always understood as setitem and getitem plainly. d = MyDict() d['a'] = 1 # this is the problem code that's I'd like to override. It's always setitem('a', getitem('a') + 3) d['a'] += 3 # i wanted to do something like my own 'appenditem' function which for example could be useful if getitem is an expensive operation which can be avoided. I hope that was clear enough of a request, it's really late at night here... thanks, RunPun -- http://mail.python.org/mailman/listinfo/python-list