Re: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
I would also like to point out that I submitted a patch related to that a couple of months ago in: http://bugs.python.org/issue839159 But it never got any attention :( I'm not sure if it is still relevant. Virgil On 13-Sep-08, at 10:20 PM, Armin Ronacher wrote: Hi everybody, In Python 2.x when iterating over a weak key dictionary for example, the common idom for doing that was calling dictionary.keys() to ensure that a list of all objects is returned it was safe to iterate over as a weak reference could stop existing during dict iteration which of course raises a runtime error by the dict iterator. This was documented behavior and worked pretty well, with the small problem that suddenly all references in the dict wouldn't die until iteration is over because the list holds references to the object. This no longer works in Python 3 because .keys() on the weak key dictionary returns a generator over the key view of the internal dict which of course has the same problem as iterkeys in Python 2.x. The following code shows the problem:: from weakref import WeakKeyDictionary f1 = Foo() f2 = Foo() d = WeakKeyDictionary() d[f1] = 42 d[f2] = 23 i = iter(d.keys()) # or use d.keyrefs() here which has the same problem print(next(i)) del f2 print(next(i)) This example essentially dies with RuntimeError: dictionary changed size during iteration as soon as f2 is deleted. Iterating over weak key dictionaries might not be the most common task but I know some situations where this is necessary. Unfortunately I can't see a way to achieve that in Python 3. Regards, Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/hsoft%40hardcoded.net ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
Hi, Josiah Carlson josiah.carlson at gmail.com writes: i = list(d.keys()) Obviously that doesn't solve the problem. list() consumes the generator one after another, objects can still die when the list is created. Imagine the following example which uses threads:: from time import sleep from weakref import WeakKeyDictionary from threading import Thread class Foo(object): pass d = WeakKeyDictionary() l = [] for x in range(10): obj = Foo() d[obj] = None l.append(obj) del obj def iterater(): for item in list(d.keys()): pass Thread(target=iterater).start() while True: del l[0] Regards, Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
Hi, Adam Olsen rhamph at gmail.com writes: IMO, this is a deeper problem than suggested. As far as I know, python does not (and should not) make promises as to when it'll collect object. We should expect weakrefs to be cleared at random points, and code defensively. It doesn't promise when objects are collected and that's not the problem here. The main problem is that the old solution for weakrefs relayed on the fact that .keys() was considered atomic. I don't say it has to become again, but the weak dictionaries have to somehow counter that problem. They could for example only remove items from the internal dict if no dict view to that dict is alive. Speaking of atom keys() / values() / items() operations: I guess we will see more of those problems in threaded situations when people start to convert code over to Python. I've seen quite a few situations where code relays on keys() holding the interpreter lock. Regards, Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
Armin Ronacher wrote: Speaking of atom keys() / values() / items() operations: I guess we will see more of those problems in threaded situations when people start to convert code over to Python. I've seen quite a few situations where code relays on keys() holding the interpreter lock. list(iter) doesn't re-enter the eval loop if the iterator is written in C and hence won't let go of the GIL. As I believe the dict views in 3.0 are all C classes, I'd like to see a failing test case along these lines that doesn't involve a dict subclass and code written in Python. Note that I'm not saying that I know for certain that there aren't any problems with list(d.keys()) that aren't seen with the direct list conversion in 2.x - I'm just saying it may not be as easy to provoke such problems as it initially appears due to the way the list constructor interacts with the supplied iterable, so it is going to take an actual failing test case to convince me. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
Nick Coghlan schrieb: Armin Ronacher wrote: Speaking of atom keys() / values() / items() operations: I guess we will see more of those problems in threaded situations when people start to convert code over to Python. I've seen quite a few situations where code relays on keys() holding the interpreter lock. list(iter) doesn't re-enter the eval loop if the iterator is written in C and hence won't let go of the GIL. As I believe the dict views in 3.0 are all C classes, I'd like to see a failing test case along these lines that doesn't involve a dict subclass and code written in Python. WeakKeyDictionary.keys() iterates over its .data dictionary's keys() instead of list(data.keys()). Therefore, the user of WeakKeyDictionary has no chance to do list(keys()) without running Python code. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
Georg Brandl wrote: Nick Coghlan schrieb: Armin Ronacher wrote: Speaking of atom keys() / values() / items() operations: I guess we will see more of those problems in threaded situations when people start to convert code over to Python. I've seen quite a few situations where code relays on keys() holding the interpreter lock. list(iter) doesn't re-enter the eval loop if the iterator is written in C and hence won't let go of the GIL. As I believe the dict views in 3.0 are all C classes, I'd like to see a failing test case along these lines that doesn't involve a dict subclass and code written in Python. WeakKeyDictionary.keys() iterates over its .data dictionary's keys() instead of list(data.keys()). Therefore, the user of WeakKeyDictionary has no chance to do list(keys()) without running Python code. Ouch, there's the dict subclass with Python code that I was asking for before I got concerned (I forgot that the weakref module is only partially implemented in C). I agree this is a problem, but I'm struggling to think of possible solutions that aren't either horrendously complicated or completely contrary to the idea of view-based dicts. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
On Sat, Sep 13, 2008 at 1:20 PM, Armin Ronacher [EMAIL PROTECTED] wrote: Hi everybody, In Python 2.x when iterating over a weak key dictionary for example, the common idom for doing that was calling dictionary.keys() to ensure that a list of all objects is returned it was safe to iterate over as a weak reference could stop existing during dict iteration which of course raises a runtime error by the dict iterator. This was documented behavior and worked pretty well, with the small problem that suddenly all references in the dict wouldn't die until iteration is over because the list holds references to the object. This no longer works in Python 3 because .keys() on the weak key dictionary returns a generator over the key view of the internal dict which of course has the same problem as iterkeys in Python 2.x. The following code shows the problem:: from weakref import WeakKeyDictionary f1 = Foo() f2 = Foo() d = WeakKeyDictionary() d[f1] = 42 d[f2] = 23 i = iter(d.keys()) # or use d.keyrefs() here which has the same problem print(next(i)) del f2 print(next(i)) This example essentially dies with RuntimeError: dictionary changed size during iteration as soon as f2 is deleted. Iterating over weak key dictionaries might not be the most common task but I know some situations where this is necessary. Unfortunately I can't see a way to achieve that in Python 3. i = list(d.keys()) - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
Josiah Carlson wrote: On Sat, Sep 13, 2008 at 1:20 PM, Armin Ronacher wrote: Iterating over weak key dictionaries might not be the most common task but I know some situations where this is necessary. Unfortunately I can't see a way to achieve that in Python 3. i = list(d.keys()) Surely i = list(d) is a more reasonable way to do this. I seldom find a reason to use .keys --Scott David Daniels [EMAIL PROTECTED] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
On Sat, Sep 13, 2008 at 5:21 PM, Scott David Daniels [EMAIL PROTECTED] wrote: Josiah Carlson wrote: On Sat, Sep 13, 2008 at 1:20 PM, Armin Ronacher wrote: Iterating over weak key dictionaries might not be the most common task but I know some situations where this is necessary. Unfortunately I can't see a way to achieve that in Python 3. i = list(d.keys()) Surely i = list(d) is a more reasonable way to do this. I seldom find a reason to use .keys Definitely. I was being lazy in my use of list(d.keys()) ;) - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
On Sat, Sep 13, 2008 at 2:20 PM, Armin Ronacher [EMAIL PROTECTED] wrote: Hi everybody, In Python 2.x when iterating over a weak key dictionary for example, the common idom for doing that was calling dictionary.keys() to ensure that a list of all objects is returned it was safe to iterate over as a weak reference could stop existing during dict iteration which of course raises a runtime error by the dict iterator. This was documented behavior and worked pretty well, with the small problem that suddenly all references in the dict wouldn't die until iteration is over because the list holds references to the object. This no longer works in Python 3 because .keys() on the weak key dictionary returns a generator over the key view of the internal dict which of course has the same problem as iterkeys in Python 2.x. The following code shows the problem:: from weakref import WeakKeyDictionary f1 = Foo() f2 = Foo() d = WeakKeyDictionary() d[f1] = 42 d[f2] = 23 i = iter(d.keys()) # or use d.keyrefs() here which has the same problem print(next(i)) del f2 print(next(i)) This example essentially dies with RuntimeError: dictionary changed size during iteration as soon as f2 is deleted. Iterating over weak key dictionaries might not be the most common task but I know some situations where this is necessary. Unfortunately I can't see a way to achieve that in Python 3. IMO, this is a deeper problem than suggested. As far as I know, python does not (and should not) make promises as to when it'll collect object. We should expect weakrefs to be cleared at random points, and code defensively. One approach I've used before is to enqueue all the cleared weakrefs, then process that queue when the WeakKeyDictionary is modified. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
