[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Guido van Rossum]

Change by Guido van Rossum :


--
nosy:  -gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Daniel Fortunov]

Change by Daniel Fortunov :


--
nosy: +dfortunov
nosy_count: 15.0 -> 16.0
pull_requests: +19902
pull_request: https://github.com/python/cpython/pull/20687

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Charles-François Natali]

Charles-François Natali added the comment:

Shouldn't the documentation be updated?

https://docs.python.org/3.6/library/weakref.html#weakref.WeakKeyDictionary

Note Caution: Because a WeakKeyDictionary is built on top of a Python 
dictionary, it must not change size when iterating over it. This can be 
difficult to ensure for a WeakKeyDictionary because actions performed by the 
program during iteration may cause items in the dictionary to vanish “by magic” 
(as a side effect of garbage collection).

--
nosy: +neologix

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Kristján Valur Jónsson]

Changes by Kristján Valur Jónsson :


--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Antoine Pitrou]

Antoine Pitrou added the comment:

Since this is backported, shouldn't it be closed?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Mark Lawrence]

Changes by Mark Lawrence :


--
nosy:  -BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Roundup Robot]

Roundup Robot added the comment:

New changeset 03fcc12282fc by Kristján Valur Jónsson in branch '2.7':
Issue #7105:  weak dict iterators are fragile because of unpredictable GC runs
http://hg.python.org/cpython/rev/03fcc12282fc

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Antoine Pitrou]

Antoine Pitrou added the comment:

I'm ok with the backport.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Kristján Valur Jónsson]

Kristján Valur Jónsson added the comment:

That's the spirit, Guido :)
I just think people are being extra careful after the "regression" introduced 
in 2.7.5.
However, IMHO we must never let the odd mistake scare us from making necessary 
moves.
Unless Antoine explicitly objects, I think I'll submit my patch from november 
and we'll just watch what happens.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Guido van Rossum]

Guido van Rossum added the comment:

I'm not sure I understand the hesitation about backporting the Python 3 
solution. We're acknowledging it's a bug, so the fix is not a feature. The 
Python 3 solution is the future. So why not fix it?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Antoine Pitrou]

Antoine Pitrou added the comment:

d) sounds like a good enough resolution at this point.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Kristján Valur Jónsson]

Kristján Valur Jónsson added the comment:

d), We could also simply issue a (documentation) warning, that the "iterator" 
methods of these dictionares are known to be fragile, and recommend that people 
use the keys(), values() and items() instead.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Kristján Valur Jónsson]

Kristján Valur Jónsson added the comment:

Strictly speaking b) is not a semantic change.  Depending on your semantic 
definition of semantics.  At any rate it is even less so than a) since the 
temporary list is hidden from view and the only side effect is additional 
memory usage.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Guido van Rossum]

Guido van Rossum added the comment:

I'm with Antoine.  Have we heard of any problems with the 3.x version of the 
patch?  How different is it?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Antoine Pitrou]

Antoine Pitrou added the comment:

> Anyway, it's not for me to decide.  We have currently three options:
> a) my first patch, which is a duplication of the 3.x work but is
> non-trivial and could bring stability issues
> b) my second patch, which will increase memory use, but to no more
> than previous versions of python used while iterating
> c) do nothing and have iterations over weak dicts randomly break when
> an underlying cycle is unraveled during iteration.

Either a) or c), for me. We shouldn't change semantics in bugfix
releases.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Kristján Valur Jónsson]

Kristján Valur Jónsson added the comment:

Yes, the old memory argument.
But is it valid? Is there a conceivable application where a dict of weak 
references would be storing a large chunk of the application memory?
Remember, all of the data must be referred to from elsewhere, or else, the weak 
refs would not exist.  An extra list of pointers is unlikely to make a 
difference.
I think the chief reason to use iterators has to do with performance by 
avoiding the creation of temporary objects, not saving memory per-se.

Before the invention of "iteritems()" and friends, all such iteration was by 
lists (and hence, memory usage).  We should try to remain nimble enough so that 
we can undo an optimization previously done, if the requirements merit us doing 
so.

As a completely unrelated example of such nimbleness:  Faced with stricter 
regulations in the 70s, american car makers had to sell their muscle cars with 
increasingly less powerful engines, efectively rolling back previous 
optimizations :)

Anyway, it's not for me to decide.  We have currently three options:
a) my first patch, which is a duplication of the 3.x work but is non-trivial 
and could bring stability issues
b) my second patch, which will increase memory use, but to no more than 
previous versions of python used while iterating
c) do nothing and have iterations over weak dicts randomly break when an 
underlying cycle is unraveled during iteration.

Cheers!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Guido van Rossum]

Guido van Rossum added the comment:

I appreciate the simplicity, but I don't think it is acceptable -- if the dict 
is large, materializing the entire list of keys/values/items might allocate a 
prohibitive amount of memory. It's worse if you have code that is expected to 
break out of the loop.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Kristján Valur Jónsson]

Kristján Valur Jónsson added the comment:

Here's a different approach.
Simply avoid the use of iterators over the underlying container.
Instead, we iterate over lists of items/keys/values etc.

--
Added file: http://bugs.python.org/file32932/weakref.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Kristján Valur Jónsson]

Kristján Valur Jónsson added the comment:

The changes for the docs are just a port of the original patch.  And indeed, 
these functions don't return an iterator, but a generator object.

I admit I'm confused by the difference, since next() can be called directly on 
the generator.  Still, a lot of code in the test explicitly calls iter() on the 
iterators before doing next().  Not sure why.

The commented out line is an artifact, I'll remove it, the correct one is the 
test for 20 items.

Otherwise, I have no vested interest in getting this in.  My porting this is 
just me contributing to 2.7.  If it's vetoed, we'll just put it in 2.8.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Antoine Pitrou]

Antoine Pitrou added the comment:

Speaking of which, this is not only about GC runs, although it is the most 
annoying scenario (since you basically don't control when it happens). Regular 
resource reaping because of reference counting can also wreak havoc.

About the patch, I don't know. It introduces new complexity in 2.7 which should 
be fairly stable by now. The one thing I don't like is your replacement of 
"iterator" by "iterable" in the docs.

(you also have one line commented out in the tests)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Kristján Valur Jónsson]

Kristján Valur Jónsson added the comment:

Yes, the "long iterator" scenario is the reason it is not ideal for this 
scenario.
The other one (gc.collect()) is easily solved by implementing this construct 
natively.  It can be done rather simply by adding an overriding "pause" 
property to gc, with the following api:

def pause(increment):
  """
  pause or unpause garbage collection.  A positive value
  increases the pause level, while a negative one reduces it.
  when paused, gc won't happen even when explicitly requested with
  gc.collect(), until the pause level drops to 0.
  """

I'm sure there are other places in the code with local execution that would 
benefit from not having an accidental GC run happen.  I'm sure I've seen such 
places, with elaborate scaffolding to safeguard itself from such cases.

Anyway, my 2 aurar worth of lateral thinking applied to the problem at hand :)

What about the patch itself?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Antoine Pitrou]

Antoine Pitrou added the comment:

> No matter how it sounds, it certainly looks cleaner in code.

It's also unsafe and invasive, since it's a process-wide setting. An
iterator can be long-lived if it's being consumed slowly, so you've
disabled garbage collection for an unknown amount of time, without the
user knowing about it. Another thread could kick in and perhaps
re-enable it for whatever reason.

Oh if someone calls gc.collect() explicitly, the solution is suddenly
defeated.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Kristján Valur Jónsson]

Kristján Valur Jónsson added the comment:

No matter how it sounds, it certainly looks cleaner in code.
Look at all this code, designed to work around an unexpected GC collection with 
various pointy bits and edge cases and special corners.

Compare to explicitly just asking GC to relent, for a bit:
def getitems(self):
  with gc.disabled():
for each in self.data.items():
  yield each

That's it.

While a native implementation of such a context manager would be better 
(faster, and could be made overriding), a simple one can be constructed thus:
@contextlib.contextmanagerd
def gc_disabled():
  enabled = gc.isenabled()
  gs.disable()
  try:
yield
  finally:
if enabled:
  gc.enable()
  

Such global "atomic" context managers are well known to stackless programmers. 
It's a very common idiom when building higher level primitives (such as locks) 
from lower level ones.
with stackless.atomic():
   do()
   various()
   stuff_that_does_not_like_being_interrupted()
(stackless.atomic prevents involuntary tasklet switching _and_ involuntary 
thread switching)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Antoine Pitrou]

Antoine Pitrou added the comment:

> Btw, I think a cleaner way to deal with unpredictable GC runs is to be able 
> to temporarily disable GC with a context manager

Disabling the GC in a library function sounds very ugly to me.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Kristján Valur Jónsson]

Kristján Valur Jónsson added the comment:

Attached is a version for 2.7

Btw, I think a cleaner way to deal with unpredictable GC runs is to be able to 
temporarily disable GC with a context manager.

--
nosy: +kristjan.jonsson
Added file: http://bugs.python.org/file32689/issue7105.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[STINNER Victor]

STINNER Victor added the comment:

What is the status of this issue? Does anyone still want to backport the fix to 
Python 2.7?

(I found this issue while searching for test_multiprocessing failures, and I 
found #7060 which refers this issue.)

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Matthew Schwickerath]

Matthew Schwickerath  added the comment:

Any plans on actually patching this in 2.7 any time soon?  This is affecting 
our software and hanging it on random occasions.

--
nosy: +qelan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Mark Dickinson]

Changes by Mark Dickinson :


--
nosy: +BreamoreBoy, ajaksu2, dcjim, elachuni, tseaver, vdupras

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Mark Dickinson]

Mark Dickinson  added the comment:

The issue still exists in 2.6 and 2.7.

Closing issue 839159 as a duplicate of this one.

--
nosy: +mark.dickinson
resolution: fixed -> 
stage: committed/rejected -> 
status: closed -> open
versions: +Python 2.6, Python 2.7 -Python 3.1, Python 3.2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Committed in r77365 (py3k) and r77366 (3.1). Thank you.

--
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Benjamin Peterson]

Benjamin Peterson  added the comment:

LGTM

--
nosy: +benjamin.peterson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Antoine Pitrou]

Changes by Antoine Pitrou :


Removed file: http://bugs.python.org/file15112/weakiter2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Antoine Pitrou]

Changes by Antoine Pitrou :


Removed file: http://bugs.python.org/file15103/weakiter.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Antoine Pitrou]

Antoine Pitrou  added the comment:

It occurs weaksets have the same problem, here is a new patch fixing
them as well.

--
Added file: http://bugs.python.org/file15114/weakiter3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Antoine Pitrou]

Antoine Pitrou  added the comment:

This new patch makes it possible to mutate the dict without messing with
the delayed removal when an iterator exists.

--
Added file: http://bugs.python.org/file15112/weakiter2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Guido van Rossum]

Guido van Rossum  added the comment:

> delay all removals until all iterators are done

+1

--
nosy: +gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Jon Parise]

Changes by Jon Parise :


--
nosy: +jon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7105] weak dict iterators are fragile because of unpredictable GC runs

[Antoine Pitrou]

New submission from Antoine Pitrou :

As mentioned in issue7060, weak dict iterators are easily broken by
cyclic garbage collection changing the size of the underlying dict storage:

  File "/home/rdmurray/python/py3k/Lib/weakref.py", line 121, in items
for wr in self.data.values():
RuntimeError: dictionary changed size during iteration

One possible solution is to delay all removals until all iterators are
done. Here is a context manager-based solution.

--
components: Library (Lib)
files: weakiter.patch
keywords: patch
messages: 93863
nosy: pitrou
priority: normal
severity: normal
stage: patch review
status: open
title: weak dict iterators are fragile because of unpredictable GC runs
type: behavior
versions: Python 3.1, Python 3.2
Added file: http://bugs.python.org/file15103/weakiter.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com