[issue41133] Insufficient description of cyclic garbage collector for C API

[Daniel Barkalow]

Daniel Barkalow  added the comment:

I think it would be helpful to have something as troubleshooting information on 
the garbage collector. If you've got a bug in your C module, it's obvious that 
you shouldn't be doing something egregiously wrong, but it's not obvious what 
you might have done wrong in order to cause that particular failure.

It wouldn't be useful to promise that, if you have a particular sort of bug, 
you'll get a particular sort of failure, or keep it the same from version to 
version, but it would be nice to be told, if you're getting tp_clear called on 
objects that aren't garbage in this particular version of Python, look for 
cases where you're calling py_VISIT on it more times than you called Py_INCREF. 
Alternatively, it would be helpful for the API documentation to say, "If you 
are having some strange problem with garbage collection, here's documentation 
(Developer Guide link) on what's really going on in the garbage collector, 
which might help explain what you could be doing wrong to get that effect."

For that matter, the documentation for tp_traverse doesn't mention that you 
have to call Py_VISIT on each object exactly as many times as references you 
hold to it; the phrasing sounds like it's fine so long as you call it at least 
once on any object you hold a reference to.

--

___
Python tracker 
<https://bugs.python.org/issue41133>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue41133] Insufficient description of cyclic garbage collector for C API

[Daniel Barkalow]

New submission from Daniel Barkalow :

Nothing in the documentation gives you enough information to find what you're 
doing wrong if you have the following bug for an object tracked by the GC: (a) 
you store a borrowed reference to an object whose lifetime is always longer 
that the buggy object (and never decrement it); (b) you visit that object in 
your tp_traverse.

Despite the bug, everything works nearly all of the time. However, when it 
fails, the effect makes no sense from a documentation-only understanding of the 
garbage collection (unless Python is built with --with-assertions, perhaps). 
The only sequence in which it fails is:

* All of the objects that will reference the victim get moved into an older 
generation.
* The victim gets exactly as many buggy references from objects in its own 
generation as there are good references, and no good references from objects in 
its own generation.
* Garbage collection runs for only the younger generation.

At this point, the victim is marked as garbage despite the fact that there was 
a good object referencing it at all times.

Reading the Python source, it becomes clear that the garbage collector handles 
references from older generations not by traversing the objects in older 
generations, but by comparing the count of references from other young objects 
to the usual reference count. That is, visiting an object you don't hold a 
reference to may cause that object to get collected, rather than protecting it, 
even if there are other reasons not to collect it.

The best fix would probably be a warning in the documentation for tp_traverse 
that visiting an object you do not hold a strong reference to can cause 
inexplicable effects, because the intuitive guess would be that it could only 
cause memory leaks.

It would probably also be worth mentioning in the documentation of the garbage 
collector something about its algorithm, so people have a hint that references 
from older objects aren't necessarily sufficient to overcome buggy use of the C 
API.

--
assignee: docs@python
components: C API, Documentation
messages: 372445
nosy: docs@python, iabervon
priority: normal
severity: normal
status: open
title: Insufficient description of cyclic garbage collector for C API
versions: Python 3.7, Python 3.8

___
Python tracker 
<https://bugs.python.org/issue41133>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com