I've had little time for Cython lately and so refcount nanny development
has been slow. But when development is going at this pace I figured it
was better to upload what I had so it doesn't start bit-rotting on my disk.
It is NOT FINISHED, but the ideas and framework is mostly in place, it
is just a matter of working out all the places it fails. I'll continue
it myself when I get time, but feel free...
By default it is disabled and Cython works as before. To enable it:
cd Cython/Runtime
./build.sh # yes, ugh! Let's deal with that when the rest is done
cd ../..
python runtests.py --refnanny ...
One can then get reference count errors for two reasons:
- Most likely: The required calls to __Pyx_GOTREF and __Pyx_GIVEREF is
missing in a given situation
- Otherwise: There's a genuine reference counting problem.
Here's the details:
- Cython generates code that can always be run with or without the
refnanny -- to run it with, compile the C code with CYTHON_REFNANNY defined.
- Some boilerplate code is added when entering and exiting all functions
(which should compile into nothing when not using the nanny)
- Many calls to Py_INCREF etc. are replaced with __Pyx_INCREF etc. (but
not in utility code which is in general trusted and doesn't use the nanny)
- New calls are added: __Pyx_GOTREF, __Pyx_GIVEREF, __Pyx_XGIVEREF which
are called after getting a reference from another function and before
giving away a reference to a function/array/struct field. I.e:
__Pyx_GIVEREF(obj)
PyList_SET_ITEM(..., obj, ...)
and
obj = Py_Get...
if (!obj) { ... }
__Pyx_GOTREF(obj)
(Also, XGIVEREF is called on __pyx_r prior to returning.)
- There's a hook module in Cython/Runtime/refnanny.pyx. For now this is
manually compiled with a Linux-only shell script, this must of course be
done something about. See [1] for more plans for this module.
- runtests.py has the --refnanny flag to turn on the refnanny (when it
is done, I expect this to be the default). What this does is loading the
hook module and compiling modules with CYTHON_REFNANNY defined.
End result:
- Extra decrefs are intercepted and only logged.
- Functions continue as normal when this happen, but when the function
returns for any reason a normal exception is raised if there's a leak or
there was an extra decref somewhere.
[1] The module can be written to contain as advanced diagnostics as we
like, and can be inspected by runtests.py to read logs etc.. It would be
entirely possible to extend this to allow, say, putting assertions on
the refcount operations Cython code performs in the doctests that
runtests.py runs:
def f():
"""
Test that this functions does the reference counting we would like.
>>> import refnanny
>>> a = 2
>>> refnanny.monitor_object(a)
>>> f(a)
>>> print refnanny.operations_on(a)
INCREFS=10, DECREFS=10
"""
# ...
That's an argument for keeping it as a seperate module, but that's all
I'm going to do about it for now.
Dag Sverre
_______________________________________________
Cython-dev mailing list
[email protected]
http://codespeak.net/mailman/listinfo/cython-dev
