Thanks! I think this will help immensely for watching for memory
leaks in Sage.
On Jan 7, 2009, at 6:12 AM, Dag Sverre Seljebotn wrote:
> I've had little time for Cython lately and so refcount nanny
> development
> has been slow. But when development is going at this pace I figured it
> was better to upload what I had so it doesn't start bit-rotting on
> my disk.
>
> It is NOT FINISHED, but the ideas and framework is mostly in place, it
> is just a matter of working out all the places it fails. I'll continue
> it myself when I get time, but feel free...
>
> By default it is disabled and Cython works as before. To enable it:
> cd Cython/Runtime
> ./build.sh # yes, ugh! Let's deal with that when the rest is done
> cd ../..
> python runtests.py --refnanny ...
>
> One can then get reference count errors for two reasons:
> - Most likely: The required calls to __Pyx_GOTREF and __Pyx_GIVEREF is
> missing in a given situation
> - Otherwise: There's a genuine reference counting problem.
>
> Here's the details:
> - Cython generates code that can always be run with or without the
> refnanny -- to run it with, compile the C code with CYTHON_REFNANNY
> defined.
> - Some boilerplate code is added when entering and exiting all
> functions
> (which should compile into nothing when not using the nanny)
> - Many calls to Py_INCREF etc. are replaced with __Pyx_INCREF etc.
> (but
> not in utility code which is in general trusted and doesn't use the
> nanny)
> - New calls are added: __Pyx_GOTREF, __Pyx_GIVEREF, __Pyx_XGIVEREF
> which
> are called after getting a reference from another function and before
> giving away a reference to a function/array/struct field. I.e:
>
> __Pyx_GIVEREF(obj)
> PyList_SET_ITEM(..., obj, ...)
>
> and
>
> obj = Py_Get...
> if (!obj) { ... }
> __Pyx_GOTREF(obj)
>
> (Also, XGIVEREF is called on __pyx_r prior to returning.)
>
> - There's a hook module in Cython/Runtime/refnanny.pyx. For now
> this is
> manually compiled with a Linux-only shell script, this must of
> course be
> done something about. See [1] for more plans for this module.
> - runtests.py has the --refnanny flag to turn on the refnanny (when it
> is done, I expect this to be the default). What this does is
> loading the
> hook module and compiling modules with CYTHON_REFNANNY defined.
>
> End result:
> - Extra decrefs are intercepted and only logged.
> - Functions continue as normal when this happen, but when the function
> returns for any reason a normal exception is raised if there's a
> leak or
> there was an extra decref somewhere.
>
> [1] The module can be written to contain as advanced diagnostics as we
> like, and can be inspected by runtests.py to read logs etc.. It
> would be
> entirely possible to extend this to allow, say, putting assertions on
> the refcount operations Cython code performs in the doctests that
> runtests.py runs:
>
> def f():
> """
> Test that this functions does the reference counting we would like.
>
>>>> import refnanny
>>>> a = 2
>>>> refnanny.monitor_object(a)
>>>> f(a)
>>>> print refnanny.operations_on(a)
> INCREFS=10, DECREFS=10
> """
> # ...
>
> That's an argument for keeping it as a seperate module, but that's all
> I'm going to do about it for now.
>
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> [email protected]
> http://codespeak.net/mailman/listinfo/cython-dev
_______________________________________________
Cython-dev mailing list
[email protected]
http://codespeak.net/mailman/listinfo/cython-dev
