On Sep 28, 1:38 pm, "William Stein" <[EMAIL PROTECTED]> wrote:
> On Sun, Sep 28, 2008 at 1:06 PM, mabshoff <[EMAIL PROTECTED]> wrote:
>
> > Hello folks,
>
> > to optimize startup time we did move numpy imports "down" into
> > individual Cython files. Unfortunately every time we do so we end up
> > leaking anywhere from 700 to 900 bytes or so. Since that import can be
> > in frequently visited operations I would strongly suggest we limit
> > ourselves to one global numpy import in all.py.
>
> You're suggesting that doing
>
> import numpy
> import numpy
It does at least happen in Cython code that way. The problem is not so
much numpy itself, but Python leaking one string object when asked to
load a numpy extension module. Since numpy loads a bunch of them it
tends to add up. That happens every time we ask to import numpy from
Cython - which is the way I found the problem. In the end it is likely
an issue with the way we create Python extension modules. None of the
python extensions seem to have that problem, otherwise I would have
seen those problems in the valgrind logs.
Cython modules created for Sage (and in general) show the same
behavior and I expect the root problem here is that in Cython code the
construct "return str(foo)" leaks. The numpy extensions do not use
Cython, so the Cython problem in the Sage library is not the root
cause there. I have added suppression files for the Cython problem so
I can actually do useful memory leak hunting with the current Sage:
==23926== LEAK SUMMARY:
==23926== definitely lost: 612 bytes in 11 blocks.
==23926== possibly lost: 299,077 bytes in 835 blocks.
==23926== still reachable: 33,361,731 bytes in 193,844 blocks.
==23926== suppressed: 329,948 bytes in 5,223 blocks.
Notice that we suppress *5223* blocks and those are all Cython modules
being imported into Sage. The is an ancient ticket to fix some closely
related problem in Cython - see #557.
> would leak twice. Or that doing import numpy in foo.py and bar.py
> would both leak. That's really weird/disturbing, and doesn't match
> at all with how I think Python behaves with respect to importing a
> module that has already been imported.
Well, can't argue with valgrind logs :)
> I'm not claiming that the leaks aren't happening, just that there is
> something extra fishy going on.
I agree.
> > In 3.1.3 we will be
> > upgrading to numpy 1.2 which import significantly faster than 1.1 and
> > especially 1.0.4. Since we fix 8 byte leaks I would certainly suggest
> > that we do the same for a 700+ byte leak, even if that slows down the
> > startup time of Sage about 0.2s on sage.math. I noticed the numpy
> > import leak all over the Sage doctests.
>
> 0.2s on sage.math can be 1s (or more) on some machines, and is
> really lame. However, numpy 1.2 may be much much better I hope;
> we'll see.
That is numpy 1.2. The numpy 1.0.4 release took 0.7s to import.
And I do remember an 8 byte leak filling up the available memory on
sage.math within a couple days when Ifti did some computations about a
year ago and I would always prefer slower startup time to unacceptable
memory leaks that eat up all available RAM for long time
computations.
> It may be possible to have a lazy import in all.py, i.e., something like
>
> def import_numpy():
> if numpy already imported do nothing
> otherwise import numpy
>
> and then any code that would do
>
> import numpy
> numpy.blah
>
> instead would call that import_numpy() function that
> would be defined in some module sage.foo,
> then use sage.foo.numpy.
Yeah, that would solve both issues, but the problem is that people
will add "import numpy" statements over and over again. But that can
be caught via the review process and if it slips by I will catch it
with valgrind anyway.
The right fix also for Cython modules would be to fix the damn string
leak in the first place. I have started looking into solving this, but
so far I have not come up with a way to fix the leak.
Cheers,
Michael
> William Stein
> Associate Professor of Mathematics
> University of Washingtonhttp://wstein.org
--~--~---------~--~----~------------~-------~--~----~
To post to this group, send email to sage-devel@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at http://groups.google.com/group/sage-devel
URLs: http://www.sagemath.org
-~----------~----~----~----~------~----~------~--~---
