On Thu, Oct 10, 2013 at 9:47 AM, Martin MOKREJŠ <mmokr...@gmail.com> wrote:
> Benjamin Root wrote:
> >
> >
> >
> > On Thu, Oct 10, 2013 at 9:05 AM, Martin MOKREJŠ <mmokr...@gmail.com<mailto:
> mmokr...@gmail.com>> wrote:
> >
> > Hi,
> > rendering some of my charts takes almost 50GB of RAM. I believe
> below is a stracktrace
> > of one such situation when it already took 15GB. Would somebody
> comments on what is
> > matplotlib doing at the very moment? Why the recursion?
> >
> > The charts had to have 262422 data points in a 2D scatter plot,
> each point has assigned
> > its own color. They are in batches so that there are 153 distinct
> colors but nevertheless,
> > I assigned to each data point a color value. There are 153 legend
> items also (one color
> > won't be used).
> >
> > ^CTraceback (most recent call last):
> > ...
> > _figure.savefig(filename, dpi=100)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py",
> line 1421, in savefig
> > self.canvas.print_figure(*args, **kwargs)
> > File
> "/usr/lib64/python2.7/site-packages/matplotlib/backend_bases.py", line
> 2220, in print_figure
> > **kwargs)
> > File
> "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py",
> line 505, in print_png
> > FigureCanvasAgg.draw(self)
> > File
> "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py",
> line 451, in draw
> > self.figure.draw(self.renderer)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py",
> line 54, in draw_wrapper
> > draw(artist, renderer, *args, **kwargs)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py",
> line 1034, in draw
> > func(*args)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py",
> line 54, in draw_wrapper
> > draw(artist, renderer, *args, **kwargs)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/axes.py", line
> 2086, in draw
> > a.draw(renderer)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py",
> line 54, in draw_wrapper
> > draw(artist, renderer, *args, **kwargs)
> > File
> "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 718,
> in draw
> > return Collection.draw(self, renderer)
> > File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py",
> line 54, in draw_wrapper
> > draw(artist, renderer, *args, **kwargs)
> > File
> "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 276,
> in draw
> > offsets, transOffset, self.get_facecolor(), self.get_edgecolor(),
> > File
> "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", line 551,
> in get_edgecolor
> > return self._edgecolors
> > KeyboardInterrupt
> > ^CError in atexit._run_exitfuncs:
> > Traceback (most recent call last):
> > File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
> > func(*targs, **kargs)
> > File
> "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 90,
> in destroy_all
> > gc.collect()
> > KeyboardInterrupt
> > Error in sys.exitfunc:
> > Traceback (most recent call last):
> > File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs
> > func(*targs, **kargs)
> > File
> "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 90,
> in destroy_all
> > gc.collect()
> > KeyboardInterrupt
> >
> > ^C
> >
> >
> > Clues what is the code doing? I use mpl-1.3.0.
> > Thank you,
> > Martin
> >
> >
> > Unfortunately, that stacktrace isn't very useful. There is no recursion
> there, but rather the perfectly normal drawing of the figure object that
> has a child axes, which has child collections which have child artist
> objects.
> >
> > Without the accompanying code, it would be difficult to determine where
> the memory hog is.
>
> Could there be places where gc.collect() could be introduced? Are there
> places where matplotlib
> could del() unnecessary objects right away? I think the problem is with
> huge lists or pythonic
> dicts. I could save 10GB of RAM when I converted one python dict to a
> bsddb3 file having just
> 10MB on disk. I speculate matplotlib in that code keeps the data in some
> huge list or more likely
> a dict and that is the same issue.
>
> Are you sure you cannot see where a problem is? It happens (is visible)
> only with huge number of
> dots, of course.
>
>
I am not going to claim that matplotlib is the most lean graphing library
out there, and we already do know where we can make continued improvements,
but the symptom you are describing (50 GB for a couple hundred thousand
scatter points) is just unheard of for matplotlib. Without a simple,
concise, complete code example to demonstrate your problem, we can only
hazard guesses. For all I know, you might be "appending" to numpy arrays in
a loop prior to plotting, which would eat up significant amount of memory
without it being the fault of matplotlib.
As far as I am aware, we don't do very large dictionaries, so I am doubtful
that is the issue either.
As a side note, I have typically found that situations where del()
significantly improved memory usage were typically situations where I was
"doing it wrong" in the first place and a simple refactor of the code
improved memory and (sometimes) speed, with an added benefit of improved
readability. I have even seen situations where calling del() in the wrong
places (say, for a list created at the beginning of the loop) actually hurt
performance because python couldn't recycle that chunk of memory.
Give us a code example that reproduces your problem, and then we can start
doing some more serious debugging.
Ben Root
> Thanks,
> Martin
>
------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk
_______________________________________________
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-users
