Michael Droettboom wrote: > On 10/10/2013 09:47 AM, Martin MOKREJŠ wrote: >> Benjamin Root wrote: >>> >>> >>> On Thu, Oct 10, 2013 at 9:05 AM, Martin MOKREJŠ <mmokr...@gmail.com >>> <mailto:mmokr...@gmail.com>> wrote: >>> >>> Hi, >>> rendering some of my charts takes almost 50GB of RAM. I believe >>> below is a stracktrace >>> of one such situation when it already took 15GB. Would somebody >>> comments on what is >>> matplotlib doing at the very moment? Why the recursion? >>> >>> The charts had to have 262422 data points in a 2D scatter plot, each >>> point has assigned >>> its own color. They are in batches so that there are 153 distinct >>> colors but nevertheless, >>> I assigned to each data point a color value. There are 153 legend >>> items also (one color >>> won't be used). >>> >>> ^CTraceback (most recent call last): >>> ... >>> _figure.savefig(filename, dpi=100) >>> File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line >>> 1421, in savefig >>> self.canvas.print_figure(*args, **kwargs) >>> File >>> "/usr/lib64/python2.7/site-packages/matplotlib/backend_bases.py", line >>> 2220, in print_figure >>> **kwargs) >>> File >>> "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py", >>> line 505, in print_png >>> FigureCanvasAgg.draw(self) >>> File >>> "/usr/lib64/python2.7/site-packages/matplotlib/backends/backend_agg.py", >>> line 451, in draw >>> self.figure.draw(self.renderer) >>> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line >>> 54, in draw_wrapper >>> draw(artist, renderer, *args, **kwargs) >>> File "/usr/lib64/python2.7/site-packages/matplotlib/figure.py", line >>> 1034, in draw >>> func(*args) >>> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line >>> 54, in draw_wrapper >>> draw(artist, renderer, *args, **kwargs) >>> File "/usr/lib64/python2.7/site-packages/matplotlib/axes.py", line >>> 2086, in draw >>> a.draw(renderer) >>> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line >>> 54, in draw_wrapper >>> draw(artist, renderer, *args, **kwargs) >>> File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", >>> line 718, in draw >>> return Collection.draw(self, renderer) >>> File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line >>> 54, in draw_wrapper >>> draw(artist, renderer, *args, **kwargs) >>> File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", >>> line 276, in draw >>> offsets, transOffset, self.get_facecolor(), self.get_edgecolor(), >>> File "/usr/lib64/python2.7/site-packages/matplotlib/collections.py", >>> line 551, in get_edgecolor >>> return self._edgecolors >>> KeyboardInterrupt >>> ^CError in atexit._run_exitfuncs: >>> Traceback (most recent call last): >>> File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs >>> func(*targs, **kargs) >>> File >>> "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 90, >>> in destroy_all >>> gc.collect() >>> KeyboardInterrupt >>> Error in sys.exitfunc: >>> Traceback (most recent call last): >>> File "/usr/lib64/python2.7/atexit.py", line 24, in _run_exitfuncs >>> func(*targs, **kargs) >>> File >>> "/usr/lib64/python2.7/site-packages/matplotlib/_pylab_helpers.py", line 90, >>> in destroy_all >>> gc.collect() >>> KeyboardInterrupt >>> >>> ^C >>> >>> >>> Clues what is the code doing? I use mpl-1.3.0. >>> Thank you, >>> Martin >>> >>> >>> Unfortunately, that stacktrace isn't very useful. There is no recursion >>> there, but rather the perfectly normal drawing of the figure object that >>> has a child axes, which has child collections which have child artist >>> objects. >>> >>> Without the accompanying code, it would be difficult to determine where the >>> memory hog is. >> Could there be places where gc.collect() could be introduced? Are there >> places where matplotlib >> could del() unnecessary objects right away? I think the problem is with huge >> lists or pythonic >> dicts. I could save 10GB of RAM when I converted one python dict to a bsddb3 >> file having just >> 10MB on disk. I speculate matplotlib in that code keeps the data in some >> huge list or more likely >> a dict and that is the same issue. >> >> Are you sure you cannot see where a problem is? It happens (is visible) only >> with huge number of >> dots, of course. > > Matplotlib generally keeps data in Numpy arrays, not lists or > dictionaries (though given that matplotlib predates Numpy, there are > some corner cases we've found recently where arrays are converted to > lists and back unintentionally).
Just a brief note. I don't use Numpy myself in my code, so consider that while replicating my use case. ;) The code is merely what I think Tony Yu of Chao Yue proposed or somebody, sorry, don't remember now, proposed to me on this list in the past. I am writing it now really from top of my head, maybe I remember rubbish. ;) Martin ------------------------------------------------------------------------------ October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register > http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk _______________________________________________ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
