Re: [Matplotlib-users] plotting large images
Hi Martin, Hi, I knw you asked for memory profiling but I could not resist and did CPU profiling on your testcase. I have attached some screenshots and in words: thanks for these tips about profiling. Stepan -- Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! Discover the easy way to master current and previous Microsoft technologies and advance your career. Get an incredible 1,500+ hours of step-by-step tutorial videos with LearnDevNow. Subscribe today and save! http://pubads.g.doubleclick.net/gampad/clk?id=58040911iu=/4140/ostg.clktrk___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] plotting large images
Hi Chris, I've used some hacky tricks to get around this, which mostly involve downsampling the image on the fly based on screen resolution. One such effort is at https://github.com/ChrisBeaumont/mpl-modest-image (https://github.com/ChrisBeaumont/mpl-modest-image). I tried your code for plotting 4kx4k image and it is another significant improvement. Originally it took 300 MB then it was reduced to 190 MB with uint8 type and using your ModestImage class it takes 70-100 MB depending on size of window. That is much better! Best Stepan -- Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! Discover the easy way to master current and previous Microsoft technologies and advance your career. Get an incredible 1,500+ hours of step-by-step tutorial videos with LearnDevNow. Subscribe today and save! http://pubads.g.doubleclick.net/gampad/clk?id=58040911iu=/4140/ostg.clktrk___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
[Matplotlib-users] plotting large images
Hi, I would like to plot multiple overlayed 4096x4096 images in one axes. If I run this code the plot takes 300 MB of memory: import numpy as np import matplotlib.pyplot as plt if __name__ == '__main__': img = np.zeros((4096, 4096)) img[100: 300, 100:1500] = 200 imgplot = plt.imshow(img) plt.show() And it takes additional 300 MB for every image with this size added into plot. Is there any way to reduce memory consumption without need of data resampling? My configuration: Matplotlib 1.2.1 Numpy 1.7.1 Ubuntu 13.04 64 bit Best Stepan -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511iu=/4140/ostg.clktrk___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] plotting large images
You could, before plotting, sum the different image arrays? Depending on whether you are plotting RGB(A) images or greyscale images, you could take the sum of the color channels, or take a weighted average. The method you use here depends strongly on the image type, but it will reduce memory consumption. Just a thought. 2013/8/27 Štěpán Turek stepan.tu...@seznam.cz Hi, I would like to plot multiple overlayed 4096x4096 images in one axes. If I run this code the plot takes 300 MB of memory: import numpy as np import matplotlib.pyplot as plt if __name__ == '__main__': img = np.zeros((4096, 4096)) img[100: 300, 100:1500] = 200 imgplot = plt.imshow(img) plt.show() And it takes additional 300 MB for every image with this size added into plot. Is there any way to reduce memory consumption without need of data resampling? My configuration: Matplotlib 1.2.1 Numpy 1.7.1 Ubuntu 13.04 64 bit Best Stepan -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511iu=/4140/ostg.clktrk ___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511iu=/4140/ostg.clktrk___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] plotting large images
Hi, You could, before plotting, sum the different image arrays? Depending on whether you are plotting RGB(A) images or greyscale images, you could take the sum of the color channels, or take a weighted average. Yes, I will probably merge the images (RGBA) before plotting. I want to create more plots and even with this optimization every plot will take 300 MB... Is there any way how to save some memory? Best Stepan -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511iu=/4140/ostg.clktrk___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] plotting large images
Those numbers actually make a lot of sense. For a 4k by 4k 2D array of 64-bit floats, you're using 128MiB of memory, just to store them. Displaying such an array with mpl would take a copy of that and add some objects for housekeeping (on my machine about 150MB to display one such array together with the housekeeping objects). You could look at whether or not you actually need 64-bit precision. Often times, 8-bit precision per color channel is justifiable, even in grayscale. My advice is to play with the dtype of your array or, as you mentioned, resample. Also, is it needed to keep all images? It sounds to me like your application will become very resource hungry if you're going to be displaying several of these 2D images over each other (and if you don't use transparency, you won't get any benefit at all from plotting them together). 2013/8/27 Štěpán Turek stepan.tu...@seznam.cz Hi, You could, before plotting, sum the different image arrays? Depending on whether you are plotting RGB(A) images or greyscale images, you could take the sum of the color channels, or take a weighted average. Yes, I will probably merge the images (RGBA) before plotting. I want to create more plots and even with this optimization every plot will take 300 MB... Is there any way how to save some memory? Best Stepan -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511iu=/4140/ostg.clktrk___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] plotting large images
You could look at whether or not you actually need 64-bit precision. Often times, 8-bit precision per color channel is justifiable, even in grayscale. My advice is to play with the dtype of your array or, as you mentioned, resample. thanks, this helped me significantly, uint8 precision is enough. Also, is it needed to keep all images? It sounds to me like your application will become very resource hungry if you're going to be displaying several of these 2D images over each other (and if you don't use transparency, you won' t get any benefit at all from plotting them together). Yes, I need them all . To avoid it I am thinking about merging them into one image and then plot it. Stepan -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511iu=/4140/ostg.clktrk___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] plotting large images
I've been burned by this before as well. MPL stores some intermediate data products (for example, scaled RGB copies) at full resolution, even though the final rendered image is downsampled depending on screen resolution. I've used some hacky tricks to get around this, which mostly involve downsampling the image on the fly based on screen resolution. One such effort is at https://github.com/ChrisBeaumont/mpl-modest-image. If you are loading your arrays from disk, you can also use memory-mapped arrays -- this prevents you from loading all the data into RAM, and further cuts down on the footprint. cheers, chris On Tue, Aug 27, 2013 at 6:49 AM, Štěpán Turek stepan.tu...@seznam.czwrote: You could look at whether or not you actually need 64-bit precision. Often times, 8-bit precision per color channel is justifiable, even in grayscale. My advice is to play with the dtype of your array or, as you mentioned, resample. thanks, this helped me significantly, uint8 precision is enough. Also, is it needed to keep all images? It sounds to me like your application will become very resource hungry if you're going to be displaying several of these 2D images over each other (and if you don't use transparency, you won't get any benefit at all from plotting them together). Yes, I need them all . To avoid it I am thinking about merging them into one image and then plot it. Stepan -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511iu=/4140/ostg.clktrk ___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511iu=/4140/ostg.clktrk___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] plotting large images
On 08/27/2013 09:49 AM, Chris Beaumont wrote: I've been burned by this before as well. MPL stores some intermediate data products (for example, scaled RGB copies) at full resolution, even though the final rendered image is downsampled depending on screen resolution. I've used some hacky tricks to get around this, which mostly involve downsampling the image on the fly based on screen resolution. One such effort is at https://github.com/ChrisBeaumont/mpl-modest-image. It looks like this wouldn't be too hard to include in matplotlib. I don't think we'd want to change the current behavior, because sometimes its tradeoff curve makes sense, but in other cases, the modest image approach also makes sense. It's just a matter of coming up with an API to switch between the two behaviors. Pull request? Cheers, Mike If you are loading your arrays from disk, you can also use memory-mapped arrays -- this prevents you from loading all the data into RAM, and further cuts down on the footprint. cheers, chris On Tue, Aug 27, 2013 at 6:49 AM, S(te(pán Turek stepan.tu...@seznam.cz mailto:stepan.tu...@seznam.cz wrote: You could look at whether or not you actually need 64-bit precision. Often times, 8-bit precision per color channel is justifiable, even in grayscale. My advice is to play with the dtype of your array or, as you mentioned, resample. thanks, this helped me significantly, uint8 precision is enough. Also, is it needed to keep all images? It sounds to me like your application will become very resource hungry if you're going to be displaying several of these 2D images over each other (and if you don't use transparency, you won't get any benefit at all from plotting them together). Yes, I need them all . To avoid it I am thinking about merging them into one image and then plot it. Stepan -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511iu=/4140/ostg.clktrk ___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net mailto:Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511iu=/4140/ostg.clktrk ___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users -- Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! Discover the easy way to master current and previous Microsoft technologies and advance your career. Get an incredible 1,500+ hours of step-by-step tutorial videos with LearnDevNow. Subscribe today and save! http://pubads.g.doubleclick.net/gampad/clk?id=58040911iu=/4140/ostg.clktrk___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
[Matplotlib-users] Plotting large images
I am trying to use imshow to plot some semi-large fits images. Here is the code: from math import * import numpy as np from matplotlib import pyplot as plt from matplotlib import cm as cm import pyfits cat = /Volumes/Sweden/PS1SC/Data/PS20090603-3/MD09/skycell.092/ fname = o4985g0263o.warp.MD09.skycell.092 hdu = pyfits.open(cat+fname+.fits) print hdu.info() img = hdu[1].data.astype(int) plt.figure(figsize=[12,12]) plt.imshow(img,cmap=cm.cool) plt.savefig(test.png) Which gives the result: Filename: /Volumes/Sweden/PS1SC/Data/PS20090603-3/MD09/skycell.092/ o4985g0263o.warp.MD09.skycell.092.fits No.Name Type Cards Dimensions Format 0PRIMARY PrimaryHDU 6 ()int16 1CompImageHDU 101 (6000, 6000) float32 None Python(23117,0xa04f2720) malloc: *** mmap(size=115200) failed (error code=12) *** error: can't allocate region *** set a breakpoint in malloc_error_break to debug Traceback (most recent call last): File quick_look.py, line 16, in module plt.savefig(test.png) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/pyplot.py, line 345, in savefig return fig.savefig(*args, **kwargs) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/figure.py, line 990, in savefig self.canvas.print_figure(*args, **kwargs) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/backend_bases.py, line 1419, in print_figure **kwargs) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/backends/backend_agg.py, line 323, in print_png FigureCanvasAgg.draw(self) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/backends/backend_agg.py, line 279, in draw self.figure.draw(self.renderer) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/figure.py, line 772, in draw for a in self.axes: a.draw(renderer) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/axes.py, line 1545, in draw im.draw(renderer) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/image.py, line 233, in draw im = self.make_image(renderer.get_image_magnification()) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/image.py, line 180, in make_image x = self.to_rgba(self._A, self._alpha) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/cm.py, line 79, in to_rgba x = self.cmap(x, alpha=alpha, bytes=bytes) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/colors.py, line 501, in __call__ rgba = np.empty(shape=xa.shape+(4,), dtype=lut.dtype) MemoryError I found the earlier thread of http://www.mail-archive.com/matplotlib-users@lists.sourceforge.net/msg11216.html but that didn't seem to produce any fixes or good explanations. I am on a Mac Pro Intel machine running 10.5.7 and [Heimdall:tgrav ~/Work/myCode/Python/pyPS1SC] python ActivePython 2.5.4.3 (ActiveState Software Inc.) based on Python 2.5.4 (r254:67916, Jan 20 2009, 14:11:42) [GCC 4.0.1 (Apple Computer, Inc. build 5250)] on darwin Type help, copyright, credits or license for more information. import numpy numpy.__version__ '1.3.0rc2' import pyfits pyfits.__version__ '2.1.1dev462' import matplotlib matplotlib.__version__ '0.98.5.2' Cheers Tommy -- ___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] Plotting large images
Because the image is so large, and matplotlib carries out various operations on the image (scaling, resampling, etc), it uses a lot of memory. This is one area where a specialized display program will be more efficient. If you need to use matplotlib, decide whether you really only want to display a subsection, or only need a lower resolution version (e.g., boxcar smooth and subsample) before displaying. I've found that image sizes well over 1kx1k can take some time to display, and those that are much larger can cause you to run out of memory. At least, that's what I think is happening. Perry On Jun 30, 2009, at 7:20 PM, Tommy Grav wrote: I am trying to use imshow to plot some semi-large fits images. Here is the code: from math import * import numpy as np from matplotlib import pyplot as plt from matplotlib import cm as cm import pyfits cat = /Volumes/Sweden/PS1SC/Data/PS20090603-3/MD09/skycell.092/ fname = o4985g0263o.warp.MD09.skycell.092 hdu = pyfits.open(cat+fname+.fits) print hdu.info() img = hdu[1].data.astype(int) plt.figure(figsize=[12,12]) plt.imshow(img,cmap=cm.cool) plt.savefig(test.png) Which gives the result: Filename: /Volumes/Sweden/PS1SC/Data/PS20090603-3/MD09/skycell.092/ o4985g0263o.warp.MD09.skycell.092.fits No.Name Type Cards Dimensions Format 0PRIMARY PrimaryHDU 6 ()int16 1CompImageHDU 101 (6000, 6000) float32 None Python(23117,0xa04f2720) malloc: *** mmap(size=115200) failed (error code=12) *** error: can't allocate region *** set a breakpoint in malloc_error_break to debug Traceback (most recent call last): File quick_look.py, line 16, in module plt.savefig(test.png) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/pyplot.py, line 345, in savefig return fig.savefig(*args, **kwargs) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/figure.py, line 990, in savefig self.canvas.print_figure(*args, **kwargs) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/backend_bases.py, line 1419, in print_figure **kwargs) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/backends/backend_agg.py, line 323, in print_png FigureCanvasAgg.draw(self) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/backends/backend_agg.py, line 279, in draw self.figure.draw(self.renderer) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/figure.py, line 772, in draw for a in self.axes: a.draw(renderer) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/axes.py, line 1545, in draw im.draw(renderer) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/image.py, line 233, in draw im = self.make_image(renderer.get_image_magnification()) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/image.py, line 180, in make_image x = self.to_rgba(self._A, self._alpha) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/cm.py, line 79, in to_rgba x = self.cmap(x, alpha=alpha, bytes=bytes) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/colors.py, line 501, in __call__ rgba = np.empty(shape=xa.shape+(4,), dtype=lut.dtype) MemoryError I found the earlier thread of http://www.mail-archive.com/matplotlib-users@lists.sourceforge.net/msg11216.html but that didn't seem to produce any fixes or good explanations. I am on a Mac Pro Intel machine running 10.5.7 and [Heimdall:tgrav ~/Work/myCode/Python/pyPS1SC] python ActivePython 2.5.4.3 (ActiveState Software Inc.) based on Python 2.5.4 (r254:67916, Jan 20 2009, 14:11:42) [GCC 4.0.1 (Apple Computer, Inc. build 5250)] on darwin Type help, copyright, credits or license for more information. import numpy numpy.__version__ '1.3.0rc2' import pyfits pyfits.__version__ '2.1.1dev462' import matplotlib matplotlib.__version__ '0.98.5.2' Cheers Tommy -- ___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users -- ___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] Plotting large images
That is what I was assuming, but it still seems a little odd that matplotlib generates that large of a memory footprint. Loading the fits file into the program using pyfits, with the code only uses 19MB of real memory and 600MB of virtual memory (strangly adding the line img = hdu[1].data, increases this to 208MB/800MB). Displaying images of various sizes I get these numbers from Activity Monitor Size Real Mem Virtual 3k x 3k 0.68GB1.57GB 4k x 4k 0.92GB1.80GB 5k x 5k 1.20GB2.10GB 5.5k x 5.5k 1.38GB2.28GB And the limit seems to be somewhere just above 5.5k by 5.5k (darn :( ) Cheers Tommy On Jun 30, 2009, at 7:27 PM, Perry Greenfield wrote: Because the image is so large, and matplotlib carries out various operations on the image (scaling, resampling, etc), it uses a lot of memory. This is one area where a specialized display program will be more efficient. If you need to use matplotlib, decide whether you really only want to display a subsection, or only need a lower resolution version (e.g., boxcar smooth and subsample) before displaying. I've found that image sizes well over 1kx1k can take some time to display, and those that are much larger can cause you to run out of memory. At least, that's what I think is happening. Perry On Jun 30, 2009, at 7:20 PM, Tommy Grav wrote: I am trying to use imshow to plot some semi-large fits images. Here is the code: from math import * import numpy as np from matplotlib import pyplot as plt from matplotlib import cm as cm import pyfits cat = /Volumes/Sweden/PS1SC/Data/PS20090603-3/MD09/skycell.092/ fname = o4985g0263o.warp.MD09.skycell.092 hdu = pyfits.open(cat+fname+.fits) print hdu.info() img = hdu[1].data.astype(int) plt.figure(figsize=[12,12]) plt.imshow(img,cmap=cm.cool) plt.savefig(test.png) Which gives the result: Filename: /Volumes/Sweden/PS1SC/Data/PS20090603-3/MD09/skycell.092/ o4985g0263o.warp.MD09.skycell.092.fits No.Name Type Cards Dimensions Format 0PRIMARY PrimaryHDU 6 ()int16 1CompImageHDU 101 (6000, 6000) float32 None Python(23117,0xa04f2720) malloc: *** mmap(size=115200) failed (error code=12) *** error: can't allocate region *** set a breakpoint in malloc_error_break to debug Traceback (most recent call last): File quick_look.py, line 16, in module plt.savefig(test.png) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/pyplot.py, line 345, in savefig return fig.savefig(*args, **kwargs) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/figure.py, line 990, in savefig self.canvas.print_figure(*args, **kwargs) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/backend_bases.py, line 1419, in print_figure **kwargs) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/backends/backend_agg.py, line 323, in print_png FigureCanvasAgg.draw(self) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/backends/backend_agg.py, line 279, in draw self.figure.draw(self.renderer) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/figure.py, line 772, in draw for a in self.axes: a.draw(renderer) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/axes.py, line 1545, in draw im.draw(renderer) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/image.py, line 233, in draw im = self.make_image(renderer.get_image_magnification()) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/image.py, line 180, in make_image x = self.to_rgba(self._A, self._alpha) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/cm.py, line 79, in to_rgba x = self.cmap(x, alpha=alpha, bytes=bytes) File /Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/matplotlib/colors.py, line 501, in __call__ rgba = np.empty(shape=xa.shape+(4,), dtype=lut.dtype) MemoryError I found the earlier thread of http://www.mail-archive.com/matplotlib-users@lists.sourceforge.net/msg11216.html but that didn't seem to produce any fixes or good explanations. I am on a Mac Pro Intel machine running 10.5.7 and [Heimdall:tgrav ~/Work/myCode/Python/pyPS1SC] python ActivePython 2.5.4.3 (ActiveState Software Inc.) based on Python 2.5.4 (r254:67916, Jan 20 2009, 14:11:42) [GCC 4.0.1 (Apple Computer, Inc. build 5250)] on darwin Type help, copyright, credits or license for more information. import numpy numpy.__version__
Re: [Matplotlib-users] Plotting large images
On Jun 30, 2009, at 7:54 PM, Tommy Grav wrote: That is what I was assuming, but it still seems a little odd that matplotlib generates that large of a memory footprint. Loading the fits file into the program using pyfits, with the code only uses 19MB of real memory and 600MB of virtual memory (strangly adding the line img = hdu[1].data, increases this to 208MB/800MB). The reason for this is that pyfits doesn't actually load the data until you 'touch' the data attribute (to minimize memory, particularly if you just are interested in the header information). As for the memory footprint of matplotlib, in order to be able to resize and handle interactive updates, it has to retain references to the original image, perhaps as well to intermediate products (and these references won't be memory collected until you clear the figure (e.g., clf()). It's one of the prices for flexibility and generality. It probably would take a lot of complexity to optimize it for large images (but John is better suited to answer this conclusively). Perry Displaying images of various sizes I get these numbers from Activity Monitor Size Real Mem Virtual 3k x 3k 0.68GB1.57GB 4k x 4k 0.92GB1.80GB 5k x 5k 1.20GB2.10GB 5.5k x 5.5k 1.38GB2.28GB And the limit seems to be somewhere just above 5.5k by 5.5k (darn :( ) Cheers Tommy -- ___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
