On 2012/09/07 4:00 AM, Benjamin Root wrote: > > > On Fri, Sep 7, 2012 at 9:49 AM, Shahar Shani-Kadmiel > <kadm...@post.bgu.ac.il <mailto:kadm...@post.bgu.ac.il>> wrote: > > On Sep 7, 2012, at 4:25 PM, Benjamin Root wrote: > >> >> >> On Fri, Sep 7, 2012 at 8:44 AM, Shahar Shani-Kadmiel >> <kadm...@post.bgu.ac.il <mailto:kadm...@post.bgu.ac.il>> wrote: >> >> 1. an ipython session is invoked with qtconsole --pylab >> 2. I load a large NetCDF grid (Grid file format: nf (# 18) GMT >> netCDF format (float) (COARDS-compliant) [DEFAULT]), approx. >> 1.15 GB >> 3. I then try to plot with imshow the data >> >> added below are the lines leading up to the error and the >> error itself. >> >> >> This is running on OS X 10.7.4 with a recently installed EPD 7.3. >> >> >> {code} >> from scipy.io <http://scipy.io/> import netcdf_file as netcdf >> data = >> netcdf('srtm_43_44_05_06_07_08.grd','r').variables['z'][::-1] >> >> fig, ax = subplots() >> >> data.shape >> Out[5]: (24004, 12002) >> >> im = ax.imshow(data, >> aspect=((data.shape[1])/float(data.shape[0])), >> interpolation='none') >> >> --------------------------------------------------------------------------- >> MemoryError Traceback (most >> recent call last) >> <ipython-input-6-f92e4c4c63b5> in <module>() >> ----> 1 im = ax.imshow(data, >> aspect=((data.shape[1])/float(data.shape[0])), >> interpolation='none') >> >> >> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/axes.py >> in imshow(self, X, cmap, norm, aspect, interpolation, alpha, >> vmin, vmax, origin, extent, shape, filternorm, filterrad, >> imlim, resample, url, **kwargs) >> 6743 im.set_clim(vmin, vmax) >> 6744 else: >> -> 6745 im.autoscale_None() >> 6746 im.set_url(url) >> 6747 >> >> >> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/cm.py >> in autoscale_None(self) >> 281 if self._A is None: >> 282 raise TypeError('You must first set_array >> for mappable') >> --> 283 self.norm.autoscale_None(self._A) >> 284 self.changed() >> 285 >> >> >> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/matplotlib/colors.py >> in autoscale_None(self, A) >> 889 ' autoscale only None-valued vmin or vmax' >> 890 if self.vmin is None: >> --> 891 self.vmin = ma.min(A) >> 892 if self.vmax is None: >> 893 self.vmax = ma.max(A) >> >> >> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc >> in min(obj, axis, out, fill_value) >> 5873 def min(obj, axis=None, out=None, fill_value=None): >> 5874 try: >> -> 5875 return obj.min(axis=axis, >> fill_value=fill_value, out=out) >> 5876 except (AttributeError, TypeError): >> 5877 # If obj doesn't have a max method, >> >> >> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc >> in min(self, axis, out, fill_value) >> 5054 # No explicit output >> 5055 if out is None: >> -> 5056 result = >> self.filled(fill_value).min(axis=axis, out=out).view(type(self)) >> 5057 if result.ndim: >> 5058 # Set the mask >> >> >> /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/numpy/ma/core.pyc >> in filled(self, fill_value) >> 3388 return self._data >> 3389 else: >> -> 3390 result = self._data.copy() >> 3391 try: >> 3392 np.putmask(result, m, fill_value) >> >> MemoryError: >> {/code} >> >> >> This is more a NumPy issue than anything else. We need to know >> the min and the max of the array in order to automatically scale >> the colormap for display. Therefore, we query the array object >> for its min/max. Because we support masked arrays, the array is >> first cast as a masked array, and then these queries are done. >> >> It appears that in numpy's masked array module, it calculates the >> array's min by making a copy of itself first. I would have >> figured that it would have done its task differently. In the >> meantime, I suspect you can work around this problem by explicitly >> setting the vmin/vmax keyword arguments to imshow if you know >> them. Therefore, there should be no need to determine the array's >> min/max in this inefficient manner. >> >> Ben Root >> > > Hi Ben, > I tried adding vmin & vmax the the imshow call but I still get a > MemoryError. > The grid file is 1.15 GB and I have ~4.5 out of 8 GB of memory > available when I launch ipython, 3.5 when I execute imshow and 2 > when I execute plt.draw().
mpl simply is not designed for image-type operations on huge arrays, vastly larger than what can be displayed. It is up to the user to down-sample or otherwise reduce the size of the array fed to imshow. > > > Well, it looks like setting vmin/vmax helped, because your traceback Did a new traceback get posted in a message that was not sent to the list? I found only the original traceback. > shows that the code made significant progress. The issue here is that > the process_value() method doesn't make a lot of sense. I am not sure > what is the rationale behind its logic. Hopefully, someone else can > chime in with an explanation of what is going on. Normalization has to handle all sorts of inputs--masked or not, all sorts of numbers, scalar or array--and it is much easier to do this efficiently if all these possibilities are reduced to a very few at the start. Specifically, it needs to supply a copy of the input (so that normalization doesn't change the original) in a floating point masked array, using float32 if possible for space efficiency. It needs to keep track of whether the input was a scalar, so that normalization can return a scalar when given a scalar input. Eric > > In the meantime, are you using a 32 or 64-bit machine? > > Ben Root > > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > > > > _______________________________________________ > Matplotlib-users mailing list > Matplotlib-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/matplotlib-users > ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
