[Numpy-discussion] memoryerror with numpy.fromfile
I am getting a MemoryError from a numpy.fromfile() call in an application I am trying to deploy. Normally I would assume that this would mean that I don't have enough memory available on the system. However, if I run vmstat (Linux) at the same time as my process, I see that I have 3+ Gigabytes of memory free, and no swap space being used. I can't think of a way to track down this problem, so I'm punting to the list. The only thing I can imagine is that someone Python has been allocated X amount of space (very small relative to the memory actually available), and is asking for more than X. I don't know if this is true, or if there is even a way to check it. Sigh. Version info: Linux RedHat 5 kernel 2.6.18-128.1.10.el5PAE Python 2.5.4 -- EPD_Py25 4.3.0 numpy 1.3.0 Can anyone suggest more tests that I can do? Thanks, Mike ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] memoryerror with numpy.fromfile
Thu, 21 May 2009 10:31:28 -0600, Michael Hearne wrote: I am getting a MemoryError from a numpy.fromfile() call in an application I am trying to deploy. Normally I would assume that this would mean that I don't have enough memory available on the system. However, if I run vmstat (Linux) at the same time as my process, I see that I have 3+ Gigabytes of memory free, and no swap space being used. If you are on a 32-bit platform, the maximum addressable memory for a single process is limited to 3 GB, and what can be allocated can be less than this because of memory fragmentation. Also, you should check that you don't have an ulimit set for virtual/RSS memory. -- Pauli Virtanen ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] memoryerror with numpy.fromfile
On Thu, May 21, 2009 at 10:31 AM, Michael Hearne mhea...@usgs.gov wrote: I am getting a MemoryError from a numpy.fromfile() call in an application I am trying to deploy. Normally I would assume that this would mean that I don't have enough memory available on the system. However, if I run vmstat (Linux) at the same time as my process, I see that I have 3+ Gigabytes of memory free, and no swap space being used. I can't think of a way to track down this problem, so I'm punting to the list. The only thing I can imagine is that someone Python has been allocated X amount of space (very small relative to the memory actually available), and is asking for more than X. I don't know if this is true, or if there is even a way to check it. Sigh. Version info: Linux RedHat 5 kernel 2.6.18-128.1.10.el5PAE Python 2.5.4 -- EPD_Py25 4.3.0 numpy 1.3.0 Can anyone suggest more tests that I can do? How big is the file and what type are you importing to? Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] memoryerror with numpy.fromfile
All: Never mind! The file I was attempting to read was part of an (apparently) silently incomplete download, and as such, the file size didn't match the metadata in the header file describing the data file, and I was reading beyond the end of the file. I would submit that MemoryError is perhaps a little misleading for this particular case, but oh well. Thanks for the comments! --Mike On May 21, 2009, at 11:46 AM, Charles R Harris wrote: On Thu, May 21, 2009 at 10:31 AM, Michael Hearne mhea...@usgs.gov wrote: I am getting a MemoryError from a numpy.fromfile() call in an application I am trying to deploy. Normally I would assume that this would mean that I don't have enough memory available on the system. However, if I run vmstat (Linux) at the same time as my process, I see that I have 3+ Gigabytes of memory free, and no swap space being used. I can't think of a way to track down this problem, so I'm punting to the list. The only thing I can imagine is that someone Python has been allocated X amount of space (very small relative to the memory actually available), and is asking for more than X. I don't know if this is true, or if there is even a way to check it. Sigh. Version info: Linux RedHat 5 kernel 2.6.18-128.1.10.el5PAE Python 2.5.4 -- EPD_Py25 4.3.0 numpy 1.3.0 Can anyone suggest more tests that I can do? How big is the file and what type are you importing to? Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] memoryerror with numpy.fromfile
Thu, 21 May 2009 12:14:18 -0600, Michael Hearne wrote: All: Never mind! The file I was attempting to read was part of an (apparently) silently incomplete download, and as such, the file size didn't match the metadata in the header file describing the data file, and I was reading beyond the end of the file. I would submit that MemoryError is perhaps a little misleading for this particular case, but oh well. Well, that it raises MemoryError in this case is a bug, so maybe a ticket should be filed. There's another bug associated with reading empty files: http://projects.scipy.org/numpy/ticket/1115 which is maybe related. -- Pauli Virtanen ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
