Re: [Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?
On Thu, 2008-06-05 at 08:24 +0200, Michael Abshoff wrote: I am not what I would call familiar with numpy internals, so is there a magic thing I can do to make numpy aware that ctypes exists? I don't think any magic is needed. As long as importing ctypes works from the python you used to build numpy, it should work. David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?
Michael Abshoff wrote: Jonathan Wright wrote: ...etc. We needed this for generating the .so library file name for ctypes Can you elaborate on this a little? The we refered to another project (not numpy) where we needed to distinguish 32 bit from 64 bit platforms. We have code for picking which shared library file to use similar to that below. Apologies if I've confused the issue. -Jon if sys.platform == linux2: if platform.architecture()[0] == '32bit': libfilename = 'libKLT32.so' if platform.architecture()[0] == '64bit': libfilename = 'libKLT64.so' if sys.platform == win32: libfilename = 'klt.dll' _libraries['klt_bin'] = CDLL( os.path.join( os.path.split(__file__)[0], libfilename ) ) # If you still don't have it you probably have a mac?? ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?
I'm using python 2.5.2 on OS X, with 8 GB of ram, and a 64-bit processor. In this, setting, I'm working with large arrays of binary data. E.g, I want to make calls like: Z = numpy.inner(a,b) where and b are fairly large -- e.g. 2 rows by 100 columns. However, when such a call is made, I get a memory error that I don't understand. Specifically: s = numpy.random.binomial(1,.5,(2,100)) #creates 2x100 bin. array r = numpy.inner(s,s) Python(1714) malloc: *** mmap(size=16) failed (error code=12) *** error: can't allocate region *** set a breakpoint in malloc_error_break to debug Naively, the numpy.inner call should be fine on my system, since my computer has enough memory. (And, when it's run, I have checked to see that at least 5 GB of RAM is free.) The error message thus suggests there's some problem to do with memory mapping going on here: that somehow, numpy.inner is calling the mmap modeul, and that the address space is being exceeded. And that all my extra RAM isn't being used here at all. So, I have three questions about this: 1) Why is mmap being called in the first place? I've written to Travis Oliphant, and he's explained that numpy.inner does NOT directly do any memory mapping and shouldn't call mmap. Instead, it should just operate with things in memory -- in which case my 8 GB should allow the computation to go through just fine. What's going on? 2) How can I stop this from happening? I want to be able to leverage large amounts of ram on my machine to scale up my computations and not be dependent on the limitations of the address space size. If the mmap is somehow being called by the OS, is there some option I can set that will make it do things in regular memory instead? (Sorry if this is a stupid question.) 3) Even if I had to use memory mapping, why is the 1.6 GB requirement failing? I'm using a recent enough version of python, and I have a 64-bit processor with sufficient amount of memory. I should be able to allocate at least 4 GB of address space, right? But the system seems to be balking at the 1.6 GB request. (Again, sorry if this is stupid.) Any help would be greatly appreciated! Thanks, Dan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?
On Wed, Jun 4, 2008 at 6:42 PM, Dan Yamins [EMAIL PROTECTED] wrote: I'm using python 2.5.2 on OS X, with 8 GB of ram, and a 64-bit processor. In this, setting, I'm working with large arrays of binary data. E.g, I want to make calls like: Z = numpy.inner(a,b) where and b are fairly large -- e.g. 2 rows by 100 columns. However, when such a call is made, I get a memory error that I don't understand. Specifically: s = numpy.random.binomial(1,.5,(2,100)) #creates 2x100 bin. array r = numpy.inner(s,s) Python(1714) malloc: *** mmap(size=16) failed (error code=12) *** error: can't allocate region *** set a breakpoint in malloc_error_break to debug Naively, the numpy.inner call should be fine on my system, since my computer has enough memory. (And, when it's run, I have checked to see that at least 5 GB of RAM is free.) The error message thus suggests there's some problem to do with memory mapping going on here: that somehow, numpy.inner is calling the mmap modeul, and that the address space is being exceeded. And that all my extra RAM isn't being used here at all. So, I have three questions about this: 1) Why is mmap being called in the first place? I've written to Travis Oliphant, and he's explained that numpy.inner does NOT directly do any memory mapping and shouldn't call mmap. Instead, it should just operate with things in memory -- in which case my 8 GB should allow the computation to go through just fine. What's going on? 2) How can I stop this from happening? I want to be able to leverage large amounts of ram on my machine to scale up my computations and not be dependent on the limitations of the address space size. If the mmap is somehow being called by the OS, is there some option I can set that will make it do things in regular memory instead? (Sorry if this is a stupid question.) 3) Even if I had to use memory mapping, why is the 1.6 GB requirement failing? I'm using a recent enough version of python, and I have a 64-bit processor with sufficient amount of memory. I should be able to allocate at least 4 GB of address space, right? But the system seems to be balking at the 1.6 GB request. (Again, sorry if this is stupid.) Are both python and your version of OS X fully 64 bits? Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?
2008/6/4 Dan Yamins [EMAIL PROTECTED]: So, I have three questions about this: 1) Why is mmap being called in the first place? I've written to Travis Oliphant, and he's explained that numpy.inner does NOT directly do any memory mapping and shouldn't call mmap. Instead, it should just operate with things in memory -- in which case my 8 GB should allow the computation to go through just fine. What's going on? 2) How can I stop this from happening? I want to be able to leverage large amounts of ram on my machine to scale up my computations and not be dependent on the limitations of the address space size. If the mmap is somehow being called by the OS, is there some option I can set that will make it do things in regular memory instead? (Sorry if this is a stupid question.) I don't know much about OSX, but I do know that many malloc() implementations take advantage of a modern operating system's virtual memory when allocating large blocks of memory. For small blocks, malloc uses memory arenas, but if you ask for a large block malloc() will request a whole bunch of pages from the operating system. This way when the memory is freed, free() can easily return the chunk of memory to the OS. On some systems, one way to get such a big hunk of memory from the system is with an anonymous mmap(). I think that's what's going on here. So I don't think you want to stop malloc() from using mmap(). You do, of course, want the memory allocation to succeed, and I'm afraid I don't have any idea why it can't. Under Linux, I know that you can run a 64-bit processor in 32-bit mode, which gives you the usual 4 GB address space limit. I've no idea what OSX does. Good luck, Anne ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?
I don't know much about OSX, but I do know that many malloc() implementations take advantage of a modern operating system's virtual memory when allocating large blocks of memory. For small blocks, malloc uses memory arenas, but if you ask for a large block malloc() will request a whole bunch of pages from the operating system. This way when the memory is freed, free() can easily return the chunk of memory to the OS. On some systems, one way to get such a big hunk of memory from the system is with an anonymous mmap(). I think that's what's going on here. So I don't think you want to stop malloc() from using mmap(). You do, of course, want the memory allocation to succeed, and I'm afraid I don't have any idea why it can't. Under Linux, I know that you can run a 64-bit processor in 32-bit mode, which gives you the usual 4 GB address space limit. I've no idea what OSX does. Good luck, Anne Anne, thanks so much for your help. I still a little confused. If your scenario about the the memory allocation is working is right, does that mean that even if I put a lot of ram on the machine, e.g. 16GB, I still can't request it in blocks larger than the limit imposed by the processor architecture (e.g. 4 GB for 32, 8 GB for 64-bit)?What I really want is to be able to have my ability to request memory limited just by the amount of memory on the machine, and not have it depend on something about paging/memory mapping limits. Is this a stupid/naive thing? (Sorry for my ignorance, and thanks again for the help!) best, Dan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?
On Wed, Jun 4, 2008 at 9:06 PM, Charles R Harris [EMAIL PROTECTED] wrote: On Wed, Jun 4, 2008 at 6:42 PM, Dan Yamins [EMAIL PROTECTED] wrote: I'm using python 2.5.2 on OS X, with 8 GB of ram, and a 64-bit processor. In this, setting, I'm working with large arrays of binary data. E.g, I want to make calls like: Z = numpy.inner(a,b) where and b are fairly large -- e.g. 2 rows by 100 columns. However, when such a call is made, I get a memory error that I don't understand. Specifically: s = numpy.random.binomial(1,.5,(2,100)) #creates 2x100 bin. array r = numpy.inner(s,s) Python(1714) malloc: *** mmap(size=16) failed (error code=12) *** error: can't allocate region *** set a breakpoint in malloc_error_break to debug Are both python and your version of OS X fully 64 bits? I'm not sure. My version of OS X is the most recent version, the one that ships with a new MacPro Dual Quad-core Xeon 3.2MHz chipset. The processor is definitely 64-bit, so I think the operating system probably is enable for that, but am not sure. (How would I find out?) As for the python version, I thought that 2.5 and above were 64-enabled, but I'm not sure how I'd check it. Thanks! Dan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?
2008/6/4 Dan Yamins [EMAIL PROTECTED]: Anne, thanks so much for your help. I still a little confused. If your scenario about the the memory allocation is working is right, does that mean that even if I put a lot of ram on the machine, e.g. 16GB, I still can't request it in blocks larger than the limit imposed by the processor architecture (e.g. 4 GB for 32, 8 GB for 64-bit)?What I really want is to be able to have my ability to request memory limited just by the amount of memory on the machine, and not have it depend on something about paging/memory mapping limits. Is this a stupid/naive thing? No, that's a perfectly reasonable thing to want, and it *should* be possible with your hardware. Maybe I'm barking up the wrong tree and the problem is something else. But if you'll bear with me: For the last ten years or so, normal computers have used 32 bits to address memory. This means that any given process can only get at 2**32 different bytes of memory, which is 4 GB. The magic of virtual memory means you can possibly have a number of different processes addressing different 4 GB chunks, some of which may temporarily be stored on disk at any given time. If you want to get at a chunk of memory bigger than 4 GB, you need all of these things: * Your CPU must be 64-bit, or else it has no hope of being able to access more than 4 GB (in fact more than 3 GB for complicated reasons) of physical memory. * Your operating system must be 64-bit (64-bit PC CPUs have a 32-bit compatibility mode they often find themselves running in) so that it can know about all the real RAM you have installed and so that it can support 64-bit programs. * Your program must be 64-bit, so that it can access huge chunks of RAM. For compatibility reasons, many 64-bit machines have most of their software compiled in 32-bit mode. This isn't necessarily a problem - the programs probably even run faster - but they can't access more than 4 GB each. So it's worth finding out whether your python interpreter is compiled as a 64-bit program (and whether that's even possible under your version of OSX). Unfortunately I don't know how to find that out in your case. Anne ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?
On Wed, Jun 4, 2008 at 7:41 PM, Dan Yamins [EMAIL PROTECTED] wrote: On Wed, Jun 4, 2008 at 9:06 PM, Charles R Harris [EMAIL PROTECTED] wrote: On Wed, Jun 4, 2008 at 6:42 PM, Dan Yamins [EMAIL PROTECTED] wrote: I'm using python 2.5.2 on OS X, with 8 GB of ram, and a 64-bit processor. In this, setting, I'm working with large arrays of binary data. E.g, I want to make calls like: Z = numpy.inner(a,b) where and b are fairly large -- e.g. 2 rows by 100 columns. However, when such a call is made, I get a memory error that I don't understand. Specifically: s = numpy.random.binomial(1,.5,(2,100)) #creates 2x100 bin. array r = numpy.inner(s,s) Python(1714) malloc: *** mmap(size=16) failed (error code=12) *** error: can't allocate region *** set a breakpoint in malloc_error_break to debug Are both python and your version of OS X fully 64 bits? I'm not sure. My version of OS X is the most recent version, the one that ships with a new MacPro Dual Quad-core Xeon 3.2MHz chipset. The processor is definitely 64-bit, so I think the operating system probably is enable for that, but am not sure. (How would I find out?) As for the python version, I thought that 2.5 and above were 64-enabled, but I'm not sure how I'd check it. Hmm, In [1]: s = numpy.random.binomial(1,.5,(2,100)) In [2]: inner(s,s) Out[2]: array([[45, 22, 17, ..., 20, 26, 23], [22, 52, 26, ..., 23, 33, 24], [17, 26, 52, ..., 27, 27, 19], ..., [20, 23, 27, ..., 46, 26, 22], [26, 33, 27, ..., 26, 54, 25], [23, 24, 19, ..., 22, 25, 44]]) This on 32 bit fedora 8 with 2GiB of actual memory. It was slow and a couple of hundred megs of something went into swap, but it did complete. So this looks to me like an OS X problem. Are there any limitations on the user memory sizes? There might be some system setting accounting for this. Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?
On Wed, 2008-06-04 at 21:38 -0400, Dan Yamins wrote: Anne, thanks so much for your help. I still a little confused. If your scenario about the the memory allocation is working is right, does that mean that even if I put a lot of ram on the machine, e.g. 16GB, I still can't request it in blocks larger than the limit imposed by the processor architecture (e.g. 4 GB for 32, 8 GB for 64-bit)? Definitely. For 32 bits, it is actually smaller than 4 Gb, because the whole address range is used by the processus and the OS, and the OS generally needs at least 1 Gb (I don't know the default for mac os X, but both linux and windows default to 2 Gb: so a processus cannot use more than 2 Gb of memory). The whole point of having a 64 bits CPU in 64 bits mode is to have more than 32 bits for the (virtual) address space. If it is not in 64 bits mode, the pointer is 32 bits (void* is 4 bytes, as returned by malloc), and you would have a hard time accessing anything above 32 bits. It is possible to address more than 32 bits on 32 bits OS using some weird extensions, with a semi segmented address (you address the memory with an offset + an address, like in the old times), but I have never used it. IOW: with more than 4 Gb of real memory, the hardware can address it, in 32 bits or 64 bits, but *you* cannot address it in 32 bits mode because your pointers are still 4 bytes. To address more than 4 Gb, you need 8 bytes pointers. So the real question is how to do it on mac os X, and unfortunately, I cannot help you. I know that mac os X supports executing both 32 bits and 64 bits (super fat binaries with the ability to put up to 4 differents kind of binaries: ppc and x86, 32 bits and 64 bits in both cases), but I don't know which subsystem you can use, and how to compile it (you will certainly have to compile your application differently). I remember having seen mentioned several times that for Leopard, you cannot build a 64 bits binary using GUI, because no GUI subsystem is 64 bits compatible yet. What I really want is to be able to have my ability to request memory limited just by the amount of memory on the machine, and not have it depend on something about paging/memory mapping limits. Is this a stupid/naive thing? It is not stupid, I think everyone whished it was possible. Unfortunately, in mac os X case at least, you cannot do it so simply (yet). David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?
Dan Yamins wrote: I'm using python 2.5.2 on OS X, with 8 GB of ram, and a 64-bit processor. In this, setting, I'm working with large arrays of binary data. E.g, I want to make calls like: Z = numpy.inner(a,b) where and b are fairly large -- e.g. 2 rows by 100 columns. Hey Dan. Now, that you mention you are using OS X, I'm fairly confident that the problem is that you are using a 32-bit version of Python (i.e. you are not running in full 64-bit mode and so the 4GB limit applies). The most common Python on OS X is 32-bit python. I think a few people in the SAGE project have successfully built Python in 64-bit mode on OSX (but I don't think they have released anything yet). You would have to use a 64-bit version of Python to compile NumPy if you want to access large memory. -Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?
On Wed, Jun 4, 2008 at 10:07 PM, David Cournapeau [EMAIL PROTECTED] wrote: On Wed, 2008-06-04 at 21:38 -0400, Dan Yamins wrote: Anne, thanks so much for your help. I still a little confused. If your scenario about the the memory allocation is working is right, does that mean that even if I put a lot of ram on the machine, e.g. 16GB, I still can't request it in blocks larger than the limit imposed by the processor architecture (e.g. 4 GB for 32, 8 GB for 64-bit)? Definitely. For 32 bits, it is actually smaller than 4 Gb, because the whole address range is used by the processus and the OS, and the OS generally needs at least 1 Gb (I don't know the default for mac os X, but both linux and windows default to 2 Gb: so a processus cannot use more than 2 Gb of memory). It is not stupid, I think everyone whished it was possible. Unfortunately, in mac os X case at least, you cannot do it so simply (yet). David Anne and David, thank you both so much for your very lucid explanations. (I apologize for my ignorance -- with your help and some additional reading, I think I understand how memory allocation works a bit better now.) Best, Dan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?
On Wed, Jun 4, 2008 at 7:41 PM, Dan Yamins [EMAIL PROTECTED] wrote: On Wed, Jun 4, 2008 at 9:06 PM, Charles R Harris [EMAIL PROTECTED] wrote: On Wed, Jun 4, 2008 at 6:42 PM, Dan Yamins [EMAIL PROTECTED] wrote: I'm using python 2.5.2 on OS X, with 8 GB of ram, and a 64-bit processor. In this, setting, I'm working with large arrays of binary data. E.g, I want to make calls like: Z = numpy.inner(a,b) where and b are fairly large -- e.g. 2 rows by 100 columns. However, when such a call is made, I get a memory error that I don't understand. Specifically: s = numpy.random.binomial(1,.5,(2,100)) #creates 2x100 bin. array r = numpy.inner(s,s) Python(1714) malloc: *** mmap(size=16) failed (error code=12) *** error: can't allocate region *** set a breakpoint in malloc_error_break to debug Are both python and your version of OS X fully 64 bits? I'm not sure. My version of OS X is the most recent version, the one that ships with a new MacPro Dual Quad-core Xeon 3.2MHz chipset. The processor is definitely 64-bit, so I think the operating system probably is enable for that, but am not sure. (How would I find out?) As for the python version, I thought that 2.5 and above were 64-enabled, but I'm not sure how I'd check it. Try In [3]: numpy.dtype(numpy.uintp).itemsize Out[3]: 4 which is the size in bytes of the integer needed to hold a pointer. The output above is for 32 bit python/numpy. Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?
Hey Dan. Now, that you mention you are using OS X, I'm fairly confident that the problem is that you are using a 32-bit version of Python (i.e. you are not running in full 64-bit mode and so the 4GB limit applies). The most common Python on OS X is 32-bit python. I think a few people in the SAGE project have successfully built Python in 64-bit mode on OSX (but I don't think they have released anything yet). You would have to use a 64-bit version of Python to compile NumPy if you want to access large memory. -Travis Travis, thanks for the message. I think you're probably right -- I didn't build python myself but instead downloaded the universal OSX binary from the python download page -- and that surely wasn't built for 64-bit system. So I guess I'll have to figure out to do the 64-bit build. Which leaves a question that I think Chuck brought up in a way: In [1]: s = numpy.random.binomial(1,.5,(2,100)) In [2]: inner(s,s) Out[2]: array([[45, 22, 17, ..., 20, 26, 23], [22, 52, 26, ..., 23, 33, 24], [17, 26, 52, ..., 27, 27, 19], ..., [20, 23, 27, ..., 46, 26, 22], [26, 33, 27, ..., 26, 54, 25], [23, 24, 19, ..., 22, 25, 44]]) This on 32 bit fedora 8 with 2GiB of actual memory. It was slow and a couple of hundred megs of something went into swap, but it did complete. So this looks to me like an OS X problem. Are there any limitations on the user memory sizes? There might be some system setting accounting for this. Chuck, is this another way of asking: why is my OS X system not paging memory the way you'd expect a system to respond to the malloc command?Is python somehow overloaded the malloc command so that when the OS says a swap would have to occur, somehow instead of just the swap, that an error message involving mmap is somehow triggered? (Sorry if this makes no sense.) I should add that the tried the same code on a 32-bit windows machine and got the same error as on OS X. Maybe the Linux python builds manage this stuff better. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?
2008/6/4 Dan Yamins [EMAIL PROTECTED]: Try In [3]: numpy.dtype(numpy.uintp).itemsize Out[3]: 4 which is the size in bytes of the integer needed to hold a pointer. The output above is for 32 bit python/numpy. Chuck Check, the answer is 4, as you got for the 32-bit. What would the answer be on a 64-bit architecture? Why is this diagnostic? In a 64-bit setting, a pointer needs to be 64 bits long, that is, eight bytes, not four. What Charles pointed out was that while the inner product is very big, it seems to fit into memory on his 32-bit Linux machine; is it possible that OSX is preventing your python process from using even the meager 2-3 GB that a 32-bit process ought to get? In particular, try running Charles' script in a fresh python interpreter and see if it works; it may be that other arrays you had allocated are taking up some of the space that this one could. You will probably still want a 64-bit python, though, in order to have a little elbow room. Anne ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?
On Wed, Jun 4, 2008 at 9:50 PM, Dan Yamins [EMAIL PROTECTED] wrote: In [3]: numpy.dtype(numpy.uintp).itemsize Out[3]: 4 which is the size in bytes of the integer needed to hold a pointer. The output above is for 32 bit python/numpy. Check, the answer is 4, as you got for the 32-bit. What would the answer be on a 64-bit architecture? Why is this diagnostic? It would be 8 on a 64-bit architecture (with a 64-bit binary): 8 bytes = 64 bits, 4 bytes = 32 bits. -- Nathan Bell [EMAIL PROTECTED] http://graphics.cs.uiuc.edu/~wnbell/ ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?
Hi Dan, On Wed, Jun 4, 2008 at 8:50 PM, Dan Yamins [EMAIL PROTECTED] wrote: Try In [3]: numpy.dtype(numpy.uintp).itemsize Out[3]: 4 which is the size in bytes of the integer needed to hold a pointer. The output above is for 32 bit python/numpy. Chuck Check, the answer is 4, as you got for the 32-bit. What would the answer be on a 64-bit architecture? Why is this diagnostic? On 64 bit ubuntu In [1]: import numpy In [2]: numpy.dtype(numpy.uintp).itemsize Out[2]: 8 This is diagnostic because a pointer of 4 bytes, 32 bits, can only address 4 GiB whereas an 8 byte pointer has 64 bits to address memory. So it looks like your numpy was compiled by a 32 python executable. I've been been googling this a bit and it looks like the standard executable python on the mac is 32 bits, even though 64 bit libraries are available if you can get a 64 bit version up and running. A lot of libraries are available in both 32 and 64 bit versions. Chuck Thanks! Dan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?
On Wed, Jun 4, 2008 at 9:07 PM, Anne Archibald [EMAIL PROTECTED] wrote: 2008/6/4 Dan Yamins [EMAIL PROTECTED]: Try In [3]: numpy.dtype(numpy.uintp).itemsize Out[3]: 4 which is the size in bytes of the integer needed to hold a pointer. The output above is for 32 bit python/numpy. Chuck Check, the answer is 4, as you got for the 32-bit. What would the answer be on a 64-bit architecture? Why is this diagnostic? In a 64-bit setting, a pointer needs to be 64 bits long, that is, eight bytes, not four. What Charles pointed out was that while the inner product is very big, it seems to fit into memory on his 32-bit Linux machine; is it possible that OSX is preventing your python process from using even the meager 2-3 GB that a 32-bit process ought to get? In particular, try running Charles' script in a fresh python interpreter and see if it works; it may be that other arrays you had allocated are taking up some of the space that this one could. You will probably still want a 64-bit python, though, in order to have a little elbow room. I think the difference is that Linux takes up 1 GiB, leaving 3 GiB to the process, while I suspect OS X is taking up 2 GiB. I don't have a Mac, so I don't really know. When I ran the the problem it took about 1.8 GiB, making it a close run thing if the Mac only gives 32 bit proccesses 2 GiB. Chuck ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?
What Charles pointed out was that while the inner product is very big, it seems to fit into memory on his 32-bit Linux machine; is it possible that OSX is preventing your python process from using even the meager 2-3 GB that a 32-bit process ought to get? Yes -- I think this is what is happening, because it's choking on calling up 1.6 GiB In particular, try running Charles' script in a fresh python interpreter and see if it works; it may be that other arrays you had allocated are taking up some of the space that this one could. I did try this a number of times. The result was by starting python freshly or deleting other arrays from memory, I am able to increase the size of the largest array I can compute the inner product on. However, even with nothing else in memory other than numpy and the original matrix whose inner product I'm taking ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?
Dan Yamins wrote: Hello folks, I did port Sage and hence Python with numpy and scipy to 64 bit OSX and below are some sample build instructions for just building python and numpy in 64 bit mode. Try In [3]: numpy.dtype(numpy.uintp).itemsize Out[3]: 4 which is the size in bytes of the integer needed to hold a pointer. The output above is for 32 bit python/numpy. Chuck Check, the answer is 4, as you got for the 32-bit. What would the answer be on a 64-bit architecture? Why is this diagnostic? Thanks! Dan First try: export OPT=-g -fwrapv -O3 -m64 -Wall -Wstrict-prototypes ./configure --disable-toolbox-glue --prefix=/Users/mabshoff/64bitnumpy/python-2.5.2-bin SNIP checking for int... yes checking size of int... 4 checking for long... yes checking size of long... 4 SNIP Oops, make fail because of the above. Let's try again: ./configure --disable-toolbox-glue --prefix=/Users/mabshoff/64bitnumpy/python-2.5.2-bin --with-gcc=gcc -m64 SNIP checking for int... yes checking size of int... 4 checking for long... yes checking size of long... 8 SNIP make make install then: bsd:python-2.5.2-bin mabshoff$ file bin/python bin/python: Mach-O 64-bit executable x86_64 Let's make the 64 bit python default: bsd:64bitnumpy mabshoff$ export PATH=/Users/mabshoff/64bitnumpy/python-2.5.2-bin/bin/:$PATH bsd:64bitnumpy mabshoff$ which python /Users/mabshoff/64bitnumpy/python-2.5.2-bin/bin//python bsd:64bitnumpy mabshoff$ file `which python` /Users/mabshoff/64bitnumpy/python-2.5.2-bin/bin//python: Mach-O 64-bit executable x86_64 Let's build numpy 1.1.0: bsd:64bitnumpy mabshoff$ tar xf numpy-1.1.0.tar.gz bsd:64bitnumpy mabshoff$ cd numpy-1.1.0 bsd:numpy-1.1.0 mabshoff$ python setup.py install SNIP bsd:python-2.5.2-bin mabshoff$ python Python 2.5.2 (r252:60911, Jun 4 2008, 20:47:16) [GCC 4.0.1 (Apple Inc. build 5465)] on darwin Type help, copyright, credits or license for more information. import numpy numpy.dtype(numpy.uintp).itemsize 8 ^D bsd:python-2.5.2-bin mabshoff$ Voila ;) Wi th numpy 1.0.4svn from 20080104 the numpy setup did not work since a conftest failed. I did report that to Stefan V. in IRC via #sage-devel and I also thought that this was still a problem with numpy 1.1.0, but fortuantely that was fixed. Now on to the test suite: numpy.test() Numpy is installed in /Users/mabshoff/64bitnumpy/python-2.5.2-bin/lib/python2.5/site-packages/numpy Numpy version 1.1.0 Python version 2.5.2 (r252:60911, Jun 4 2008, 20:47:16) [GCC 4.0.1 (Apple Inc. build 5465)] Found 18/18 tests for numpy.core.tests.test_defmatrix Found 3/3 tests for numpy.core.tests.test_errstate Found 3/3 tests for numpy.core.tests.test_memmap Found 286/286 tests for numpy.core.tests.test_multiarray Found 70/70 tests for numpy.core.tests.test_numeric Found 36/36 tests for numpy.core.tests.test_numerictypes Found 12/12 tests for numpy.core.tests.test_records Found 143/143 tests for numpy.core.tests.test_regression Found 7/7 tests for numpy.core.tests.test_scalarmath Found 2/2 tests for numpy.core.tests.test_ufunc Found 16/16 tests for numpy.core.tests.test_umath Found 63/63 tests for numpy.core.tests.test_unicode Found 4/4 tests for numpy.distutils.tests.test_fcompiler_gnu Found 5/5 tests for numpy.distutils.tests.test_misc_util Found 2/2 tests for numpy.fft.tests.test_fftpack Found 3/3 tests for numpy.fft.tests.test_helper Found 24/24 tests for numpy.lib.tests.test__datasource Found 10/10 tests for numpy.lib.tests.test_arraysetops Found 1/1 tests for numpy.lib.tests.test_financial Found 53/53 tests for numpy.lib.tests.test_function_base Found 5/5 tests for numpy.lib.tests.test_getlimits Found 6/6 tests for numpy.lib.tests.test_index_tricks Found 15/15 tests for numpy.lib.tests.test_io Found 1/1 tests for numpy.lib.tests.test_machar Found 4/4 tests for numpy.lib.tests.test_polynomial Found 1/1 tests for numpy.lib.tests.test_regression Found 49/49 tests for numpy.lib.tests.test_shape_base Found 15/15 tests for numpy.lib.tests.test_twodim_base Found 43/43 tests for numpy.lib.tests.test_type_check Found 1/1 tests for numpy.lib.tests.test_ufunclike Found 89/89 tests for numpy.linalg.tests.test_linalg Found 3/3 tests for numpy.linalg.tests.test_regression Found 94/94 tests for numpy.ma.tests.test_core Found 15/15 tests for numpy.ma.tests.test_extras Found 17/17 tests for numpy.ma.tests.test_mrecords Found 36/36 tests for numpy.ma.tests.test_old_ma Found 4/4 tests for numpy.ma.tests.test_subclassing Found 7/7 tests for numpy.tests.test_random Found 16/16 tests for numpy.testing.tests.test_utils Found 5/5 tests for numpy.tests.test_ctypeslib
Re: [Numpy-discussion] A memory problem: why does mmap come up in numpy.inner?
Dan Yamins wrote:
On Wed, Jun 4, 2008 at 9:06 PM, Charles R Harris
[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote:
Are both python and your version of OS X fully 64 bits?
I'm not sure.
From python:
python2.5 -c 'import platform;print platform.architecture()'
('32bit', 'ELF')
versus :
('64bit', 'ELF')
You can also try the unix file command (eg: from a terminal):
$ file `which python2.5`
/sware/exp/fable/standalone/redhate4-a64/bin/python: ELF 64-bit LSB
executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.4.0,
dynamically linked (uses shared libs), not stripped
...etc. We needed this for generating the .so library file name for
ctypes, and got the answer from comp.lang.python. I hope it also works
for OS X.
Best,
Jon
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
