On Feb 19, 12:26 pm, Carl Banks <pavlovevide...@gmail.com> wrote: > On Feb 19, 10:36 am, Lionel <lionel.ke...@gmail.com> wrote: > > > > > > > On Feb 19, 9:51 am, Carl Banks <pavlovevide...@gmail.com> wrote: > > > > On Feb 19, 9:34 am, Lionel <lionel.ke...@gmail.com> wrote: > > > > > On Feb 18, 12:35 pm, Carl Banks <pavlovevide...@gmail.com> wrote: > > > > > > On Feb 18, 10:48 am, Lionel <lionel.ke...@gmail.com> wrote: > > > > > > > Thanks Carl, I like your solution. Am I correct in my understanding > > > > > > that memory is allocated at the slicing step in your example i.e. > > > > > > when > > > > > > "reshaped_data" is sliced using "interesting_data = reshaped_data[:, > > > > > > 50:100]"? In other words, given a huge (say 1Gb) file, a memmap > > > > > > object > > > > > > is constructed that memmaps the entire file. Some relatively small > > > > > > amount of memory is allocated for the memmap operation, but the bulk > > > > > > memory allocation occurs when I generate my final numpy sub-array by > > > > > > slicing, and this accounts for the memory efficiency of using > > > > > > memmap? > > > > > > No, what accounts for the memory efficienty is there is no bulk > > > > > allocation at all. The ndarray you have points to the memory that's > > > > > in the mmap. There is no copying data or separate array allocation. > > > > > Does this mean that everytime I iterate through an ndarray that is > > > > sourced from a memmap, the data is read from the disc? The sliced > > > > array is at no time wholly resident in memory? What are the > > > > performance implications of this? > > > > Ok, sorry for the confusion. What I should have said is that there is > > > no bulk allocation *by numpy* at all. The call to mmap does allocate > > > a chunk of RAM to reflect file contents, but the numpy arrays don't > > > allocate any memory of their own: they use the same memory as was > > > allocated by the mmap call. > > > > Carl Banks- Hide quoted text - > > > > - Show quoted text - > > > Thanks for the explanations Carl. I'm sorry, but it's me who's the > > confused one here, not anyone else :-) > > > I hate to waste everyone's time again, but something is just not > > "clicking" in that black-hole I call a brain. So..."numpy.memmap" > > allocates a chunk off the heap to coincide with the file contents. If > > I memmap the entire 1 Gb file, a corresponding amount (approx. 1 Gb) > > is allocated? That seems to contradict what is stated in the numpy > > documentation: > > > "class numpy.memmap > > Create a memory-map to an array stored in a file on disk. > > > Memory-mapped files are used for accessing small segments of large > > files on disk, without reading the entire file into memory." > > Yes, it allocates room for the whole file in your process's LOGICAL > address space. However, it doesn't actually reserve any PHYSICAL > memory, or read in any data from the disk, until you've actually > access the data. And then it only reads small chunks in, not the > whole file. > > So when you mmap your 1GB file, the OS sets aside a 1 GB chunk of > address to use for your memory map. That's all it does: it doesn't > read anything from disk, it doesn't reserve any physical RAM. Later, > when you access a byte in the mmap via a pointer, the OS notes that it > hasn't yet loaded the data at that address, so it grabs a small chunk > of physical ram and reads in the a small amount of data from the disk > containing the byte you are accessing. This all happens automatically > and transparently to you. > > > In my previous example that we were working with (100x100 data file), > > you used an offset to memmap the "lower-half" of the array. Does this > > mean that in the process of memmapping that lower half, RAM was set > > aside for 50x100 32-bit complex numbers? If so, and I decide to memmap > > an entire file, there is no memory benefit in doing so. > > The mmap call sets aside room for all 100x100 32-bit complex numbers > in logical address space, regardless of whether you use the offset > parameter or not. However, it might only read in part of the file in > from disk, and will only reserve physical RAM for the parts it reads > in. > > > At this point do you (or anyone else) recommend I just write a little > > function for my class that takes the coords I intend to load and "roll > > my own" function? Seems like the best way to keep memory to a minimum, > > I'm just worried about performance. On the other hand, the most I'd be > > loading would be around 1k x 1k worth of data.- > > No, if your file is not too large to mmap, just do it the way you've > been doing it. The documentation you've been reading is pretty much > correct, even if you approach it naively. It is both memory and I/O > efficient. You're overthinking things here; don't try to outsmart the > operating system. It'll take care of the performance issues > satisfactorily. > > The only thing you have to worry about is if the file is too large to > fit into your process's logical address space, which on a typical 32- > bit system is 2-3 GB (depending on configuration) minus the space > occupied by Python and other heap objects, which is probably only a > few MB. > > Carl Banks- Hide quoted text - > > - Show quoted text -
I see. That was very well explained Carl, thank you. -- http://mail.python.org/mailman/listinfo/python-list