On Sun, Jun 20, 2010 at 11:32 PM, P Kishor <[email protected]> wrote:
> On Sun, Jun 20, 2010 at 11:22 PM, David Mertens > <[email protected]> wrote: > > On Sun, Jun 20, 2010 at 3:44 PM, P Kishor <[email protected]> wrote: > >> > >> On Sun, Jun 20, 2010 at 2:55 PM, P Kishor <[email protected]> wrote: > >> > I sent this out too soon... > >> > > >> > Actually, my computer becomes unresponsive even when reading the > >> > stored piddle. Are there other IO suggestions that I could use, ones > >> > that would allow me to work with very large piddles stored on the > >> > disk? An iterator kinda like a file handle that would read in the > >> > different parts of the piddle only when needed. > >> > >> > >> Tried PDL::IO::FlexRaw.. wow! That is fast, close to instantaneous, > >> compared to PDL::IO::Storable. > > > > Also, you should be able to memory-map the file if you're on a linux > > machine, which will save you from reading the entire piddle into memory. > > That should give you even faster reads and writes. However, there is a > bug > > in the latest PDL that breaks mmapping. > > Interesting. I am on a Mac, and mapflex seems to work fine for me. > Although, I must say that I haven't done anything more than > > perldl> $a = mapflex('file.dat') > > and then queried a few values back out of $a > > I am assuming that if I can use mapflex, memory-mapping is working for > me. And, yes, it seems faster than readflex, but I will have to do > some more tests to determine convincingly. > > A novice's question -- is memory-mapping constrained by the amount of > memory available? In other words, can I memory-map several large > piddles, or will I run out of memory? Seems not, from your comment > above that memory-map will "save you from reading the entire piddle > into memory." However, that seems counter to the name of the technique > "memory-map" > > > I think I know what's going on but I > > haven't had a chance to fix it yet. It may be a month or so before I get > to > > it, so let me know if you need it. > > > > David > > > > -- > > Sent via my carrier pigeon. > > > Here's wikipedia's article on memory mapping: http://en.wikipedia.org/wiki/Memory-mapped_file Very insightful. I thought it was useful mostly for the "lazy loading" referred to a few paragraphs from the top, but clearly it has other uses I did not know until you asked. :-) David -- Sent via my carrier pigeon.
_______________________________________________ Perldl mailing list [email protected] http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
