On Sun, Jun 20, 2010 at 11:47 PM, David Mertens <[email protected]>wrote:
> On Sun, Jun 20, 2010 at 11:32 PM, P Kishor <[email protected]> wrote: > >> On Sun, Jun 20, 2010 at 11:22 PM, David Mertens >> <[email protected]> wrote: >> > On Sun, Jun 20, 2010 at 3:44 PM, P Kishor <[email protected]> wrote: >> >> >> >> On Sun, Jun 20, 2010 at 2:55 PM, P Kishor <[email protected]> wrote: >> >> > I sent this out too soon... >> >> > >> >> > Actually, my computer becomes unresponsive even when reading the >> >> > stored piddle. Are there other IO suggestions that I could use, ones >> >> > that would allow me to work with very large piddles stored on the >> >> > disk? An iterator kinda like a file handle that would read in the >> >> > different parts of the piddle only when needed. >> >> >> >> >> >> Tried PDL::IO::FlexRaw.. wow! That is fast, close to instantaneous, >> >> compared to PDL::IO::Storable. >> > >> > Also, you should be able to memory-map the file if you're on a linux >> > machine, which will save you from reading the entire piddle into memory. >> > That should give you even faster reads and writes. However, there is a >> bug >> > in the latest PDL that breaks mmapping. >> >> Interesting. I am on a Mac, and mapflex seems to work fine for me. >> Although, I must say that I haven't done anything more than >> >> perldl> $a = mapflex('file.dat') >> >> and then queried a few values back out of $a >> >> I am assuming that if I can use mapflex, memory-mapping is working for >> me. And, yes, it seems faster than readflex, but I will have to do >> some more tests to determine convincingly. >> >> A novice's question -- is memory-mapping constrained by the amount of >> memory available? In other words, can I memory-map several large >> piddles, or will I run out of memory? Seems not, from your comment >> above that memory-map will "save you from reading the entire piddle >> into memory." However, that seems counter to the name of the technique >> "memory-map" >> >> > I think I know what's going on but I >> > haven't had a chance to fix it yet. It may be a month or so before I get >> to >> > it, so let me know if you need it. >> > >> > David >> > >> > -- >> > Sent via my carrier pigeon. >> > >> > > Here's wikipedia's article on memory mapping: > http://en.wikipedia.org/wiki/Memory-mapped_file > > Very insightful. I thought it was useful mostly for the "lazy loading" > referred to a few paragraphs from the top, but clearly it has other uses I > did not know until you asked. :-) > > > David > > -- > Sent via my carrier pigeon. > For anybody with some round tuits and some mad XS skills, you might want to look into the internals of File::Map on CPAN for cross-platform memory mapping ideas. Presently, memory mapping only works on unix-like machines. David -- Sent via my carrier pigeon.
_______________________________________________ Perldl mailing list [email protected] http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
