On Sun, Jun 20, 2010 at 11:22 PM, David Mertens
<[email protected]> wrote:
> On Sun, Jun 20, 2010 at 3:44 PM, P Kishor <[email protected]> wrote:
>>
>> On Sun, Jun 20, 2010 at 2:55 PM, P Kishor <[email protected]> wrote:
>> > I sent this out too soon...
>> >
>> > Actually, my computer becomes unresponsive even when reading the
>> > stored piddle. Are there other IO suggestions that I could use, ones
>> > that would allow me to work with very large piddles stored on the
>> > disk? An iterator kinda like a file handle that would read in the
>> > different parts of the piddle only when needed.
>>
>>
>> Tried PDL::IO::FlexRaw.. wow! That is fast, close to instantaneous,
>> compared to PDL::IO::Storable.
>
> Also, you should be able to memory-map the file if you're on a linux
> machine, which will save you from reading the entire piddle into memory.
> That should give you even faster reads and writes. However, there is a bug
> in the latest PDL that breaks mmapping.
Interesting. I am on a Mac, and mapflex seems to work fine for me.
Although, I must say that I haven't done anything more than
perldl> $a = mapflex('file.dat')
and then queried a few values back out of $a
I am assuming that if I can use mapflex, memory-mapping is working for
me. And, yes, it seems faster than readflex, but I will have to do
some more tests to determine convincingly.
A novice's question -- is memory-mapping constrained by the amount of
memory available? In other words, can I memory-map several large
piddles, or will I run out of memory? Seems not, from your comment
above that memory-map will "save you from reading the entire piddle
into memory." However, that seems counter to the name of the technique
"memory-map"
> I think I know what's going on but I
> haven't had a chance to fix it yet. It may be a month or so before I get to
> it, so let me know if you need it.
>
> David
>
> --
> Sent via my carrier pigeon.
>
--
Puneet Kishor http://www.punkish.org
Carbon Model http://carbonmodel.org
Charter Member, Open Source Geospatial Foundation http://www.osgeo.org
Science Commons Fellow, http://sciencecommons.org/about/whoweare/kishor
Nelson Institute, UW-Madison http://www.nelson.wisc.edu
-----------------------------------------------------------------------
Assertions are politics; backing up assertions with evidence is science
=======================================================================
_______________________________________________
Perldl mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
