Maggie - For sparse matrix support, see http://p3rl.org/PDL::CCS::Nd. This was one of Bryan Jurish's many modules posted late last year/early this year. I have not examined the internals to see how it works or what sort of hit on speed one has to take to use the library, but it's at least available as a starting point.
As for cross-machine parallelization, the only solution of which I am aware is http://p3rl.org/PDL::Parallel::MPI. That requires an MPI setup, though, and I had to tweak the Makefile.PL in order to get it to run on UIUC's local cluster. But the underlying code worked. As a matter of fact, I tried to contact Darin a few years ago to see if I could take over maintenance of the module (because the changes to Makefile.PL are pretty trivial), but he never got back to me. I wonder if I should follow up on that... David On Sun, Jul 1, 2012 at 8:18 PM, Maggie X <[email protected]> wrote: > For "big data" we need sparse matrix and maybe even more importantly, the > ability to parallelize across machines. PDL is great if the data can fit in > memory on a single machine. For big data, the assumption should be that the > data will NOT fit in memory and multiple machines are necessary to finish > processing in a reasonable amount of time. I would love to see more work in > that direction, if possible. > > > Best, > Maggie > -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." -- Brian Kernighan
_______________________________________________ Perldl mailing list [email protected] http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
