morning all, Apologies for barging in on the discussion (this being my first post to this list); just saw one of my buzzwords and thought I might have something to contribute (and of course a chance for a shameless plug)... ;-)
On 2011-09-28 16:04, chm wrote: > On 9/28/2011 8:15 AM, David Mertens wrote: >> Hey Rob, >> >> I've CC'd the PDL list in case somebody there can speak more to your >> concerns about PDL. I've never heard of anybody needing 10s of >> thousands of >> dimensions, and PDL might only support 256. Anybody know? As far as >> sparse >> matrix support, I never used it and it's not a crowning feature. Can >> anybody >> else speak more to the matter? PDL's sparse matrix support is basically non-existent, afaik. There's a proof-of-concept implementation floating around somewhere in the core I looked at a few years ago, but iirc any kind of operation at all on a sparse piddle will blow it up into a dense one (or at least returns a dense pdl, which often amounts to the same thing as far as memory overflow is concerned). I also needed large sparse pdls in my work (very sparse 3d word co-occurrence matrices on the order of 50000**3), and wound up rolling my own bastard implementation out of PDL::PP and perl hackery; the module is available here: http://search.cpan.org/~moocow/PDL-CCS-1.14/CCS/Nd.pm The basic idea is to store a sparse N-dimensional pdl as a pair ($indices,$values), where $indices is a lexicographically sorted list of index-vectors (a la whichND(), indexND()), and $values are the corresponding (non-zero) values. Index operations are logarithmic time with vsearchvec(), and basic arithmetic operations and ufuncs can be twiddled with some block-wise processing heuristics. Abstract matrix multiplication is hairy, and when it comes down to it these things are perl objects and not "real" pdls in their own right, so a lot of pdl's functionality gets lost. It might be useful for the word-matrix application though... That having been said, I'd love to see some "real" sparse n-d matrix support in the pdl core; I'm just not sure to what degree the assumptions implicit in the core code might prevent an efficient sparse matrix implementation (e.g. explicit iteration over all possible values, implicit assumption of dense output pdls, etc.) marmosets, Bryan -- Bryan Jurish "There is *always* one more bug." [email protected] -Lubarsky's Law of Cybernetic Entomology _______________________________________________ Perldl mailing list [email protected] http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
