Hi Nan- I see Craig answered your specific question. A couple of thoughts.
You don't say what version of PDL you are using (or your platform) but the current release has support for memory mapped IO (see docs for PDL::IO::FlexRaw). There is also support for allocating piddles (the PDL data objects) having more than 2**31 elements in them. Some of the internal indexing in PDL is in 32bit signed integers which means some operations might fail due to counter overflow or wrap-around. If you stick to single dimensions less than 2**31 elements and work with slices smaller than than size it is believed to allow one to work around the limitations. I would advised you to stay below that size in your work until you have experience with the processing flow. After that, we're always looking for help with debugging large-piddle support! :-) --Chris On Tue, Oct 30, 2012 at 12:21 AM, Nan Yu <[email protected]> wrote: > > I’m working with a large 2D matrix: about 4000 x 50000. > > I used to use perl 2D array to do the job (when working with > smaller dimensions) but the memory became an issue when the dimension grows > into this size. So I decided to use PDL. > > After I finished my code and started testing it, I found out that just to > initiate the 2D matrix takes a long time. > > I start with an 1D array of 4000 elements and then use “$matrix = > $matrix->glue(1,$oneDarray)” within a loop (from 1 to 50000) to create the > 2D matrix. The first 5000-10000 were reasonable quick (within 10 mins) > then the speed slows down a lot. > > I’m wondering if I used a wrong way to do the initiate of > the matrix. > > Could someone please tell me how to work with the large 2D > matrix? > > Thanks! > Nan Yu _______________________________________________ Perldl mailing list [email protected] http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
