On Tue, Oct 30, 2012 at 10:31 PM, Nan Yu <[email protected]> wrote: > Hi, Chris: > Thanks for your reply. > I'm using PDL v2.4.11 on a 64bit Scientific Linux machine > (Linux version 2.6.32-220.4.1.el6.x86_64). The PDL should be the most recent > version. > > 2**31=2,147,483,648 and my matrix has about 4000 * 50000 = > 200,000,000 elements, > storing number values, with absolute value less than 1000 in general and with > 2 digit accuracy > (such as 123.45). So, I should be fine if I only do some simple > manipulations, such as > sum/subtraction etc., right?
200,000,000 < 2,147,483,648 so you have the full power of PDL at your disposal. The comment about limiting the size of slices for operation are only relevant if your piddles have more than 2**31 elements. Happy PDL-ing! Chris > Thanks! > Nan > > -----Original Message----- > From: Chris Marshall [mailto:[email protected]] > Sent: Tuesday, October 30, 2012 12:44 PM > To: Nan Yu > Cc: [email protected] > Subject: Re: [Perldl] how to work with large 2D matrix > > Hi Nan- > > I see Craig answered your specific question. A couple of thoughts. > > You don't say what version of PDL you are using (or your > platform) but the current release has support for memory mapped IO (see docs > for PDL::IO::FlexRaw). > > There is also support for allocating piddles (the PDL data objects) having > more > than 2**31 elements in them. Some of the internal indexing in PDL is in 32bit > signed integers which means some operations might fail due to counter overflow > or wrap-around. > > If you stick to single dimensions less than 2**31 elements and work with > slices > smaller than that size it is believed to allow one to work around the > limitations. > I would advise you to stay below that size in your work until you have > experience > with the processing flow. > > After that, we're always looking for help with debugging large-piddle > support! :-) > > --Chris > > On Tue, Oct 30, 2012 at 12:21 AM, Nan Yu <[email protected]> wrote: >> >> I'm working with a large 2D matrix: about 4000 x 50000. >> >> I used to use perl 2D array to do the job (when working with smaller >> dimensions) but the memory became an issue when the dimension grows >> into this size. So I decided to use PDL. >> >> After I finished my code and started testing it, I found out that just >> to initiate the 2D matrix takes a long time. >> >> I start with an 1D array of 4000 elements and then use "$matrix = >> $matrix->glue(1,$oneDarray)" within a loop (from 1 to 50000) to create >> the 2D matrix. The first 5000-10000 were reasonable quick (within 10 >> mins) then the speed slows down a lot. >> >> I'm wondering if I used a wrong way to do the initiate of the matrix. >> >> Could someone please tell me how to work with the large 2D matrix? >> >> Thanks! >> Nan Yu > > Please note that my email address has changed to [email protected] > > ________________________________ > *** The information contained in this communication may be confidential, is > intended only for the use of the recipient(s) named above, and may be legally > privileged. If the reader of this message is not the intended recipient, you > are hereby notified that any dissemination, distribution, or copying of this > communication, or any of its contents, is strictly prohibited. If you have > received this communication in error, please return it to the sender > immediately and delete the original message and any copies of it. If you have > any questions concerning this message, please contact the sender. *** > > _______________________________________________ > Perldl mailing list > [email protected] > http://mailman.jach.hawaii.edu/mailman/listinfo/perldl _______________________________________________ Perldl mailing list [email protected] http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
