You want to allocate the matrix all at once, then populate it. The ->glue call
makes a copy of
the matrix, so by the time you're up to 4000x10000, you're copying between 40MB
and 320MB with each glue call. That can eat up some time.
Far better do do something like:
$PDL::BIGPDL = 1;
$matrix = PDL::new_from_specification(byte,4000,50000); # use the smallest
type you can!
print "Loading rows";
for $row(0..$maxrow) {
$matrix->(:,($row)) .= get_a_row();
print "." unless($row%500);
}
On Oct 29, 2012, at 10:21 PM, Nan Yu <[email protected]> wrote:
> Hi,
> I’m working with a large 2D matrix: about 4000 x 50000.
> I used to use perl 2D array to do the job (when working with
> smaller dimensions) but the memory became an issue when the dimension grows
> into this size. So I decided to use PDL.
> After I finished my code and started testing it, I found out that just to
> initiate the 2D matrix takes a long time.
> I start with an 1D array of 4000 elements and then use “$matrix =
> $matrix->glue(1,$oneDarray)” within a loop (from 1 to 50000) to create the 2D
> matrix.
> The first 5000-10000 were reasonable quick (within 10 mins) then the speed
> slows down a lot.
> I’m wondering if I used a wrong way to do the initiate of the
> matrix.
> Could someone please tell me how to work with the large 2D
> matrix?
> Thanks!
> Nan Yu
>
>
> Please note that my email address has changed to [email protected]
>
> *** The information contained in this communication may be confidential, is
> intended only for the use of the recipient(s) named above, and may be legally
> privileged. If the reader of this message is not the intended recipient, you
> are hereby notified that any dissemination, distribution, or copying of this
> communication, or any of its contents, is strictly prohibited. If you have
> received this communication in error, please return it to the sender
> immediately and delete the original message and any copies of it. If you have
> any questions concerning this message, please contact the sender. ***
> _______________________________________________
> Perldl mailing list
> [email protected]
> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
_______________________________________________
Perldl mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
