Hi, Craig:
Thanks for helping me. With PDL::BIGPDL, things seems to be
working.
When I use $matrix = zeros(4000,50000), I will get the matrix
with all 0 values.
If I would like to make it all "-99" as default (instead of 0), is there are a
quick way for me to do that?
Thanks!
Nan
From: Craig DeForest [mailto:[email protected]]
Sent: Monday, October 29, 2012 11:38 PM
To: Nan Yu
Cc: Craig DeForest; [email protected]
Subject: Re: [Perldl] how to work with large 2D matrix
You want to allocate the matrix all at once, then populate it. The ->glue call
makes a copy of
the matrix, so by the time you're up to 4000x10000, you're copying between 40MB
and 320MB with each glue call. That can eat up some time.
Far better do do something like:
$PDL::BIGPDL = 1;
$matrix = PDL::new_from_specification(byte,4000,50000); # use the smallest
type you can!
print "Loading rows";
for $row(0..$maxrow) {
$matrix->(:,($row)) .= get_a_row();
print "." unless($row%500);
}
On Oct 29, 2012, at 10:21 PM, Nan Yu
<[email protected]<mailto:[email protected]>> wrote:
Hi,
I'm working with a large 2D matrix: about 4000 x 50000.
I used to use perl 2D array to do the job (when working with
smaller dimensions) but the memory became an issue when the dimension grows
into this size. So I decided to use PDL.
After I finished my code and started testing it, I found out that just to
initiate the 2D matrix takes a long time.
I start with an 1D array of 4000 elements and then use "$matrix =
$matrix->glue(1,$oneDarray)" within a loop (from 1 to 50000) to create the 2D
matrix.
The first 5000-10000 were reasonable quick (within 10 mins) then the speed
slows down a lot.
I'm wondering if I used a wrong way to do the initiate of the
matrix.
Could someone please tell me how to work with the large 2D
matrix?
Thanks!
Nan Yu
Please note that my email address has changed to
[email protected]<mailto:[email protected]>
________________________________
*** The information contained in this communication may be confidential, is
intended only for the use of the recipient(s) named above, and may be legally
privileged. If the reader of this message is not the intended recipient, you
are hereby notified that any dissemination, distribution, or copying of this
communication, or any of its contents, is strictly prohibited. If you have
received this communication in error, please return it to the sender
immediately and delete the original message and any copies of it. If you have
any questions concerning this message, please contact the sender. ***
_______________________________________________
Perldl mailing list
[email protected]<mailto:[email protected]>
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
Please note that my email address has changed to [email protected]
________________________________
*** The information contained in this communication may be confidential, is
intended only for the use of the recipient(s) named above, and may be legally
privileged. If the reader of this message is not the intended recipient, you
are hereby notified that any dissemination, distribution, or copying of this
communication, or any of its contents, is strictly prohibited. If you have
received this communication in error, please return it to the sender
immediately and delete the original message and any copies of it. If you have
any questions concerning this message, please contact the sender. ***
_______________________________________________
Perldl mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
