If your matrix is not necessarily sparse, you will have to process it all
through memory. PDL is optimized for problems that fit in your machine's RAM
limit. 15000x15000 floats is 900 MB, which should fit within most machines.
(15000x15000 double-precision values is 1.8 GB, which should also be OK).
You'll need to set the global variable $PDL::BIGPDL to 1 to let Perl know you
plan to work with arrays that large.
My laptop computer has 16GB of RAM. This works fine:
use PDL;
$a = random(15000,15000); # generate 15000x15000 array of random numbers
$b = random(15000,15000); # generate another one
If you're running out of memory you may be trying to do something silly like
read all the numbers in as Perl scalars...?
On the other hand, this may take a while:
$c = $a x $b; # brute-force matrix multiply -- ~200 hours to complete
The reason is that the final step requires (8 * 15000 * 3 * 15000 * 15000)
memory
accesses.
Finding eigenvalues of a 15000x15000 matrix is a nontrivial process. PDL has an
eigenvalue solver ("eigens") but it is a general purpose tool for small
matrices, it would take considerably longer than the age of the Universe to
find the eigenvalues of a 15000x15000 nonsparse matrix -- so your project might
be a little late if you use that.
Working with large matrices is its own computational subject. PDL makes a nice
framework for it, but for any serious operations you can't just use the kind of
general purpose tools that work fine on (say) a 10x10 matrix.
On Sep 5, 2014, at 9:12 AM, Ronak Agrawal <ronagra...@gmail.com> wrote:
> Thank You Sir for the early response.
>
> I am new to Perl and have been assigned project on Topic Modeling where I
> have to search, browse and find information from large archives of texts.
>
> Matrix operation is one of the operation and as per requirement my matrix may
> be sparse or dense. Is it possible for you help me with both the cases.
>
> More to that can you tell me some good methods to handle large data in Perl.
>
> Once again thank you for the response
>
>
> On Fri, Sep 5, 2014 at 7:36 PM, Craig DeForest <defor...@boulder.swri.edu>
> wrote:
> Glad to help. First, a few questions. Is the matrix sparse? (i.e. are less
> than, say 10^-3 of the elements nonzero?) How close to tridiagonal is it?
>
>
> On Sep 5, 2014, at 6:27 AM, Ronak Agrawal <ronagra...@gmail.com> wrote:
>
>> Hi
>> I am doing a project in Topic Modelling which involves large matrix
>> operations.
>> I have a sql database from where I have to generate 15000 x 15000 matix -
>> transform and obtain A'A.Later I have to find Eigen Values and Eigen Vectors.
>> Can you suggest me ways to do this in Perl.I get "Out of Memory" while
>> storing the matrix in memory.
>> Your input will help in handling big data and therby making my project
>> success
>> Thank You
>> Ronak
>> _______________________________________________
>> Perldl mailing list
>> Perldl@jach.hawaii.edu
>> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
>
>
_______________________________________________
Perldl mailing list
Perldl@jach.hawaii.edu
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
