On my PC (2.8GHZ i7) it takes about an hour for the multiply
just using $a x $b as Craig shows. I haven't tried using the
autothreading support to see how that changes things.
As discussed already, GPU acceleration could allow for much
faster computation. For a start Nvidia has a cuBLAS library
which implements matrix multiply which could be used to
optimize the performance.
--Chris
On Fri, Sep 5, 2014 at 11:49 AM, Craig DeForest
<defor...@boulder.swri.edu> wrote:
> If your matrix is not necessarily sparse, you will have to process it all
> through memory. PDL is optimized for problems that fit in your machine's
> RAM limit. 15000x15000 floats is 900 MB, which should fit within most
> machines. (15000x15000 double-precision values is 1.8 GB, which should also
> be OK). You'll need to set the global variable $PDL::BIGPDL to 1 to let
> Perl know you plan to work with arrays that large.
>
> My laptop computer has 16GB of RAM. This works fine:
>
> use PDL;
> $a = random(15000,15000); # generate 15000x15000 array of random numbers
> $b = random(15000,15000); # generate another one
>
> If you're running out of memory you may be trying to do something silly like
> read all the numbers in as Perl scalars...?
>
> On the other hand, this may take a while:
>
> $c = $a x $b; # brute-force matrix multiply -- ~200 hours to complete
>
> The reason is that the final step requires (8 * 15000 * 3 * 15000 * 15000)
> memory
> accesses.
>
> Finding eigenvalues of a 15000x15000 matrix is a nontrivial process. PDL has
> an eigenvalue solver ("eigens") but it is a general purpose tool for small
> matrices, it would take considerably longer than the age of the Universe to
> find the eigenvalues of a 15000x15000 nonsparse matrix -- so your project
> might be a little late if you use that.
>
> Working with large matrices is its own computational subject. PDL makes a
> nice framework for it, but for any serious operations you can't just use the
> kind of general purpose tools that work fine on (say) a 10x10 matrix.
>
>
>
> On Sep 5, 2014, at 9:12 AM, Ronak Agrawal <ronagra...@gmail.com> wrote:
>
> Thank You Sir for the early response.
>
> I am new to Perl and have been assigned project on Topic Modeling where I
> have to search, browse and find information from large archives of texts.
>
> Matrix operation is one of the operation and as per requirement my matrix
> may be sparse or dense. Is it possible for you help me with both the cases.
>
> More to that can you tell me some good methods to handle large data in Perl.
>
> Once again thank you for the response
>
>
> On Fri, Sep 5, 2014 at 7:36 PM, Craig DeForest <defor...@boulder.swri.edu>
> wrote:
>>
>> Glad to help. First, a few questions. Is the matrix sparse? (i.e. are
>> less than, say 10^-3 of the elements nonzero?) How close to tridiagonal is
>> it?
>>
>>
>> On Sep 5, 2014, at 6:27 AM, Ronak Agrawal <ronagra...@gmail.com> wrote:
>>
>> Hi
>>
>> I am doing a project in Topic Modelling which involves large matrix
>> operations.
>>
>> I have a sql database from where I have to generate 15000 x 15000 matix -
>> transform and obtain A'A.Later I have to find Eigen Values and Eigen
>> Vectors.
>>
>> Can you suggest me ways to do this in Perl.I get "Out of Memory" while
>> storing the matrix in memory.
>>
>> Your input will help in handling big data and therby making my project
>> success
>>
>> Thank You
>>
>> Ronak
>>
>> _______________________________________________
>> Perldl mailing list
>> Perldl@jach.hawaii.edu
>> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
>>
>>
>
>
>
> _______________________________________________
> Perldl mailing list
> Perldl@jach.hawaii.edu
> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
>
_______________________________________________
Perldl mailing list
Perldl@jach.hawaii.edu
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
