Cool! Maybe there are more cache hits than I expected (which was none)... (Mobile)
> On Sep 6, 2014, at 1:54 PM, Chris Marshall <[email protected]> wrote: > > On my PC (2.8GHZ i7) it takes about an hour for the multiply > just using $a x $b as Craig shows. I haven't tried using the > autothreading support to see how that changes things. > > As discussed already, GPU acceleration could allow for much > faster computation. For a start Nvidia has a cuBLAS library > which implements matrix multiply which could be used to > optimize the performance. > > --Chris > > > On Fri, Sep 5, 2014 at 11:49 AM, Craig DeForest > <[email protected]> wrote: >> If your matrix is not necessarily sparse, you will have to process it all >> through memory. PDL is optimized for problems that fit in your machine's >> RAM limit. 15000x15000 floats is 900 MB, which should fit within most >> machines. (15000x15000 double-precision values is 1.8 GB, which should also >> be OK). You'll need to set the global variable $PDL::BIGPDL to 1 to let >> Perl know you plan to work with arrays that large. >> >> My laptop computer has 16GB of RAM. This works fine: >> >> use PDL; >> $a = random(15000,15000); # generate 15000x15000 array of random numbers >> $b = random(15000,15000); # generate another one >> >> If you're running out of memory you may be trying to do something silly like >> read all the numbers in as Perl scalars...? >> >> On the other hand, this may take a while: >> >> $c = $a x $b; # brute-force matrix multiply -- ~200 hours to complete >> >> The reason is that the final step requires (8 * 15000 * 3 * 15000 * 15000) >> memory >> accesses. >> >> Finding eigenvalues of a 15000x15000 matrix is a nontrivial process. PDL has >> an eigenvalue solver ("eigens") but it is a general purpose tool for small >> matrices, it would take considerably longer than the age of the Universe to >> find the eigenvalues of a 15000x15000 nonsparse matrix -- so your project >> might be a little late if you use that. >> >> Working with large matrices is its own computational subject. PDL makes a >> nice framework for it, but for any serious operations you can't just use the >> kind of general purpose tools that work fine on (say) a 10x10 matrix. >> >> >> >> On Sep 5, 2014, at 9:12 AM, Ronak Agrawal <[email protected]> wrote: >> >> Thank You Sir for the early response. >> >> I am new to Perl and have been assigned project on Topic Modeling where I >> have to search, browse and find information from large archives of texts. >> >> Matrix operation is one of the operation and as per requirement my matrix >> may be sparse or dense. Is it possible for you help me with both the cases. >> >> More to that can you tell me some good methods to handle large data in Perl. >> >> Once again thank you for the response >> >> >> On Fri, Sep 5, 2014 at 7:36 PM, Craig DeForest <[email protected]> >> wrote: >>> >>> Glad to help. First, a few questions. Is the matrix sparse? (i.e. are >>> less than, say 10^-3 of the elements nonzero?) How close to tridiagonal is >>> it? >>> >>> >>> On Sep 5, 2014, at 6:27 AM, Ronak Agrawal <[email protected]> wrote: >>> >>> Hi >>> >>> I am doing a project in Topic Modelling which involves large matrix >>> operations. >>> >>> I have a sql database from where I have to generate 15000 x 15000 matix - >>> transform and obtain A'A.Later I have to find Eigen Values and Eigen >>> Vectors. >>> >>> Can you suggest me ways to do this in Perl.I get "Out of Memory" while >>> storing the matrix in memory. >>> >>> Your input will help in handling big data and therby making my project >>> success >>> >>> Thank You >>> >>> Ronak >>> >>> _______________________________________________ >>> Perldl mailing list >>> [email protected] >>> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl >> >> >> >> _______________________________________________ >> Perldl mailing list >> [email protected] >> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl > _______________________________________________ Perldl mailing list [email protected] http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
