Edward, Here is what I would like to benchmark
M.v / (||M||.||v||) where M is an m.n matrix being 80% sparse, same for the vector where m =10^6 and n = 5.10^5 M and v can be initialized as random sparse matrix/vector. Do you think this can be done using hama as it is now ? Thanks On Thu, Jun 4, 2009 at 10:24 AM, Edward J. Yoon <[email protected]>wrote: > Yes, the goal is to handle really huge matrices, for example, matrix > operations for large-scale statistical processing, matrix > decomposition of huge web link graph/social graph. > > It's the tests on 5 nodes and 10 nodes. In the future, I'll try them > on a thousand nodes. > > On Thu, Jun 4, 2009 at 1:19 AM, tog <[email protected]> wrote: > > Hi Edward, > > > > I had a look to the benchmarks ... > > Well a 5000 dense matrix multiply is roughly 30 seconds on my laptop. I > have > > been doing out-of-core parallel matrix factor on solve with dense systems > up > > to 350000 > > so I guess this is at least probably for larger matrix that Hama could be > > interesting > > Do you plan to do such tests with really huge matrices ? > > Otherwise what is your business case ? > > > > Cheers > > Guillaume > > > > On Wed, Jun 3, 2009 at 6:14 PM, Edward J. Yoon <[email protected] > >wrote: > > > >> FYI, I ran some benchmarks - > >> http://wiki.apache.org/hama/PerformanceEvaluation > >> > >> If you need any help, Pls let us know. > >> > >> Thanks. > >> > >> On Wed, Jun 3, 2009 at 6:55 PM, tog <[email protected]> wrote: > >> > Yes I understand the difference between MPI and Hadoop - I have been > >> using > >> > MPI before it actually exists :) > >> > But as you phrased it, I had the impression that Hama was working on a > 1 > >> > node/core cluster !! > >> > > >> > Regards > >> > Guillaume > >> > > >> > On Wed, Jun 3, 2009 at 5:44 PM, Edward J. Yoon <[email protected] > >> >wrote: > >> > > >> >> Hi, > >> >> > >> >> There is some difference between Map/Reduce and MPI programming. MPI > >> >> is based on and designed for fast parallel computing using network > >> >> communication on small cluster. Since MPI requires network > >> >> communication, Increased node numbers, there is a linear increase of > >> >> network cost at same time. On the contrary, Map/Reduce is designed to > >> >> distributed processing by connecting many commodity computers > >> >> together. Therefore, The algorithms should avoid large amounts of > >> >> communication for best performance and that key is the 'sequential > >> >> process'. > >> >> > >> >> Thanks. > >> >> > >> >> On Wed, Jun 3, 2009 at 6:07 PM, tog <[email protected]> > wrote: > >> >> > Hi Edward > >> >> > > >> >> > I have a test to do which is basically Sparce Mat Vec > multiplication > >> and > >> >> Mat > >> >> > norm computation. So that should be possible with Hama in its > current > >> >> state > >> >> > I guess. > >> >> > What do you mean by "sequentially executed" > >> >> > > >> >> > Cheers > >> >> > Guillaume > >> >> > > >> >> > On Wed, Jun 3, 2009 at 5:00 PM, Edward J. Yoon < > [email protected] > >> >> >wrote: > >> >> > > >> >> >> Hi, > >> >> >> > >> >> >> Currently, the basic matrix operations are implemented based on > the > >> >> >> map/reduce programming model. For example, the matrix get/set > >> methods, > >> >> >> the matrix norms, matrix-matrix multiplication/addition, matrix > >> >> >> transpose. In near future, SVD, Eigenvalue decomposition and some > >> >> >> graph algorithms will be implemented. All the operations are > >> >> >> sequentially executed. > >> >> >> > >> >> >> Thanks. > >> >> >> > >> >> >> On Wed, Jun 3, 2009 at 5:45 PM, tog <[email protected]> > >> wrote: > >> >> >> > > >> >> >> > Hi, > >> >> >> > > >> >> >> > I would like to know what is the status of Hama ? > >> >> >> > What am I able to do with it ? > >> >> >> > What are the future directions ? > >> >> >> > > >> >> >> > Cheers > >> >> >> > Guillaume > >> >> >> > >> >> >> > >> >> >> > >> >> >> -- > >> >> >> Best Regards, Edward J. Yoon @ NHN, corp. > >> >> >> [email protected] > >> >> >> http://blog.udanax.org > >> >> >> > >> >> > > >> >> > > >> >> > > >> >> > -- > >> >> > > >> >> > PGP KeyID: 1024D/47172155 > >> >> > FingerPrint: C739 8B3C 5ABF 127F CCFA 5835 F673 370B 4717 2155 > >> >> > > >> >> > http://cheztog.blogspot.com > >> >> > > >> >> > >> >> > >> >> > >> >> -- > >> >> Best Regards, Edward J. Yoon @ NHN, corp. > >> >> [email protected] > >> >> http://blog.udanax.org > >> >> > >> > > >> > > >> > > >> > -- > >> > > >> > PGP KeyID: 1024D/47172155 > >> > FingerPrint: C739 8B3C 5ABF 127F CCFA 5835 F673 370B 4717 2155 > >> > > >> > http://cheztog.blogspot.com > >> > > >> > >> > >> > >> -- > >> Best Regards, Edward J. Yoon @ NHN, corp. > >> [email protected] > >> http://blog.udanax.org > >> > > > > > > > > -- > > > > PGP KeyID: 1024D/47172155 > > FingerPrint: C739 8B3C 5ABF 127F CCFA 5835 F673 370B 4717 2155 > > > > http://cheztog.blogspot.com > > > > > > -- > Best Regards, Edward J. Yoon @ NHN, corp. > [email protected] > http://blog.udanax.org > -- PGP KeyID: 1024D/47172155 FingerPrint: C739 8B3C 5ABF 127F CCFA 5835 F673 370B 4717 2155 http://cheztog.blogspot.com
