If you're have very large and very sparse matrix represented as (i, j, value) entries, then you can try the algorithms mentioned in the post <https://groups.google.com/forum/#!topic/spark-users/CGfEafqiTsA> brought up earlier.
Reza On Fri, Nov 7, 2014 at 8:31 AM, Duy Huynh <[email protected]> wrote: > thanks reza. i'm not familiar with the "block matrix multiplication", but > is it a good fit for "very large dimension, but extremely sparse" matrix? > > if not, what is your recommendation on implementing matrix multiplication > in spark on "very large dimension, but extremely sparse" matrix? > > > > > On Thu, Nov 6, 2014 at 5:50 PM, Reza Zadeh <[email protected]> wrote: > >> See this thread for examples of sparse matrix x sparse matrix: >> https://groups.google.com/forum/#!topic/spark-users/CGfEafqiTsA >> >> We thought about providing matrix multiplies on CoordinateMatrix, >> however, the matrices have to be very dense for the overhead of having many >> little (i, j, value) objects to be worth it. For this reason, we are >> focused on doing block matrix multiplication first. The goal is version 1.3. >> >> Best, >> Reza >> >> On Wed, Nov 5, 2014 at 11:48 PM, Wei Tan <[email protected]> wrote: >> >>> I think Xiangrui's ALS code implement certain aspect of it. You may want >>> to check it out. >>> Best regards, >>> Wei >>> >>> --------------------------------- >>> Wei Tan, PhD >>> Research Staff Member >>> IBM T. J. Watson Research Center >>> >>> >>> [image: Inactive hide details for Xiangrui Meng ---11/05/2014 01:13:40 >>> PM---You can use breeze for local sparse-sparse matrix multiplic]Xiangrui >>> Meng ---11/05/2014 01:13:40 PM---You can use breeze for local sparse-sparse >>> matrix multiplication and then define an RDD of sub-matri >>> >>> From: Xiangrui Meng <[email protected]> >>> To: Duy Huynh <[email protected]> >>> Cc: user <[email protected]> >>> Date: 11/05/2014 01:13 PM >>> Subject: Re: sparse x sparse matrix multiplication >>> ------------------------------ >>> >>> >>> >>> You can use breeze for local sparse-sparse matrix multiplication and >>> then define an RDD of sub-matrices >>> >>> RDD[(Int, Int, CSCMatrix[Double])] (blockRowId, blockColId, sub-matrix) >>> >>> and then use join and aggregateByKey to implement this feature, which >>> is the same as in MapReduce. >>> >>> -Xiangrui >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> >>> >> >
