GML is a fast, distributed, in-memory sparse (and dense) matrix libraries.

It does not use RDDs for resilience. Instead we have examples that use Resilient X10 (which provides recovery of distributed control structures in case of node failure) and Hazelcast for stable storage.

We are looking to benchmark with RDDs to compare overhead, and also looking to see how the same ideas could be realized on top of RDDs.


On 2/28/15 7:25 PM, Joseph Bradley wrote:
Hi Shahab,

There are actually a few distributed Matrix types which support sparse representations: RowMatrix, IndexedRowMatrix, and CoordinateMatrix. The documentation has a bit more info about the various uses: http://spark.apache.org/docs/latest/mllib-data-types.html#distributed-matrix

The Spark 1.3 RC includes a new one: BlockMatrix.

But since these are distributed, they are represented using RDDs, so they of course will not be as fast as computations on smaller, locally stored matrices.

Joseph

On Fri, Feb 27, 2015 at 4:39 AM, Ritesh Kumar Singh <[email protected] <mailto:[email protected]>> wrote:

    try using breeze (scala linear algebra library)

    On Fri, Feb 27, 2015 at 5:56 PM, shahab <[email protected]
    <mailto:[email protected]>> wrote:

        Thanks a lot Vijay, let me see how it performs.

        Best
        Shahab


        On Friday, February 27, 2015, Vijay Saraswat
        <[email protected] <mailto:[email protected]>> wrote:

            Available in GML --

            
http://x10-lang.org/x10-community/applications/global-matrix-library.html

            We are exploring how to make it available within Spark.
            Any ideas would be much appreciated.

            On 2/27/15 7:01 AM, shahab wrote:

                Hi,

                I just wonder if there is any Sparse Matrix
                implementation available  in Spark, so it can be used
                in spark application?

                best,
                /Shahab



            
---------------------------------------------------------------------
            To unsubscribe, e-mail: [email protected]
            For additional commands, e-mail: [email protected]




Reply via email to