Re: Is there any Sparse Matrix implementation in Spark/MLib?

GML is a fast, distributed, in-memory sparse (and dense) matrix libraries. It does not use RDDs for resilience. Instead we have examples that use Resilient X10 (which provides recovery of distributed control structures in case of node failure) and Hazelcast for stable storage. We are looking

Re: Is there any Sparse Matrix implementation in Spark/MLib?

Thanks Vijay, but the setup requirement for GML was not straightforward for me at all, so I put it aside for a while. best, /Shahab On Sun, Mar 1, 2015 at 9:34 AM, Vijay Saraswat vi...@saraswat.org wrote: GML is a fast, distributed, in-memory sparse (and dense) matrix libraries. It does

Re: Is there any Sparse Matrix implementation in Spark/MLib?

Thanks Josef for the comments, I think I need to do some benchmarking. best, /Shahab On Sun, Mar 1, 2015 at 1:25 AM, Joseph Bradley jos...@databricks.com wrote: Hi Shahab, There are actually a few distributed Matrix types which support sparse representations: RowMatrix, IndexedRowMatrix,

Re: Is there any Sparse Matrix implementation in Spark/MLib?

Hi Shahab, There are actually a few distributed Matrix types which support sparse representations: RowMatrix, IndexedRowMatrix, and CoordinateMatrix. The documentation has a bit more info about the various uses: http://spark.apache.org/docs/latest/mllib-data-types.html#distributed-matrix The

Re: Is there any Sparse Matrix implementation in Spark/MLib?

Available in GML -- http://x10-lang.org/x10-community/applications/global-matrix-library.html We are exploring how to make it available within Spark. Any ideas would be much appreciated. On 2/27/15 7:01 AM, shahab wrote: Hi, I just wonder if there is any Sparse Matrix implementation

Re: Is there any Sparse Matrix implementation in Spark/MLib?

Thanks a lot Vijay, let me see how it performs. Best Shahab On Friday, February 27, 2015, Vijay Saraswat vi...@saraswat.org wrote: Available in GML -- http://x10-lang.org/x10-community/applications/global-matrix-library.html We are exploring how to make it available within Spark. Any ideas

Re: Is there any Sparse Matrix implementation in Spark/MLib?

Thanks, But do you know if access to Coordinated matrix elements is almost as fast as a normal matrix or it has access time similar to RDD ( relatively slow)? I am looking for some fast access sparse matrix data structure. On Friday, February 27, 2015, Peter Rudenko petro.rude...@gmail.com

Re: Is there any Sparse Matrix implementation in Spark/MLib?

try using breeze (scala linear algebra library) On Fri, Feb 27, 2015 at 5:56 PM, shahab shahab.mok...@gmail.com wrote: Thanks a lot Vijay, let me see how it performs. Best Shahab On Friday, February 27, 2015, Vijay Saraswat vi...@saraswat.org wrote: Available in GML --

Is there any Sparse Matrix implementation in Spark/MLib?

Hi, I just wonder if there is any Sparse Matrix implementation available in Spark, so it can be used in spark application? best, /Shahab

Re: Is there any Sparse Matrix implementation in Spark/MLib?

Yes, it's called Coordinated Matrix(http://spark.apache.org/docs/latest/mllib-data-types.html#coordinatematrix) you need to fill it with elemets of type MatrixEntry( (Long, Long, Double)) Thanks, Peter Rudenko On 2015-02-27 14:01, shahab wrote: Hi, I just wonder if there is any Sparse