Java heap error during matrix multiplication

2017-01-25 Thread Petr Shestov
Hi all! I'm using Spark 2.0.1 with two workers (one executor each) with 20Gb each. And run following code: JavaRDD entries = ...; // filing the dataCoordinateMatrix cmatrix = new CoordinateMatrix(entries.rdd());BlockMatrix matrix = cmatrix.toBlockMatrix(100, 1000);BlockMatrix cooc =

Proper saving/loading of MatrixFactorizationModel

2015-07-20 Thread Petr Shestov
Hi all! I have MatrixFactorizationModel object. If I'm trying to recommend products to single user right after constructing model through ALS.train(...) then it takes 300ms (for my data and hardware). But if I save model to disk and load it back then recommendation takes almost 2000ms. Also