When I call the following minimal working example, the accumulator matrix is 32-by-100K, and each executor has 64G but I get an out of memory error:
Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit Here BDM is a Breeze DenseMatrix object BDMAccumulatorParam extends AccumulatorParam[BDM[Double]] { def zero(initialValue: BDM[Double]): BDM[Double] = { BDM.zeros[Double](initialValue.rows, initialValue.cols) } def addInPlace(m1: BDM[Double], m2: BDM[Double]) : BDM[Double] = { m1 += m2 } } def testfun(mat: IndexedRowMatrix, lhs: DenseMatrix) : DenseMatrix = { val accum = mat.rows.context.accumulator(BDM.zeros[Double](lhs.numRows.toInt, mat.numCols.toInt))(BDMAccumulatorParam) mat.rows.foreach(row => accum += BDM.ones[Double](lhs.numRows.toInt, mat.numCols.toInt)) fromBreeze(accum.value) } Any ideas or suggestions on how to avoid this error? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/small-accumulator-gives-out-of-memory-error-tp23864.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org