Hi, I'm trying to call Breeze::LBFGS from the master on each partition but getting *NonSerializable* error. I guess it's well-known that the Breeze DiffFunction is not serializable. /// import breeze.linalg.{Vector => BV, DenseVector=>BDV, SparseVector=>BSV}
val lbfgs = new breeze.optimize.LBFGS[BDV[Double]] val wInit: BDV[Double] = Array.fill(numFeatures)(0.0).toBreeze def localUpdate(d:Array[(Double, BV[Double])], w:BDV[Double]) : BDV[Double] { def getObj = new DiffFunction[BDV[Double]] { def calculate(w: BDV[Double]) : (Double, BDV[Double]) = { ... } } lbfgs.minimize(getObj, w) } rdd.mapPartitions{ iter: Iterator[(Double, BV[Double])] => { val d : Array[(Double, BV[Double])] = iter.toArray val w : BDV[Double] = localUpdate(d, wInit) Iterator(w) } The following link talks about using the KyroSerializationWrapper as a solution: http://http://stackoverflow.com/questions/23050067/spark-task-not-serializable-how-to-work-with-complex-map-closures-that-call-o <http://http://stackoverflow.com/questions/23050067/spark-task-not-serializable-how-to-work-with-complex-map-closures-that-call-o> But I didn't have good luck yet. Can some one points to a work-around way to do the serialization? Thanks a lot. Xuepeng -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Breeze-DiffFunction-not-serializable-tp8996.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org