Hi,
I'm trying to call Breeze::LBFGS from the master on each partition but
getting *NonSerializable* error.
I guess it's well-known that the Breeze DiffFunction is not serializable.
///
import breeze.linalg.{Vector => BV, DenseVector=>BDV, SparseVector=>BSV}
val lbfgs = new breeze.optimize.LBFGS[BDV[Double]]
val wInit: BDV[Double] = Array.fill(numFeatures)(0.0).toBreeze
def localUpdate(d:Array[(Double, BV[Double])], w:BDV[Double]) : BDV[Double]
{
def getObj = new DiffFunction[BDV[Double]] {
def calculate(w: BDV[Double]) : (Double, BDV[Double]) =
{
...
}
}
lbfgs.minimize(getObj, w)
}
rdd.mapPartitions{
iter: Iterator[(Double, BV[Double])] => {
val d : Array[(Double, BV[Double])] = iter.toArray
val w : BDV[Double] = localUpdate(d, wInit)
Iterator(w)
}
The following link talks about using the KyroSerializationWrapper as a
solution:
http://http://stackoverflow.com/questions/23050067/spark-task-not-serializable-how-to-work-with-complex-map-closures-that-call-o
<http://http://stackoverflow.com/questions/23050067/spark-task-not-serializable-how-to-work-with-complex-map-closures-that-call-o>
But I didn't have good luck yet. Can some one points to a work-around way to
do the serialization?
Thanks a lot.
Xuepeng
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/Breeze-DiffFunction-not-serializable-tp8996.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
