I think I can reproduce this error. The following code cannot work and report "Foo" cannot be serialized. (log in gist https://gist.github.com/zsxwing/4f9f17201d4378fe3e16):
class Foo { def foo() = Array(1.0) } val t = new Foo val m = t.foo val r1 = sc.parallelize(List(1, 2, 3)) val r2 = r1.map(_ + m(0)) r2.toArray But the following code can work (log in gist https://gist.github.com/zsxwing/802cade0facb36a37656): class Foo { def foo() = Array(1.0) } var m: Array[Double] = null { val t = new Foo m = t.foo } val r1 = sc.parallelize(List(1, 2, 3)) val r2 = r1.map(_ + m(0)) r2.toArray Best Regards, Shixiong Zhu 2014-08-14 22:11 GMT+08:00 Christopher Nguyen <c...@adatao.com>: > Hi Hoai-Thu, the issue of private default constructor is unlikely the > cause here, since Lance was already able to load/deserialize the model > object. > > And on that side topic, I wish all serdes libraries would just use > constructor.setAccessible(true) by default :-) Most of the time that > privacy is not about serdes reflection restrictions. > > Sent while mobile. Pls excuse typos etc. > On Aug 14, 2014 1:58 AM, "Hoai-Thu Vuong" <thuv...@gmail.com> wrote: > >> A man in this community give me a video: >> https://www.youtube.com/watch?v=sPhyePwo7FA. I've got a same question in >> this community and other guys helped me to solve this problem. I'm trying >> to load MatrixFactorizationModel from object file, but compiler said that, >> I can not create object because the constructor is private. To solve this, >> I put my new object to same package as MatrixFactorizationModel. Luckly it >> works. >> >> >> On Wed, Aug 13, 2014 at 9:20 PM, Christopher Nguyen <c...@adatao.com> >> wrote: >> >>> Lance, some debugging ideas: you might try model.predict(RDD[Vector]) to >>> isolate the cause to serialization of the loaded model. And also try to >>> serialize the deserialized (loaded) model "manually" to see if that throws >>> any visible exceptions. >>> >>> Sent while mobile. Pls excuse typos etc. >>> On Aug 13, 2014 7:03 AM, "lancezhange" <lancezha...@gmail.com> wrote: >>> >>>> my prediction codes are simple enough as follows: >>>> >>>> *val labelsAndPredsOnGoodData = goodDataPoints.map { point => >>>> val prediction = model.predict(point.features) >>>> (point.label, prediction) >>>> }* >>>> >>>> when model is the loaded one, above code just can't work. Can you catch >>>> the >>>> error? >>>> Thanks. >>>> >>>> PS. i use spark-shell under standalone mode, version 1.0.0 >>>> >>>> >>>> >>>> >>>> -- >>>> View this message in context: >>>> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-save-mllib-model-to-hdfs-and-reload-it-tp11953p12035.html >>>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>> For additional commands, e-mail: user-h...@spark.apache.org >>>> >>>> >> >> >> -- >> Thu. >> >