Following codes works, too
class Foo1 extends Serializable { def foo() = Array(1.0) }
val t1 = new Foo1
val m1 = t1.foo
val r11 = sc.parallelize(List(1, 2, 3))
val r22 = r11.map(_ + m1(0))
r22.toArray
On Thu, Aug 14, 2014 at 10:55 PM, Shixiong Zhu [via Apache Spark User List]
<ml-node+s1001560n12112...@n3.nabble.com> wrote:
> I think I can reproduce this error.
>
> The following code cannot work and report "Foo" cannot be serialized. (log
> in gist https://gist.github.com/zsxwing/4f9f17201d4378fe3e16):
>
> class Foo { def foo() = Array(1.0) }
> val t = new Foo
> val m = t.foo
> val r1 = sc.parallelize(List(1, 2, 3))
> val r2 = r1.map(_ + m(0))
> r2.toArray
>
> But the following code can work (log in gist
> https://gist.github.com/zsxwing/802cade0facb36a37656):
>
> class Foo { def foo() = Array(1.0) }
> var m: Array[Double] = null
> {
> val t = new Foo
> m = t.foo
> }
> val r1 = sc.parallelize(List(1, 2, 3))
> val r2 = r1.map(_ + m(0))
> r2.toArray
>
>
> Best Regards,
> Shixiong Zhu
>
>
> 2014-08-14 22:11 GMT+08:00 Christopher Nguyen <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=12112&i=0>>:
>
>> Hi Hoai-Thu, the issue of private default constructor is unlikely the
>> cause here, since Lance was already able to load/deserialize the model
>> object.
>>
>> And on that side topic, I wish all serdes libraries would just use
>> constructor.setAccessible(true) by default :-) Most of the time that
>> privacy is not about serdes reflection restrictions.
>>
>> Sent while mobile. Pls excuse typos etc.
>> On Aug 14, 2014 1:58 AM, "Hoai-Thu Vuong" <[hidden email]
>> <http://user/SendEmail.jtp?type=node&node=12112&i=1>> wrote:
>>
>>> A man in this community give me a video:
>>> https://www.youtube.com/watch?v=sPhyePwo7FA. I've got a same question
>>> in this community and other guys helped me to solve this problem. I'm
>>> trying to load MatrixFactorizationModel from object file, but compiler said
>>> that, I can not create object because the constructor is private. To solve
>>> this, I put my new object to same package as MatrixFactorizationModel.
>>> Luckly it works.
>>>
>>>
>>> On Wed, Aug 13, 2014 at 9:20 PM, Christopher Nguyen <[hidden email]
>>> <http://user/SendEmail.jtp?type=node&node=12112&i=2>> wrote:
>>>
>>>> Lance, some debugging ideas: you might try model.predict(RDD[Vector])
>>>> to isolate the cause to serialization of the loaded model. And also try to
>>>> serialize the deserialized (loaded) model "manually" to see if that throws
>>>> any visible exceptions.
>>>>
>>>> Sent while mobile. Pls excuse typos etc.
>>>> On Aug 13, 2014 7:03 AM, "lancezhange" <[hidden email]
>>>> <http://user/SendEmail.jtp?type=node&node=12112&i=3>> wrote:
>>>>
>>>>> my prediction codes are simple enough as follows:
>>>>>
>>>>> *val labelsAndPredsOnGoodData = goodDataPoints.map { point =>
>>>>> val prediction = model.predict(point.features)
>>>>> (point.label, prediction)
>>>>> }*
>>>>>
>>>>> when model is the loaded one, above code just can't work. Can you
>>>>> catch the
>>>>> error?
>>>>> Thanks.
>>>>>
>>>>> PS. i use spark-shell under standalone mode, version 1.0.0
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-save-mllib-model-to-hdfs-and-reload-it-tp11953p12035.html
>>>>> Sent from the Apache Spark User List mailing list archive at
>>>>> Nabble.com.
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: [hidden email]
>>>>> <http://user/SendEmail.jtp?type=node&node=12112&i=4>
>>>>> For additional commands, e-mail: [hidden email]
>>>>> <http://user/SendEmail.jtp?type=node&node=12112&i=5>
>>>>>
>>>>>
>>>
>>>
>>> --
>>> Thu.
>>>
>>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-save-mllib-model-to-hdfs-and-reload-it-tp11953p12112.html
> To unsubscribe from How to save mllib model to hdfs and reload it, click
> here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=11953&code=bGFuY2V6aGFuZ2VAZ21haWwuY29tfDExOTUzfDEyOTg0NDQwMjM=>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
-- 张喜升
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-save-mllib-model-to-hdfs-and-reload-it-tp11953p12114.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
