t;
>
>
>
> *From:* Sumona Routh [mailto:sumos...@gmail.com]
> *Sent:* Thursday, January 12, 2017 6:20 PM
> *To:* ayan guha <guha.a...@gmail.com>; user@spark.apache.org
> *Subject:* Re: Can't load a RandomForestClassificationModel in Spark job
>
>
>
> Yes, I save it
guha <guha.a...@gmail.com>;
user@spark.apache.org
Subject: Re: Can't load a RandomForestClassificationModel in Spark job
Hey there,
Creating a new SparkContext on workers will not work, only the driver is
allowed to own a SparkContext. Are you trying to distribute your model to
workers so you can cr
that run machine
> learning models on the workers.
>
>
>
>
>
> Thanks,
>
> Jianhong
>
>
>
>
>
> *From:* Sumona Routh [mailto:sumos...@gmail.com]
> *Sent:* Thursday, January 12, 2017 6:20 PM
> *To:* ayan guha <guha.a...@gmail.com>; user@s
; user@spark.apache.org
Subject: Re: Can't load a RandomForestClassificationModel in Spark job
Yes, I save it to S3 in a different process. It is actually the
RandomForestClassificationModel.load method (passed an s3 path) where I run
into problems.
When you say you load it during map stages, do you mean th
Yes, I save it to S3 in a different process. It is actually the
RandomForestClassificationModel.load method (passed an s3 path) where I run
into problems.
When you say you load it during map stages, do you mean that you are able
to directly load a model from inside of a transformation? When I try
Hi
Given training and predictions are two different applications, I typically
save model objects to hdfs and load it back during prediction map stages.
Best
Ayan
On Fri, 13 Jan 2017 at 5:39 am, Sumona Routh wrote:
> Hi all,
> I've been working with Spark mllib 2.0.2
Hi all,
I've been working with Spark mllib 2.0.2 RandomForestClassificationModel.
I encountered two frustrating issues and would really appreciate some
advice:
1) RandomForestClassificationModel is effectively not serializable (I
assume it's referencing something that can't be serialized, since