Re: Can't load a RandomForestClassificationModel in Spark job

2017-02-16 Thread Russell Jurney
t; > > > > *From:* Sumona Routh [mailto:sumos...@gmail.com] > *Sent:* Thursday, January 12, 2017 6:20 PM > *To:* ayan guha <guha.a...@gmail.com>; user@spark.apache.org > *Subject:* Re: Can't load a RandomForestClassificationModel in Spark job > > > > Yes, I save it

RE: Can't load a RandomForestClassificationModel in Spark job

2017-02-16 Thread Jianhong Xia
guha <guha.a...@gmail.com>; user@spark.apache.org Subject: Re: Can't load a RandomForestClassificationModel in Spark job Hey there, Creating a new SparkContext on workers will not work, only the driver is allowed to own a SparkContext. Are you trying to distribute your model to workers so you can cr

Re: Can't load a RandomForestClassificationModel in Spark job

2017-02-14 Thread Hollin Wilkins
that run machine > learning models on the workers. > > > > > > Thanks, > > Jianhong > > > > > > *From:* Sumona Routh [mailto:sumos...@gmail.com] > *Sent:* Thursday, January 12, 2017 6:20 PM > *To:* ayan guha <guha.a...@gmail.com>; user@s

RE: Can't load a RandomForestClassificationModel in Spark job

2017-02-14 Thread Jianhong Xia
; user@spark.apache.org Subject: Re: Can't load a RandomForestClassificationModel in Spark job Yes, I save it to S3 in a different process. It is actually the RandomForestClassificationModel.load method (passed an s3 path) where I run into problems. When you say you load it during map stages, do you mean th

Re: Can't load a RandomForestClassificationModel in Spark job

2017-01-12 Thread Sumona Routh
Yes, I save it to S3 in a different process. It is actually the RandomForestClassificationModel.load method (passed an s3 path) where I run into problems. When you say you load it during map stages, do you mean that you are able to directly load a model from inside of a transformation? When I try

Re: Can't load a RandomForestClassificationModel in Spark job

2017-01-12 Thread ayan guha
Hi Given training and predictions are two different applications, I typically save model objects to hdfs and load it back during prediction map stages. Best Ayan On Fri, 13 Jan 2017 at 5:39 am, Sumona Routh wrote: > Hi all, > I've been working with Spark mllib 2.0.2

Can't load a RandomForestClassificationModel in Spark job

2017-01-12 Thread Sumona Routh
Hi all, I've been working with Spark mllib 2.0.2 RandomForestClassificationModel. I encountered two frustrating issues and would really appreciate some advice: 1) RandomForestClassificationModel is effectively not serializable (I assume it's referencing something that can't be serialized, since