Re: save SPark ml

Hi , please i need help about that question 2017-05-15 10:32 GMT+02:00 issues solution : > Hi, > I am under Pyspark 1.6 i want save my model in hdfs file like parquet > > how i can do this ? > > > My model it s a RandomForestClassifier performed with

save SPark ml

Hi, I am under Pyspark 1.6 i want save my model in hdfs file like parquet how i can do this ? My model it s a RandomForestClassifier performed with corssvalidation like this rf_csv2 = CrossValidator() how i can save it ? thx for adavance

Re: How to save spark-ML model in Java?

cv.fit is going to give you a CrossValidatorModel, if you want to extract the real model built. You need to do val cvModel = cv.fit(data) val plmodel = cvModel.bestModel.asInstanceOf[PipelineModel] val model = plmodel.stages(2).asInstanceOf[whatever_model] then you can model.save

Re: How to save spark-ML model in Java?

Hi, Thanks Rezaul and Asher Krim. The method suggested by Rezaul works fine for NaiveBayes but still fails for RandomForest and Multi-layer perceptron classifier. Everything properly is saved until this stage. CrossValidator cv = new CrossValidator() .setEstimator(pipeline)

Re: How to save spark-ML model in Java?

What version of Spark are you on? Although it's cut off, I think your error is with RandomForestClassifier, is that correct? If so, you should upgrade to spark 2 since I think this class only became writeable/readable in Spark 2 ( https://github.com/apache/spark/pull/12118) On Thu, Jan 12, 2017

Re: How to save spark-ML model in Java?

Hi Malshan, The error says that one (or more) of the estimators/stages is either not writable or compatible that supports overwrite/model write operation. Suppose you want to configure an ML pipeline consisting of three stages (i.e. estimator): tokenizer, hashingTF, and nb: val nb = new

How to save spark-ML model in Java?

Hi, When I try to save a pipeline model using spark ML (Java) , the following exception is thrown. java.lang.UnsupportedOperationException: Pipeline write will fail on this Pipeline because it contains a stage which does not implement Writable. Non-Writable stage: rfc_98f8c9e0bd04 of type class