Hi Malshan,
The error says that one (or more) of the estimators/stages is either not
writable or compatible that supports overwrite/model write operation.
Suppose you want to configure an ML pipeline consisting of three stages
(i.e. estimator): tokenizer, hashingTF, and nb:
val nb = new NaiveBayes().setSmoothing(0.00001)
val tokenizer = new
Tokenizer().setInputCol("label").setOutputCol("label")
val hashingTF = new
HashingTF().setInputCol(tokenizer.getOutputCol).setOutputCol("features")
val pipeline = new Pipeline().setStages(Array(tokenizer, hashingTF,
nb))
Now check if all the stages are writable. And to make it ease try saving
stages individually: -e.g. tokenizer.write.save("path")
hashingTF.write.save("path")
After that suppose you want to perform a 10-fold cross-validation as
follows:
val cv = new CrossValidator()
.setEstimator(pipeline)
.setEvaluator(new BinaryClassificationEvaluator)
.setEstimatorParamMaps(paramGrid)
.setNumFolds(10)
Where:
val paramGrid = new ParamGridBuilder()
.addGrid(hashingTF.numFeatures, Array(10, 100,
1000))
.addGrid(nb.smoothing, Array(0.001, 0.0001))
.build()
Now the model that you trained using the training set should be writable if
all of the stages are okay:
val model = cv.fit(trainingData)
model.write.overwrite().save("output/NBModel")
Hope that helps.
Regards,
_________________________________
*Md. Rezaul Karim*, BSc, MSc
PhD Researcher, INSIGHT Centre for Data Analytics
National University of Ireland, Galway
IDA Business Park, Dangan, Galway, Ireland
Web: http://www.reza-analytics.eu/index.html
<http://139.59.184.114/index.html>
On 12 January 2017 at 09:09, Minudika Malshan <minudika...@gmail.com> wrote:
> Hi,
>
> When I try to save a pipeline model using spark ML (Java) , the following
> exception is thrown.
>
>
> java.lang.UnsupportedOperationException: Pipeline write will fail on this
> Pipeline because it contains a stage which does not implement Writable.
> Non-Writable stage: rfc_98f8c9e0bd04 of type class org.apache.spark.ml.
> classification.Rand
>
>
> Here is my code segment.
>
>
> model.write().overwrite,save
>
>
> model.write().overwrite().save("path
> model.write().overwrite().save("mypath");
>
>
> How to resolve this?
>
> Thanks and regards!
>
> Minudika
>
>
