[
https://issues.apache.org/jira/browse/SPARK-32232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154311#comment-17154311
]
Huaxin Gao commented on SPARK-32232:
------------------------------------
I think the problem is this:
model2.transform(df) -> wrapper._transfer_params_to_java -> get the default
value for solver which is 'auto', this causes Exception because it is an
invalid solver.
On python side, we set solver default value to 'l-bfgs' in
MultilayerPerceptron, but we don't set solver default value in
MultilayerPerceptronModel. This makes sense because for fitted model, we don't
need the solver value any more, so no need to set it. However, since we don't
set default value for solver in MultilayerPerceptronModel, when calling
wrapper._transfer_params_to_java, the default value for solver in HasSolver is
used, which is 'auto', and this caused Exception because it is an invalid
solver.
Seems to me that we don't need to set solver or set any of the params for
MultilayerPerceptronModel. I will fix this problem.
> IllegalArgumentException: MultilayerPerceptronClassifier_... parameter solver
> given invalid value auto
> ------------------------------------------------------------------------------------------------------
>
> Key: SPARK-32232
> URL: https://issues.apache.org/jira/browse/SPARK-32232
> Project: Spark
> Issue Type: Bug
> Components: ML
> Affects Versions: 3.0.0
> Reporter: steven taylor
> Priority: Major
>
> I believe I have discovered a bug when loading
> MultilayerPerceptronClassificationModel in spark 3.0.0, scala 2.1.2 which I
> have tested and can see is not there in at least Spark 2.4.3, Scala 2.11.
> (I'm not sure if the Scala version is important).
>
> I am using pyspark on a databricks cluster and importing the library "from
> pyspark.ml.classification import MultilayerPerceptronClassificationModel"
>
> When running model=MultilayerPerceptronClassificationModel.("load") and then
> model. transform (df) I get the following error: IllegalArgumentException:
> MultilayerPerceptronClassifier_8055d1368e78 parameter solver given invalid
> value auto.
>
>
> This issue can be easily replicated by running the example given on the spark
> documents:
> [http://spark.apache.org/docs/latest/ml-classification-regression.html#multilayer-perceptron-classifier]
>
> Then adding a save model, load model and transform statement as such:
>
> *from* *pyspark.ml.classification* *import* MultilayerPerceptronClassifier
> *from* *pyspark.ml.evaluation* *import* MulticlassClassificationEvaluator
>
> _# Load training data_
> data = spark.read.format("libsvm")\
> .load("data/mllib/sample_multiclass_classification_data.txt")
>
> _# Split the data into train and test_
> splits = data.randomSplit([0.6, 0.4], 1234)
> train = splits[0]
> test = splits[1]
>
> _# specify layers for the neural network:_
> _# input layer of size 4 (features), two intermediate of size 5 and 4_
> _# and output of size 3 (classes)_
> layers = [4, 5, 4, 3]
>
> _# create the trainer and set its parameters_
> trainer = MultilayerPerceptronClassifier(maxIter=100, layers=layers,
> blockSize=128, seed=1234)
>
> _# train the model_
> model = trainer.fit(train)
>
> _# compute accuracy on the test set_
> result = model.transform(test)
> predictionAndLabels = result.select("prediction", "label")
> evaluator = MulticlassClassificationEvaluator(metricName="accuracy")
> *print*("Test set accuracy = " + str(evaluator.evaluate(predictionAndLabels)))
>
> *from* *pyspark.ml.classification* *import* MultilayerPerceptronClassifier,
> MultilayerPerceptronClassificationModel
> model.save(Save_location)
> model2. MultilayerPerceptronClassificationModel.load(Save_location)
>
> result_from_loaded = model2.transform(test)
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
