Hyukjin Kwon created SPARK-61630:
------------------------------------
Summary: Make the native LogisticRegression and evaluators follow
pyspark.ml, not the deprecated pyspark.ml.connect, and stop converting array
features implicitly, in the Python client
Key: SPARK-61630
URL: https://issues.apache.org/jira/browse/SPARK-61630
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
The native ML classes of the Python client took the deprecated
pyspark.ml.connect API (deprecated since Spark 4.0, PyTorch and pandas based)
as their compatibility target in several places. The target is the standard
pyspark.ml API over Spark Connect. (1) The native LogisticRegression has the
six params of the deprecated PyTorch LogisticRegression, outputs a bigint
prediction and array<double> probabilities without a raw prediction, checks the
labels on the client as the deprecated _fit does, and raises the deprecated
Transformer's ValueError for an existing output column. It should have the
params of pyspark.ml.classification.LogisticRegression, with setParams, and the
JVM model's outputs and errors. (2) The native RegressionEvaluator,
BinaryClassificationEvaluator and MulticlassClassificationEvaluator have the
params and defaults of the deprecated evaluators (probabilityCol instead of
rawPredictionCol, an accuracy default instead of f1, positional arguments, the
scoring of array probabilities). They should have those of
pyspark.ml.evaluation, with setParams. (3) The native estimators convert an
array features column to a vector implicitly, as the deprecated classes take
arrays. pyspark.ml sends the column as it is, and the JVM rejects a column that
is not a vector.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]