[
https://issues.apache.org/jira/browse/SPARK-61630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61630:
------------------------------------
Assignee: Hyukjin Kwon
> Make the native LogisticRegression and evaluators follow pyspark.ml, not the
> deprecated pyspark.ml.connect, and stop converting array features implicitly,
> in the Python client
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61630
> URL: https://issues.apache.org/jira/browse/SPARK-61630
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The native ML classes of the Python client took the deprecated
> pyspark.ml.connect API (deprecated since Spark 4.0, PyTorch and pandas based)
> as their compatibility target in several places. The target is the standard
> pyspark.ml API over Spark Connect. (1) The native LogisticRegression has the
> six params of the deprecated PyTorch LogisticRegression, outputs a bigint
> prediction and array<double> probabilities without a raw prediction, checks
> the labels on the client as the deprecated _fit does, and raises the
> deprecated Transformer's ValueError for an existing output column. It should
> have the params of pyspark.ml.classification.LogisticRegression, with
> setParams, and the JVM model's outputs and errors. (2) The native
> RegressionEvaluator, BinaryClassificationEvaluator and
> MulticlassClassificationEvaluator have the params and defaults of the
> deprecated evaluators (probabilityCol instead of rawPredictionCol, an
> accuracy default instead of f1, positional arguments, the scoring of array
> probabilities). They should have those of pyspark.ml.evaluation, with
> setParams. (3) The native estimators convert an array features column to a
> vector implicitly, as the deprecated classes take arrays. pyspark.ml sends
> the column as it is, and the JVM rejects a column that is not a vector.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]