Hyukjin Kwon created SPARK-61631:
------------------------------------
Summary: Make the native evaluators follow pyspark.ml.evaluation,
not the deprecated pyspark.ml.connect, in the Python client
Key: SPARK-61631
URL: https://issues.apache.org/jira/browse/SPARK-61631
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
The native RegressionEvaluator, BinaryClassificationEvaluator and
MulticlassClassificationEvaluator of the Python client follow the deprecated
pyspark.ml.connect evaluators (deprecated since Spark 4.0).
BinaryClassificationEvaluator takes probabilityCol instead of rawPredictionCol.
MulticlassClassificationEvaluator defaults to accuracy instead of f1 and takes
positional arguments. evaluate casts the prediction and the probability, and
scores element 1 of an array of probabilities. The evaluators lack the params
weightCol, throughOrigin, numBins, metricLabel, beta and eps, and setParams.
They should follow the standard pyspark.ml.evaluation API over Spark Connect:
its keyword-only signatures, params, defaults, setters, getters, setParams and
isLargerBetter, with the dataset sent as pyspark.ml sends it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]