Hyukjin Kwon created SPARK-61631:
------------------------------------

             Summary: Make the native evaluators follow pyspark.ml.evaluation, 
not the deprecated pyspark.ml.connect, in the Python client
                 Key: SPARK-61631
                 URL: https://issues.apache.org/jira/browse/SPARK-61631
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


The native RegressionEvaluator, BinaryClassificationEvaluator and 
MulticlassClassificationEvaluator of the Python client follow the deprecated 
pyspark.ml.connect evaluators (deprecated since Spark 4.0). 
BinaryClassificationEvaluator takes probabilityCol instead of rawPredictionCol. 
MulticlassClassificationEvaluator defaults to accuracy instead of f1 and takes 
positional arguments. evaluate casts the prediction and the probability, and 
scores element 1 of an array of probabilities. The evaluators lack the params 
weightCol, throughOrigin, numBins, metricLabel, beta and eps, and setParams. 
They should follow the standard pyspark.ml.evaluation API over Spark Connect: 
its keyword-only signatures, params, defaults, setters, getters, setParams and 
isLargerBetter, with the dataset sent as pyspark.ml sends it.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to