Hyukjin Kwon created SPARK-61531:
------------------------------------
Summary: Let the native evaluators take the numeric and array
columns and the pandas DataFrames that PySpark's evaluators take
Key: SPARK-61531
URL: https://issues.apache.org/jira/browse/SPARK-61531
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark's pyspark.ml.connect evaluators compute with torch on the client. They
read the columns as tensors: int labels and predictions, float32 or array
probabilities (column 1 of a 2-D array is the positive class), and they take a
pandas DataFrame too. The native evaluators of this client send the DataFrame
unchanged to Spark's JVM evaluators, which require a double prediction, and a
double or vector raw prediction. So PySpark's own docstring and test inputs
failed:
- MulticlassClassificationEvaluator on int columns: "Column prediction must be
of type ... DoubleType but was actually ... LongType".
- BinaryClassificationEvaluator on an array<double> or float probability: "must
be of type equal to one of the following types: [double, struct<...>] but was
actually of type array<double>".
- A pandas DataFrame raised "argument 'df': 'DataFrame' object is not an
instance of 'DataFrame'".
The evaluators now:
- take `dataset`, PySpark's name of the argument, as a DataFrame or a pandas
DataFrame, which is sent through createDataFrame of the active session;
- cast a prediction or a probability of another numeric type to a double;
- send an array probability as a vector by array_to_vector, which the JVM
evaluator reads at its element 1.
docs/compatibility.md now says:
- `setProbabilityCol` is a setter of this client, as PySpark's evaluators have
none;
- a model without a probability column (the native LinearSVC) is scored with
probabilityCol="rawPrediction";
- the native scalers take vector features, as the other native estimators do.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]