[ 
https://issues.apache.org/jira/browse/SPARK-61629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ruifeng Zheng updated SPARK-61629:
----------------------------------
    Fix Version/s: connect-rust-4.3.0

> Reject pandas DataFrames in native Rust-backed Python ML methods
> ----------------------------------------------------------------
>
>                 Key: SPARK-61629
>                 URL: https://issues.apache.org/jira/browse/SPARK-61629
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Ruifeng Zheng
>            Priority: Major
>             Fix For: connect-rust-4.3.0
>
>
> The native Rust-backed Python ML methods currently accept pandas DataFrames 
> by implicitly creating a Spark DataFrame with the active session, and model 
> transforms may collect results back to pandas. This follows the deprecated 
> pyspark.ml.connect input contract rather than standard pyspark.ml over Spark 
> Connect.
> Require Spark DataFrames for native fit, transform and evaluate. Reject 
> pandas inputs consistently with TypeError before parameter validation, schema 
> access or RPCs, and direct callers to convert explicitly with 
> spark.createDataFrame. Remove the implicit pandas conversion and model-output 
> conversion helpers. Cover empty inputs, pandas subclasses, vector cells, all 
> native estimator/evaluator entry points, and explicit conversion against a 
> live Spark server. Keep vendored Python scaler behavior unchanged.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to