Ruifeng Zheng created SPARK-61629:
-------------------------------------

             Summary: Reject pandas DataFrames in native Rust-backed Python ML 
methods
                 Key: SPARK-61629
                 URL: https://issues.apache.org/jira/browse/SPARK-61629
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Ruifeng Zheng


The native Rust-backed Python ML methods currently accept pandas DataFrames by 
implicitly creating a Spark DataFrame with the active session, and model 
transforms may collect results back to pandas. This follows the deprecated 
pyspark.ml.connect input contract rather than standard pyspark.ml over Spark 
Connect.

Require Spark DataFrames for native fit, transform and evaluate. Reject pandas 
inputs consistently with TypeError before parameter validation, schema access 
or RPCs, and direct callers to convert explicitly with spark.createDataFrame. 
Remove the implicit pandas conversion and model-output conversion helpers. 
Cover empty inputs, pandas subclasses, vector cells, all native 
estimator/evaluator entry points, and explicit conversion against a live Spark 
server. Keep vendored Python scaler behavior unchanged.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to