Hyukjin Kwon created SPARK-61714:
------------------------------------

             Summary: Add PySpark's Pipeline, CrossValidator, 
TrainValidationSplit and their models to the Rust API
                 Key: SPARK-61714
                 URL: https://issues.apache.org/jira/browse/SPARK-61714
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


Part of completing the native ML classes and the Rust API to the full 
pyspark.ml 4.3.0 surface, after the validators of the Python classes 
(SPARK-61708).

The Rust API gets PySpark's meta-algorithms, of any estimators, transformers, 
models and evaluators, as PySpark 4.3.0 runs them over Spark Connect, on the 
client:
- stages of cloneable estimators, transformers and models, and param maps of 
the uid of a stage and the name of its param, with PySpark's ParamGridBuilder;
- Pipeline and PipelineModel, of Pipeline._fit;
- CrossValidator, TrainValidationSplit and their models: PySpark's splits, 
cached, the fits of the param maps in parallel, numpy's metrics and best index, 
and the fit of the best map;
- the persistence of PySpark's Python format, which PySpark loads of the Rust 
API and the Rust API of PySpark.

They replace the Pipeline of stage names and the CrossValidator of numeric 
params of the Rust API, whose fits raised NOT_IMPLEMENTED.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to