[
https://issues.apache.org/jira/browse/SPARK-61714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61714:
------------------------------------
Assignee: Hyukjin Kwon
> Add PySpark's Pipeline, CrossValidator, TrainValidationSplit and their models
> to the Rust API
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-61714
> URL: https://issues.apache.org/jira/browse/SPARK-61714
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Part of completing the native ML classes and the Rust API to the full
> pyspark.ml 4.3.0 surface, after the validators of the Python classes
> (SPARK-61708).
> The Rust API gets PySpark's meta-algorithms, of any estimators, transformers,
> models and evaluators, as PySpark 4.3.0 runs them over Spark Connect, on the
> client:
> - stages of cloneable estimators, transformers and models, and param maps of
> the uid of a stage and the name of its param, with PySpark's ParamGridBuilder;
> - Pipeline and PipelineModel, of Pipeline._fit;
> - CrossValidator, TrainValidationSplit and their models: PySpark's splits,
> cached, the fits of the param maps in parallel, numpy's metrics and best
> index, and the fit of the best map;
> - the persistence of PySpark's Python format, which PySpark loads of the Rust
> API and the Rust API of PySpark.
> They replace the Pipeline of stage names and the CrossValidator of numeric
> params of the Rust API, whose fits raised NOT_IMPLEMENTED.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]