[
https://issues.apache.org/jira/browse/SPARK-61923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61923:
------------------------------------
Assignee: Hyukjin Kwon
> Send and save the ML params as PySpark's serialize_ml_params does: params of
> no converter, params of subclasses, and the class of subclasses of
> meta-algorithms
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61923
> URL: https://issues.apache.org/jira/browse/SPARK-61923
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> The saves of the native ML classes of the drop-in differ from PySpark 4.3.0's:
> - RegexTokenizer.gaps, a param of no converter, is never written by save, so
> a saved tokenizer (also inside a Pipeline or PipelineModel) loads with
> gaps=True and gives other results; values that PySpark's save refuses
> (gaps=(1,), 2**63, 1, "yes", [True]) save silently.
> - The params of a Python subclass that are set are not sent, at a fit,
> transform, evaluate or save, where PySpark's serialize_ml_params sends every
> item of _paramMap, which the server refuses.
> - Subclasses of OneVsRest, Pipeline, CrossValidator and TrainValidationSplit
> save a constant class name, where PySpark writes the instance's own module
> and class name.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]