Hyukjin Kwon created SPARK-61923:
------------------------------------

             Summary: Send and save the ML params as PySpark's 
serialize_ml_params does: params of no converter, params of subclasses, and the 
class of subclasses of meta-algorithms
                 Key: SPARK-61923
                 URL: https://issues.apache.org/jira/browse/SPARK-61923
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


The saves of the native ML classes of the drop-in differ from PySpark 4.3.0's:
- RegexTokenizer.gaps, a param of no converter, is never written by save, so a 
saved tokenizer (also inside a Pipeline or PipelineModel) loads with gaps=True 
and gives other results; values that PySpark's save refuses (gaps=(1,), 2**63, 
1, "yes", [True]) save silently.
- The params of a Python subclass that are set are not sent, at a fit, 
transform, evaluate or save, where PySpark's serialize_ml_params sends every 
item of _paramMap, which the server refuses.
- Subclasses of OneVsRest, Pipeline, CrossValidator and TrainValidationSplit 
save a constant class name, where PySpark writes the instance's own module and 
class name.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to