[ 
https://issues.apache.org/jira/browse/SPARK-62014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-62014.
----------------------------------
    Fix Version/s: connect-rust-4.4.0
       Resolution: Fixed

Issue resolved by pull request 859
[https://github.com/apache/spark-connect-rust/pull/859]

> Keep the state of Python subclasses of the native ML classes in their pickles 
> and copies, give pyspark.ml.stat PySpark's docs, and make the ML objects' 
> pickles, uids, metadata and attributes PySpark's
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62014
>                 URL: https://issues.apache.org/jira/browse/SPARK-62014
>             Project: Spark
>          Issue Type: Bug
>          Components: ML
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> From a review of the documented differences of ML (docs/compatibility.md, 
> dev/parity/known-differences.md) against PySpark 4.3.0, and the other 
> session's review of the last 100 commits:
> - A Python subclass's __getstate__, __setstate__ and __reduce__ of a native 
> ML class are ignored: a pickle or a deep copy of a subclass of Tokenizer 
> loses its uid and Params (getInputCol raises "Param ... does not belong to"), 
> and with __getstate__ only, a pickled maxIter=3 becomes 100; copy.copy and 
> Params.copy ignore them too. uid is a native property, not in __dict__.
> - The static methods of ChiSquareTest, Correlation, KolmogorovSmirnovTest and 
> Summarizer lack PySpark's signatures, annotations and docstrings, and the 
> classes PySpark's docstrings.
> - DataFrames of the vendored pyspark.ml models pickle and deep copy, where 
> PySpark raises "cannot pickle '_thread.lock'"; the plan walk of the pickle 
> check does not enter subqueries.
> - A validator subclass's own Params are not saved in its metadata; a subclass 
> of PipelineModel, OneVsRestModel or a validator model has the uid of its base 
> class; Pipeline, OneVsRest, the validators and their models have a _java_obj 
> attribute, which PySpark's do not; an object with sparkSession but no _plan 
> raises a TypeError where PySpark raises the AttributeError of _plan; the 
> _remote_model_obj of a summary is not the model's reference; members of the 
> native classes do not cloudpickle; a Pipeline of a one-shot iterator of 
> stages saves no stages.
> - Several ML entries of the docs are stale or lack their reasons (uid 
> assignment, CrossValidator's setParams, OneVsRest's write, the cause of the 
> RandomForest fold failure, the non-active session, shallow copies).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to