[
https://issues.apache.org/jira/browse/SPARK-62014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-62014.
----------------------------------
Fix Version/s: connect-rust-4.4.0
Resolution: Fixed
Issue resolved by pull request 859
[https://github.com/apache/spark-connect-rust/pull/859]
> Keep the state of Python subclasses of the native ML classes in their pickles
> and copies, give pyspark.ml.stat PySpark's docs, and make the ML objects'
> pickles, uids, metadata and attributes PySpark's
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62014
> URL: https://issues.apache.org/jira/browse/SPARK-62014
> Project: Spark
> Issue Type: Bug
> Components: ML
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> From a review of the documented differences of ML (docs/compatibility.md,
> dev/parity/known-differences.md) against PySpark 4.3.0, and the other
> session's review of the last 100 commits:
> - A Python subclass's __getstate__, __setstate__ and __reduce__ of a native
> ML class are ignored: a pickle or a deep copy of a subclass of Tokenizer
> loses its uid and Params (getInputCol raises "Param ... does not belong to"),
> and with __getstate__ only, a pickled maxIter=3 becomes 100; copy.copy and
> Params.copy ignore them too. uid is a native property, not in __dict__.
> - The static methods of ChiSquareTest, Correlation, KolmogorovSmirnovTest and
> Summarizer lack PySpark's signatures, annotations and docstrings, and the
> classes PySpark's docstrings.
> - DataFrames of the vendored pyspark.ml models pickle and deep copy, where
> PySpark raises "cannot pickle '_thread.lock'"; the plan walk of the pickle
> check does not enter subqueries.
> - A validator subclass's own Params are not saved in its metadata; a subclass
> of PipelineModel, OneVsRestModel or a validator model has the uid of its base
> class; Pipeline, OneVsRest, the validators and their models have a _java_obj
> attribute, which PySpark's do not; an object with sparkSession but no _plan
> raises a TypeError where PySpark raises the AttributeError of _plan; the
> _remote_model_obj of a summary is not the model's reference; members of the
> native classes do not cloudpickle; a Pipeline of a one-shot iterator of
> stages saves no stages.
> - Several ML entries of the docs are stale or lack their reasons (uid
> assignment, CrossValidator's setParams, OneVsRest's write, the cause of the
> RandomForest fold failure, the non-active session, shallow copies).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]