Hyukjin Kwon created SPARK-62014:
------------------------------------

             Summary: Keep the state of Python subclasses of the native ML 
classes in their pickles and copies, give pyspark.ml.stat PySpark's docs, and 
make the ML objects' pickles, uids, metadata and attributes PySpark's
                 Key: SPARK-62014
                 URL: https://issues.apache.org/jira/browse/SPARK-62014
             Project: Spark
          Issue Type: Bug
          Components: ML
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


>From a review of the documented differences of ML (docs/compatibility.md, 
>dev/parity/known-differences.md) against PySpark 4.3.0, and the other 
>session's review of the last 100 commits:
- A Python subclass's __getstate__, __setstate__ and __reduce__ of a native ML 
class are ignored: a pickle or a deep copy of a subclass of Tokenizer loses its 
uid and Params (getInputCol raises "Param ... does not belong to"), and with 
__getstate__ only, a pickled maxIter=3 becomes 100; copy.copy and Params.copy 
ignore them too. uid is a native property, not in __dict__.
- The static methods of ChiSquareTest, Correlation, KolmogorovSmirnovTest and 
Summarizer lack PySpark's signatures, annotations and docstrings, and the 
classes PySpark's docstrings.
- DataFrames of the vendored pyspark.ml models pickle and deep copy, where 
PySpark raises "cannot pickle '_thread.lock'"; the plan walk of the pickle 
check does not enter subqueries.
- A validator subclass's own Params are not saved in its metadata; a subclass 
of PipelineModel, OneVsRestModel or a validator model has the uid of its base 
class; Pipeline, OneVsRest, the validators and their models have a _java_obj 
attribute, which PySpark's do not; an object with sparkSession but no _plan 
raises a TypeError where PySpark raises the AttributeError of _plan; the 
_remote_model_obj of a summary is not the model's reference; members of the 
native classes do not cloudpickle; a Pipeline of a one-shot iterator of stages 
saves no stages.
- Several ML entries of the docs are stale or lack their reasons (uid 
assignment, CrossValidator's setParams, OneVsRest's write, the cause of the 
RandomForest fold failure, the non-active session, shallow copies).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to