[
https://issues.apache.org/jira/browse/SPARK-61877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61877:
------------------------------------
Assignee: Hyukjin Kwon
> Make the native ML summaries and stat classes construct, copy, pickle and
> document as PySpark's, and fetch StopWordsRemover's defaults once
> -------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61877
> URL: https://issues.apache.org/jira/browse/SPARK-61877
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> The native ML summaries and stat classes of the drop-in differ from PySpark
> 4.3.0's:
> - The DataFrame members of a summary built as cls("<id>.summary") return a
> lazy DataFrame that fails at execution, where PySpark raises AttributeError
> '_summary_dataset'. A summary built by the user (cls() or cls("abc")) cannot
> be copied or pickled, and its _java_obj is None, where PySpark's copies and
> pickles as a plain object and keeps the given value. Pickling a fitted
> training summary or an evaluate() summary raises the session's
> AttributeError, where PySpark raises the TypeError of pickling a lock.
> Summary members have empty docstrings.
> - ChiSquareTest, Correlation, KolmogorovSmirnovTest, Summarizer and
> SummaryBuilder cannot be constructed, where PySpark's are plain classes;
> SummaryBuilder's signature and errors are JavaWrapper's.
> - pyspark.ml.stat defines an __all__ that PySpark's lacks.
> - StopWordsRemover fetches its server defaults twice: 4 requests at
> construction where PySpark sends 2, and 5 at a load where PySpark sends 3.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]