Hyukjin Kwon created SPARK-61627:
------------------------------------
Summary: Let copy, deepcopy and pickle copy GroupedData, the
readers, the writers, the catalog and the other DataFrame helpers as PySpark
does in the Python client
Key: SPARK-61627
URL: https://issues.apache.org/jira/browse/SPARK-61627
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In the Python client, copy.copy, copy.deepcopy and pickle raise 'TypeError:
cannot pickle ...' for GroupedData (and the result of pivot),
DataFrameNaFunctions, DataFrameStatFunctions, DataFrameReader, DataFrameWriter,
DataFrameWriterV2, Catalog and RuntimeConf, where PySpark 4.3.0 copies them, as
plain Python objects. Copy them as PySpark does: a deep copy deep-copies the
DataFrame or session that they hold through the memo, so that
copy.deepcopy({'df': df, 'g': df.groupBy('id')}) keeps the copied g on the
copied df; they pickle where PySpark's pickle; and copy.deepcopy(spark.conf)
fails as PySpark's does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]