Hyukjin Kwon created SPARK-61627:
------------------------------------

             Summary: Let copy, deepcopy and pickle copy GroupedData, the 
readers, the writers, the catalog and the other DataFrame helpers as PySpark 
does in the Python client
                 Key: SPARK-61627
                 URL: https://issues.apache.org/jira/browse/SPARK-61627
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


In the Python client, copy.copy, copy.deepcopy and pickle raise 'TypeError: 
cannot pickle ...' for GroupedData (and the result of pivot), 
DataFrameNaFunctions, DataFrameStatFunctions, DataFrameReader, DataFrameWriter, 
DataFrameWriterV2, Catalog and RuntimeConf, where PySpark 4.3.0 copies them, as 
plain Python objects. Copy them as PySpark does: a deep copy deep-copies the 
DataFrame or session that they hold through the memo, so that 
copy.deepcopy({'df': df, 'g': df.groupBy('id')}) keeps the copied g on the 
copied df; they pickle where PySpark's pickle; and copy.deepcopy(spark.conf) 
fails as PySpark's does.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to