Hyukjin Kwon created SPARK-61402:
------------------------------------

             Summary: Pickle VariantVal, Geometry and Geography as PySpark's 
plain classes in the Python client, so that the server's Python workers load 
them
                 Key: SPARK-61402
                 URL: https://issues.apache.org/jira/browse/SPARK-61402
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


SPARK-61291 made the Python client's VariantVal pickle its state as a tuple 
(value, metadata, __dict__) with a __setstate__ of its own. PySpark's 
VariantVal is a plain class without __setstate__, whose pickle BUILD takes a 
dict state only, so official PySpark cannot load a VariantVal pickled by the 
Python client: "_pickle.UnpicklingError: state is not a dictionary". The 
server's Python workers run official PySpark, so a UDF, pandas UDF, UDTF, 
foreachBatch or other function that refers to a VariantVal fails on the server; 
before SPARK-61291 it ran.

Geometry and Geography values of the Python client cannot be pickled, copied or 
subclassed at all ("cannot pickle 'pyspark.sql.types.Geometry' object", "not an 
acceptable base type"), so neither can a Row that holds one or a UDF that 
refers to one. PySpark's are plain classes, which do all of these.

Fix: pickle the three classes as PySpark pickles its plain classes, 
copyreg.__newobj__ with the dict of their attributes (and the slots of a 
subclass of slots, as object.__reduce_ex__ does), take that state and PySpark's 
pickles of them in __setstate__, and let Geometry and Geography be subclassed 
as VariantVal can.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to