Hyukjin Kwon created SPARK-61881:
------------------------------------

             Summary: Pickle the functions of UDFs, UDTFs and the map and 
group-map functions when the plan is built, as PySpark does
                 Key: SPARK-61881
                 URL: https://issues.apache.org/jira/browse/SPARK-61881
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


The drop-in cloudpickles the functions of UDFs, pandas UDFs, UDTFs and the map 
and group-map functions (mapInPandas, mapInArrow, applyInPandas, applyInArrow, 
cogroup, applyInPandasWithState, transformWithState) when they are called. 
PySpark 4.3.0 pickles them each time it builds a plan (PythonUDF.to_plan, 
PythonUDTF.to_plan). So a global that the function reads, changed after the UDF 
is applied, is seen by PySpark's next action but not by the drop-in's, and an 
unpicklable function fails at the call in the drop-in, where PySpark fails at 
the first request (df.schema or collect) with PicklingError, or 
UDTF_SERIALIZATION_ERROR for UDTFs, and sends nothing.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to