Hyukjin Kwon created SPARK-61881:
------------------------------------
Summary: Pickle the functions of UDFs, UDTFs and the map and
group-map functions when the plan is built, as PySpark does
Key: SPARK-61881
URL: https://issues.apache.org/jira/browse/SPARK-61881
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
The drop-in cloudpickles the functions of UDFs, pandas UDFs, UDTFs and the map
and group-map functions (mapInPandas, mapInArrow, applyInPandas, applyInArrow,
cogroup, applyInPandasWithState, transformWithState) when they are called.
PySpark 4.3.0 pickles them each time it builds a plan (PythonUDF.to_plan,
PythonUDTF.to_plan). So a global that the function reads, changed after the UDF
is applied, is seen by PySpark's next action but not by the drop-in's, and an
unpicklable function fails at the call in the drop-in, where PySpark fails at
the first request (df.schema or collect) with PicklingError, or
UDTF_SERIALIZATION_ERROR for UDTFs, and sends nothing.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]