Yicong Huang created SPARK-58130:
------------------------------------
Summary: Refactor legacy SQL_ARROW_TABLE_UDF pandas path and
remove ArrowStreamPandasUDTFSerializer
Key: SPARK-58130
URL: https://issues.apache.org/jira/browse/SPARK-58130
Project: Spark
Issue Type: Sub-task
Components: PySpark
Affects Versions: 5.0.0
Reporter: Yicong Huang
Following SPARK-57394 which refactored the non-legacy SQL_ARROW_TABLE_UDF path
onto ArrowStreamSerializer, move the legacy pandas conversion logic out of
ArrowStreamPandasUDTFSerializer and into the read_udtf wrapper in worker.py, so
the legacy path also uses a plain ArrowStreamSerializer for I/O. The legacy
branch is rewritten flat to mirror the non-legacy path's structure
(verify_result / check_return_value / evaluate_rows / func). Once the
serializer has no remaining callers, it is removed. Behavior is unchanged: the
spark.sql.legacy.execution.pythonUDTF.pandas.conversion.enabled flag and all
legacy tests are retained.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
