Yicong-Huang opened a new pull request, #57273: URL: https://github.com/apache/spark/pull/57273
### What changes were proposed in this pull request? Following SPARK-57394, which refactored the non-legacy `SQL_ARROW_TABLE_UDF` path onto `ArrowStreamSerializer`, this PR does the same for the legacy pandas conversion path (`spark.sql.legacy.execution.pythonUDTF.pandas.conversion.enabled=true`): - The pandas (de)serialization is moved out of `ArrowStreamPandasUDTFSerializer` and into the `read_udtf` wrapper in `worker.py`, so the legacy path uses a plain `ArrowStreamSerializer` for I/O like the non-legacy path. Input Arrow batches are converted to pandas Series via `ArrowBatchTransformer.to_pandas`, and the output pandas `DataFrame` is converted back via `PandasToArrowConversion.convert(..., is_legacy=True)` -- the same parameters the serializer used, so behavior is unchanged. - The legacy branch is rewritten flat to mirror the non-legacy path's structure (`verify_result` / `check_return_value` / `evaluate_rows` / `func`), dropping the nested `wrap_arrow_udtf` factory. - With no remaining callers, `ArrowStreamPandasUDTFSerializer` is removed. The legacy flag and the parent serializers (`ArrowStreamPandasUDFSerializer` / `ArrowStreamPandasSerializer`) are retained. ### Why are the changes needed? This is part of the worker.py eval-type refactor (SPARK-55388) that consolidates each eval type's serialization onto `ArrowStreamSerializer` and keeps the (de)serialization logic self-contained in `read_udtf` / `read_udfs`. It removes the last caller of `ArrowStreamPandasUDTFSerializer` and aligns the legacy and non-legacy `SQL_ARROW_TABLE_UDF` paths structurally. ### Does this PR introduce _any_ user-facing change? No. The legacy path's behavior is unchanged; the conversion logic is only relocated. ### How was this patch tested? Existing tests. No behavior change. The legacy path is fully covered by `LegacyUDTFArrowTests` and `LegacyArrowUDTFParityTests` (flag=true), which pass unchanged, along with the non-legacy `UDTFArrowTests` / `ArrowUDTFParityTests`. ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
