Yicong-Huang opened a new pull request, #58260: URL: https://github.com/apache/spark/pull/58260
### What changes were proposed in this pull request? This PR simplifies the Arrow collect path in `PandasConversionMixin` (`python/pyspark/sql/pandas/conversion.py`) by consolidating how empty (zero-record) results are handled. - `toArrow()` now handles the empty-result case by building the table from `schema.empty_table()` (the `schema` it already computes), mirroring what `toPandas()` already does with `arrow_schema.empty_table()`. Both call sites now share the same shape: `pa.Table.from_batches(batches)` when there are records, `schema.empty_table()` otherwise. - `_collect_as_arrow()` drops its `empty_list_if_zero_records` parameter and the dead `else` branch that fabricated a single empty `RecordBatch` from a separately-built schema. The method now always returns the collected list (empty when there are no records) and ends in a single `return batches`. - The `prefers_large_var_types` parameter of `_collect_as_arrow()` is removed as well. It was introduced in SPARK-54300/SPARK-54396 solely to build the schema for that empty-batch branch; with the branch gone it is dead, so the two call sites no longer pass it (each still computes its own schema locally). ### Why are the changes needed? The empty-result handling was split across two methods: `toArrow()` computed the full Arrow schema (with `error_on_duplicated_field_names_in_struct=True`, `timezone="UTC"`, and large-var-types), while `_collect_as_arrow()` separately rebuilt a schema without `error_on_duplicated_field_names_in_struct=True` just to fabricate an empty batch. `toArrow()` already knows the schema, so the empty case belongs there; moving it removes the duplicated/inconsistent schema construction and a rarely-exercised code path, and lets `_collect_as_arrow()` become a straightforward collect helper. ### Does this PR introduce _any_ user-facing change? No. `toArrow()` and `toPandas()` return the same empty table with the same schema as before; only the internal code path changes. ### How was this patch tested? Existing Arrow tests in `python/pyspark/sql/tests/arrow/test_arrow.py` cover the empty-result paths (`test_toArrow_empty_rows`, `test_toArrow_empty_columns`, `test_toPandas_empty_*`) and the direct `_collect_as_arrow` self-destruct call. Extended `test_toArrow_duplicate_field_names` to assert that an empty result (`df.limit(0).toArrow()`) still rejects duplicated struct field names, keeping empty and non-empty behavior consistent. ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
