Spenserrrr opened a new pull request, #57939:
URL: https://github.com/apache/spark/pull/57939
### What changes were proposed in this pull request?
Adds golden-file drift tests for `pa.Table.to_pandas()` under the
SPARK-54936 umbrella (monitoring upstream PyArrow / pandas behavior). New test
file `python/pyspark/tests/upstream/pyarrow/test_pyarrow_table_to_pandas.py`
with a shared testless base and two test classes:
- `PyArrowTableToPandasDefaultTests` — pins multi-column DataFrame assembly
and the empty-table edges (0-column and 0-row) under a bare `to_pandas()`.
- `PyArrowTableToPandasCoerceTemporalTests` — pins
`to_pandas(coerce_temporal_nanoseconds=True)` with `date_as_object` at its
default (True) and with `date_as_object=False`, plus timestamp and duration
overflow rows.
Because `Table.to_pandas()` returns a `DataFrame` (whereas
`Array`/`ChunkedArray.to_pandas()` return a `Series`, already covered by
`test_pyarrow_arrow_to_pandas_{default,non_default}.py`), two small helpers
were added to `python/pyspark/testing/goldenutils.py`:
- a `pa.Table` branch in `repr_value` (`repr_arrow_table_value`) to render
the input-table anchor column;
- `repr_pandas_value` now formats a DataFrame per-column via `tolist()`
instead of `to_json()` — this is overflow-safe for out-of-range datetimes (e.g.
a year-9999 date) and more readable. No existing golden produced a DataFrame,
so no committed golden is affected by this change.
Table-specific coverage rationale: a Table's per-column conversion is
byte-identical to the Array path (verified across all 121 inventory types), so
these tests do not re-enumerate per-type conversion. They target what only a
Table exercises — multi-column DataFrame assembly, the empty-table edges, and
temporal coercion in an assembly context. Spark reaches `Table.to_pandas()` at
`python/pyspark/sql/pandas/conversion.py` (the 0-column early return) and, in
Spark Connect, at `python/pyspark/sql/connect/client/core.py` (a bare
whole-Table conversion of a SQL command result).
### Why are the changes needed?
Part of SPARK-54936, which pins upstream PyArrow / pandas conversion
behavior so that a library version bump changing it fails loudly in CI instead
of silently returning wrong data. `pa.Table.to_pandas` is the remaining
conversion in this family with no golden coverage. Writing the tests also
surfaced a real cross-version divergence worth pinning: on pandas 3 the Table
path converts string columns to the dedicated `str` dtype starting at PyArrow
19 (uniformly for empty, all-null, and populated columns), whereas the Array
path switches its empty string columns only at PyArrow 24.
### Does this PR introduce _any_ user-facing change?
No. This is test-only, plus test-infrastructure helpers in `goldenutils.py`.
### How was this patch tested?
New golden tests. The goldens were generated on pandas 2.3.3 / PyArrow
24.0.0 and validated across a PyArrow 18-25 x pandas 2/3 sweep (16
combinations), each in a fresh virtualenv running the committed test files
against the committed golden files. Version-legitimate differences are recorded
as `LooseVersion`-guarded entries in the `overrides` dict rather than by
regenerating. The sweep covers Linux/x86 only, so a platform-specific follow-up
(e.g. ARM) may be needed once CI runs the full matrix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
