Spenserrrr opened a new pull request, #57999:
URL: https://github.com/apache/spark/pull/57999
### What changes were proposed in this pull request?
Follow-up to #57939 (SPARK-58720). That PR added `pa.Table.to_pandas`
golden-file tests with a local `_repr_dataframe` helper that formats a pandas
DataFrame result via per-column `tolist()`, deliberately bypassing
`GoldenFileTestMixin.repr_pandas_value` (which used `DataFrame.to_json`). This
PR promotes that formatter into the shared mixin so there is a single DataFrame
repr:
- `GoldenFileTestMixin.repr_pandas_value` now renders each column with
`tolist()` (byte-identical to the old local `_repr_dataframe`) instead of
`DataFrame.to_json`.
- `test_pyarrow_table_to_pandas.py` drops its local `_repr_dataframe` and
formats results through `repr_value(pdf, max_len=0)`.
- `test_pandas_udf_return_type`'s golden files (`pandas_2` and `pandas_3`)
are regenerated. It is the only other test that renders a pandas DataFrame
through `repr_value` — the `pd.DataFrame({"_1": [1, 2]})` column header. The
regeneration changes only that header; all coercion result cells are unchanged.
### Why are the changes needed?
`DataFrame.to_json` defaults to an epoch date format that serializes
temporal values to an int64 nanosecond count. That overflows on
out-of-nanosecond-range dates (e.g. year 9999 under the default
`date_as_object=True`, which returns an object column of Python
`datetime.date`) and misreads non-nanosecond units on pandas 2 (so the same
cell drifts between pandas 2 and 3). `tolist()` bypasses the epoch machinery
entirely, giving a stable, version-independent representation — exactly what a
drift-canary golden needs — and matches `repr_pandas_series_value`, which
already uses `tolist()` for the Series case. Keeping the formatter local to one
test file left the shared mixin with the fragile `to_json` behavior for any
future DataFrame consumer.
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
- `test_pyarrow_table_to_pandas` passes unchanged against its committed
golden files, confirming the switch to `repr_value` is byte-identical
(golden-inert).
- `test_pandas_udf_return_type` was regenerated and re-run in compare mode
under both pandas 2 and pandas 3; the golden diff is the single DataFrame
column header, with every data cell byte-identical.
- The full `python/pyspark/tests/upstream/pyarrow` suite and the
`python/pyspark/sql/tests/coercion` suite pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
