This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 7ca3cc3d7e0d [SPARK-57997][PYTHON][TEST] Fix ts_unit assumption in
test_pyarrow_array_type_inference for pandas 3
7ca3cc3d7e0d is described below
commit 7ca3cc3d7e0d357f158fbb1ec95cafe49f3bf3c0
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Wed Jul 8 09:02:43 2026 +0900
[SPARK-57997][PYTHON][TEST] Fix ts_unit assumption in
test_pyarrow_array_type_inference for pandas 3
### What changes were proposed in this pull request?
Fix an incorrect pandas-3 resolution assumption in
`test_pandas_series_numpy_backed` (in
`test_pyarrow_array_type_inference.py`).
The test used a single `ts_unit = "us" if pandas >= 3 else "ns"` for all
numpy-backed temporal cases. Empirically, pandas 3 only produces microsecond
resolution for parse/construct/tz-aware inputs (`pd.to_datetime`,
`pd.Timestamp("...")`, `pd.to_timedelta`, tz-aware Series). The boundary
constants `pd.Timestamp.min/max` and `pd.Timedelta.min/max`, and the
`pd.Timedelta(0)` scalar, remain nanosecond-backed in both pandas 2 and 3,
because those values only fit in nanosecond resolution.
This PR pins those boundary/scalar cases to `"ns"` and keeps `ts_unit` for
the remaining cases, and clarifies the comment.
### Why are the changes needed?
The scheduled `Build / Python-only (Python 3.12, Pandas 3)` job fails on
master:
```
File ".../test_pyarrow_array_type_inference.py", line 348, in
test_pandas_series_numpy_backed
self._run_inference_tests(cases)
AssertionError: TimestampType(timestamp[ns]) != TimestampType(timestamp[us])
```
The test loop stops at the first mismatch (`pd.Timestamp.min`), which is why
only one failure surfaced. The wrong branch was never exercised until the
Pandas-3 job ran.
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
Ran the suite in two local conda environments (with `SPARK_TESTING=1`):
- pandas 3.0.2 / pyarrow 23.0.1: 10/10 pass
- pandas 2.3.3: 10/10 pass (no regression)
`ruff format --check` and `ruff check` both pass on the changed file.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
Closes #57076 from zhengruifeng/SPARK-pyarrow-tsunit-dev6.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../upstream/pyarrow/test_pyarrow_array_type_inference.py | 15 +++++++++------
1 file changed, 9 insertions(+), 6 deletions(-)
diff --git
a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_type_inference.py
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_type_inference.py
index 169ddd2c55f4..dafb08646abb 100644
--- a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_type_inference.py
+++ b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_type_inference.py
@@ -299,7 +299,10 @@ class PyArrowArrayTypeInferenceTests(unittest.TestCase):
# pandas >= 3 infers large_string instead of string for object-dtype
string Series
string_type = pa.large_string() if LooseVersion(pd.__version__) >=
"3.0.0" else pa.string()
- # pandas >= 3 defaults to microsecond resolution instead of nanosecond
+ # pandas >= 3 parses/constructs datetime and timedelta at microsecond
resolution
+ # instead of nanosecond (e.g. to_datetime, Timestamp(str),
to_timedelta, tz-aware).
+ # The min/max boundary constants and the Timedelta(0) scalar stay
nanosecond-backed
+ # in both pandas 2 and 3, since those values only fit in nanosecond
resolution.
ts_unit = "us" if LooseVersion(pd.__version__) >= "3.0.0" else "ns"
sg = ZoneInfo("Asia/Singapore")
@@ -328,12 +331,12 @@ class PyArrowArrayTypeInferenceTests(unittest.TestCase):
(pd.Series([date1, date2]), pa.date32()),
(pd.Series(pd.to_datetime(["2024-01-01", "2024-01-02"])),
pa.timestamp(ts_unit)),
(pd.Series([pd.Timestamp("1970-01-01")]), pa.timestamp(ts_unit)),
- (pd.Series([pd.Timestamp.min]), pa.timestamp(ts_unit)),
- (pd.Series([pd.Timestamp.max]), pa.timestamp(ts_unit)),
+ (pd.Series([pd.Timestamp.min]), pa.timestamp("ns")),
+ (pd.Series([pd.Timestamp.max]), pa.timestamp("ns")),
(pd.Series(pd.to_timedelta(["1 day", "2 hours"])),
pa.duration(ts_unit)),
- (pd.Series([pd.Timedelta(0)]), pa.duration(ts_unit)),
- (pd.Series([pd.Timedelta.min]), pa.duration(ts_unit)),
- (pd.Series([pd.Timedelta.max]), pa.duration(ts_unit)),
+ (pd.Series([pd.Timedelta(0)]), pa.duration("ns")),
+ (pd.Series([pd.Timedelta.min]), pa.duration("ns")),
+ (pd.Series([pd.Timedelta.max]), pa.duration("ns")),
# Timezone-aware
(pd.Series([dt1_sg, dt2_sg]), pa.timestamp(ts_unit,
tz="Asia/Singapore")),
(pd.Series([ts1_la, ts2_la]), pa.timestamp(ts_unit, tz=la)),
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
