jrbourbeau opened a new issue, #15079:
URL: https://github.com/apache/arrow/issues/15079
### Describe the bug, including details regarding any error messages,
version, and platform.
While adding support in Dask for the upcoming `pandas=2.0` release, some of
our unit tests began failing due to `pyarrow` no longer round-tripping
`datetime64` data in the same way `pandas` does (or will in the `pandas=2.0`
release).
Here's a minimal reproducer:
```python
import pandas as pd
import pyarrow as pa
df = pd.DataFrame({"x": [3, 2, 1]}).astype("M8[us]")
print(f"{df.x.dtypes = }")
# Roundtrip through pyarrow.Table
t = pa.Table.from_pandas(df)
df2 = t.to_pandas()
print(f"{df2.x.dtypes = }")
pd.testing.assert_frame_equal(df, df2)
```
which output
```
df.x.dtypes = dtype('<M8[us]')
df2.x.dtypes = dtype('<M8[ns]')
Traceback (most recent call last):
File "/Users/james/projects/dask/dask/test-roundtrip.py", line 11, in
<module>
pd.testing.assert_frame_equal(df, df2)
File
"/Users/james/mambaforge/envs/dask/lib/python3.10/site-packages/pandas/_testing/asserters.py",
line 1187, in assert_frame_equal
assert_series_equal(
File
"/Users/james/mambaforge/envs/dask/lib/python3.10/site-packages/pandas/_testing/asserters.py",
line 898, in assert_series_equal
assert_attr_equal("dtype", left, right, obj=f"Attributes of {obj}")
File
"/Users/james/mambaforge/envs/dask/lib/python3.10/site-packages/pandas/_testing/asserters.py",
line 408, in assert_attr_equal
raise_assert_detail(obj, msg, left_attr, right_attr)
File
"/Users/james/mambaforge/envs/dask/lib/python3.10/site-packages/pandas/_testing/asserters.py",
line 592, in raise_assert_detail
raise AssertionError(msg)
AssertionError: Attributes of DataFrame.iloc[:, 0] (column name="x") are
different
Attribute "dtype" are different
[left]: datetime64[us]
[right]: datetime64[ns]
```
when using the nightly version of `pandas` installed with
```
python -m pip install --no-deps --pre -i
https://pypi.anaconda.org/scipy-wheels-nightly/simple pandas
```
My guess is this is related to this upcoming change in `pandas`
https://pandas.pydata.org/docs/dev/whatsnew/v2.0.0.html#construction-with-datetime64-or-timedelta64-dtype-with-unsupported-resolution
(though that's just a guess).
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
