tritsystem opened a new pull request, #51167:
URL: https://github.com/apache/arrow/pull/51167
### Rationale for this change
`pd.read_parquet()` (and `Table.to_pandas()` generally, with no
`types_mapper`/`dtype_backend="pyarrow"`) raises `TypeError` for any
column whose pandas metadata records a complex `ArrowDtype` -- list,
struct, or dictionary -- as its `numpy_type`, e.g.
`"list<item: string>[pyarrow]"`. This has been broken since the metadata
round-tripping was introduced, and is the long-standing root cause of
pandas-dev/pandas#53011 (open since 2023, still reproducing today with
no workaround at the pandas level).
```python
import pandas as pd
import pyarrow as pa
data = pd.DataFrame({
"a": pd.Series([["x"], ["x", "y"]],
dtype=pd.ArrowDtype(pa.list_(pa.string()))),
})
data.to_parquet("data.parquet") # SUCCESS
pd.read_parquet("data.parquet") # TypeError: data type 'list<item:
string>[pyarrow]' not understood
```
#39914 (this PR's linked issue) previously fixed the case where an
explicit `types_mapper` / `dtype_backend="pyarrow"` is passed at read
time -- #44720 resolved those columns earlier in `_get_extension_dtypes`,
so the buggy string-parsing branch is skipped entirely for that path.
The **default backend**, and any metadata-only path with no
`types_mapper`, still hit the original crash -- which is why the pandas
issue has stayed open and users keep re-discovering it (see the thread's
comments through 2025-08).
### What changes are included in this PR?
In `_get_extension_dtypes()`, when `pandas_dtype(dtype)` raises
`TypeError` on the metadata's `numpy_type` string, fall back to building
the `ArrowDtype` directly from the schema's actual field type
(`table.schema.field(name).type`) instead of propagating the error. The
schema always carries the precise type regardless of whether the
metadata string happens to be parseable, and this is exactly the pattern
several commenters on the pandas issue independently proposed (e.g.
[this
comment](https://github.com/pandas-dev/pandas/issues/53011#issuecomment-1548942282))
but that was never implemented in a merged fix.
### Are these changes tested?
Added `test_to_pandas_extension_dtypes_mapping_complex_type_no_types_mapper`,
covering both a list and a struct `ArrowDtype` column round-tripped
through `Table.to_pandas()` with **no** `types_mapper` -- the exact case
that previously crashed. Confirmed red on unpatched code / green after
the fix.
Also manually verified (not just the new test) against pandas 3.0.4 +
pyarrow 24.0.0:
- list, struct, and dictionary `ArrowDtype` columns round-trip through
both the default backend and `dtype_backend="pyarrow"`.
- A double write/read/write/read round-trip of a struct column (the
scenario from [this
comment](https://github.com/pandas-dev/pandas/issues/53011#issuecomment-1758894825))
still works.
- Plain numpy dtypes and simple extension dtypes (`Int64`) are
unaffected -- no regression to the already-working paths.
### Are there any user-facing changes?
**This PR contains a "Critical Fix".**
This makes `pd.read_parquet(...)` (default backend, no
`dtype_backend="pyarrow"` needed) work for any DataFrame containing a
complex `ArrowDtype` column (list/struct/dictionary), which currently
raises an opaque `TypeError` and has no workaround short of stripping the
pandas metadata before writing.
* GitHub Issue: #39914
### AI Generation Disclosure
Generated with [Claude Code](https://claude.com/claude-code). The bug
was found by searching for confirmed, still-open, high-severity issues
across large open-source projects; root-caused and fixed by tracing the
actual `pandas_compat.py` source against the linked pandas issue and the
prior partial fix (#44720), then verified against a fresh install of the
latest pandas/pyarrow rather than assumed from the issue thread alone.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
