Thanatos9404 opened a new pull request, #41201:
URL: https://github.com/apache/beam/pull/41201
Fixes #41100
`ReadFromParquet(as_rows=True)` declares struct columns as nested rows in
its element type, but `PyarrowBatchConverter.explode_batch` built each element
from `pa.Scalar.as_py()`, which returns structs as dicts. So `row.address.city`
(or a Beam YAML `MapToFields` expression on a struct field) failed with
`AttributeError: 'dict' object has no attribute 'city'`.
`explode_batch` now builds the declared row type for struct values,
including rows in optional fields and in lists.
`PyarrowArrayBatchConverter.explode_batch` had the same problem for row
elements and gets the same fix. Each converter works out once which fields need
converting, so rows without nested structs take the same path as before.
A struct read with only some of its columns (`columns=['address.city']`)
gets `None` for the fields that were not read.
Behaviour change: nested struct values are rows now, so code that worked
around this by indexing them as dicts (`row.address['city']`) has to use
attribute access. The `CHANGES.md` entry says so.
Not changed here:
- Map values stay as `as_py()` returns them.
- Writing nested `beam.Row` values (`produce_batch`) is #41101. Nested rows
that Beam generates are NamedTuples, which pyarrow already accepts. I checked a
`ReadFromParquet(as_rows=True)` -> `Filter` -> `WriteToParquet` pipeline with a
struct column and a list of structs, and it writes the same data back.
Tests:
- New explode tests in `arrow_type_compatibility_test.py`, for:
- schema-generated row types;
- user NamedTuples;
- a `pa.Array` batch.
- One new test in `parquetio_test.py`. It reads a Parquet file with a struct
column and a list of structs, and also does a partial struct read.
- These four fail without the change and pass with it.
- I also added a list-of-structs entry to the parameterized converter tests
for round-trip coverage.
- `arrow_type_compatibility_test.py` and `batch_test.py` pass, along with
the schema'd tests in `parquetio_test.py`.
- yapf, ruff and pyrefly are clean.
I also added a `CHANGES.md` entry under Bugfixes for 2.79.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
