linliu-code opened a new pull request, #669:
URL: https://github.com/apache/hudi-rs/pull/669
**Stacked on #639–#668** — review only the last two commits.
## The bug
A parquet file with an `array<map>` column written by parquet-avro fails to
read:
```
Map cannot be repeated
```
**Both readers, and both file kinds** — base files and parquet log blocks.
The file is simply unreadable.
## Cause
The parquet-avro writer defaults to `write-old-list-structure=true`,
encoding the column as a 2-level legacy list whose element is a **REPEATED map
group**:
```
required group obj_ids (LIST) {
repeated group array (MAP) { // element IS the map, and
REPEATED
repeated group key_value (MAP_KEY_VALUE) { required key; required value;
}
}
}
```
parquet-rs walks the LIST node, takes the repeated child for the list
wrapper and the map for the element, then dispatches it to `visit_map` — which
rejects a REPEATED map **unconditionally**.
arrow-rs and arrow-cpp both reject this physical schema, so the encoding
genuinely is legacy. But **Hudi's own reader accepts it**, and tables already
written this way have to stay readable.
## Fix
The reject can't be avoided at the Arrow level, so the *parquet* schema is
rewritten before the Arrow build — the legacy repeated map becomes a synthetic
repeated `list` wrapper around a required map. The footer parse never trips the
reject; only the Arrow build does. Normalizing in between fixes every reader
built from that metadata.
**Nothing about the data changes.** Every leaf keeps its definition and
repetition levels and its DFS position, so the same bytes decode to the same
values and the original row groups are reused as they are.
**The rewrite itself was already in the tree**
(`schema/parquet_list_norm.rs`), ported with its own tests, and had no caller.
This wires it up.
## Two commits
1. **Base files** — `ParquetBaseFileReader::open_builder_with_size`. This is
shared code: the legacy reader, the v2 engine, file listing, schema resolution
and statistics all read base files through it, so this fixes the **current**
reader too.
2. **Parquet log blocks** — `Decoder::decode_parquet_record_content`, same
shape.
## Tests
One per path, both reading the real captured fixture — a genuine
Hudi-written block in the rejected encoding.
I verified the base-file test is non-vacuous by disabling the normalization:
```
normalization off → ParquetError(ArrowError("Map cannot be repeated"))
FAILED
normalization on → ok
```
Full workspace green: 1188 lib + 79 table-read + 39 datafusion + 21 + 12.
Clippy clean in the changed files.
## Not included
Row-filter pushdown on parquet log blocks — upstream also installs one
there, gated on the predicate touching only primary keys. That needs the filter
threaded through `LogFileReader` into `Decoder`, which has no such state today,
so it's a separate change.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
