skywalker0618 opened a new pull request, #20337:
URL: https://github.com/apache/hudi/pull/20337
### Describe the issue this Pull Request addresses
Reading a Hudi table through the Flink COW Parquet reader fails with a
`NullPointerException` when a column is an array of rows that itself contains
an array of rows, e.g. `ARRAY<ROW<name STRING, items ARRAY<ROW<id STRING>>>>`:
```
java.lang.NullPointerException
at
org.apache.hudi.table.format.cow.vector.reader.NestedColumnReader.readRow(NestedColumnReader.java:112)
```
This is a follow-up to the Dremel-style nested Parquet reader backport
(FLINK-35702, #18636 / #18700 / #18809). We hit it streaming a production COW
table with deeply nested schemas.
Root cause: in `ParquetSplitReaderUtil.createWritableColumnVector`, the
`ARRAY` branch passes the Parquet `LIST` group itself as the physical type of
the array's element. For an array of rows, the element `ROW` then looks its
fields up by name in the `LIST` wrapper instead of in the element group, finds
none, and fills every slot with a constant placeholder vector. That's harmless
for leaf fields, but the placeholder for an inner array has no child vector, so
`NestedColumnReader.readRow` dereferences null.
### Summary and Changelog
Arrays of rows that contain nested arrays of rows can now be read by the
Flink COW reader.
- `ParquetSplitReaderUtil` (all five Flink modules: 1.18, 1.19, 1.20, 2.0,
2.1): the `ARRAY` branch of `createWritableColumnVector` now locates the
element type using the same Parquet `LIST` backward-compatibility rules as
`getArrayElementColumn`, so the vector tree lines up with the `ParquetField`
tree:
- standard 3-level lists (`LIST` -> repeated group -> element),
- 2-level lists whose repeated group is named `array` or `<list
name>_tuple`,
- repeated groups with multiple fields and repeated primitives (the
repeated type is the element).
The depth passed down is incremented at each unwrapped level, matching the
`ParquetField` construction. Row fields are still resolved by name, so schema
evolution (fields missing from the file) keeps working.
- New `TestParquetSplitReaderUtil` in each Flink module. The tests write
real Parquet files with `ExampleParquetWriter` and read them back through
`ParquetColumnarRowSplitReader`:
- array of rows containing an array of rows, parameterized over the three
list encodings (3-level, 2-level `array`, 2-level `_tuple`), including empty
and null arrays,
- the same shape with row fields missing from the file (schema evolution),
- 3-level and 2-level arrays of primitives.
### Impact
Fixes a read failure for nested array-of-row schemas; no public API or
config changes, and no behavior change for schemas that were already readable.
### Risk Level
Low. The change is confined to how the element type is located when building
column vectors for arrays, and it reuses the rules `getArrayElementColumn`
already applies when building the reader tree. Verification:
- Without the fix, 6 of the 8 new tests fail with the `NullPointerException`
above (the two primitive-array tests pass either way).
- With the fix, all 8 pass on each of `-Dflink1.18`, `-Dflink1.19`,
`-Dflink1.20`, `-Dflink2.0`, `-Dflink2.1`.
- `checkstyle:check` passes for all five modules.
- The same change has been running in a production Flink streaming job
reading a ~10B-row COW table with these nested schemas.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
