vinishjail97 opened a new pull request, #19126:
URL: https://github.com/apache/hudi/pull/19126
### Describe the issue this Pull Request addresses
Closes #19125
Building the column-stats index on a table version 9 (timeline layout V2)
Merge-on-Read table crashes during the inline log-append upsert when the
col-stats column list contains a leaf nested inside a `MAP` or `ARRAY` (e.g.
`my_map.key_value.value`, `my_array.list.element`):
```
Caused by: java.lang.IllegalStateException: Cannot get field from schema
type: MAP
at
org.apache.hudi.common.schema.HoodieSchema.getField(HoodieSchema.java:1172)
at
org.apache.hudi.avro.AvroRecordContext.getFieldValueFromIndexedRecord(AvroRecordContext.java:90)
at
org.apache.hudi.common.model.HoodieAvroIndexedRecord.getColumnValueAsJava(HoodieAvroIndexedRecord.java:193)
at
org.apache.hudi.metadata.HoodieTableMetadataUtil.collectColumnRangeFieldValueV2(HoodieTableMetadataUtil.java:354)
at
org.apache.hudi.io.HoodieInlineLogAppendHandle.collectColumnStats(HoodieInlineLogAppendHandle.java:182)
```
### Summary and Changelog
#17694 added column stats for primitives nested inside `MAP`/`ARRAY` by
teaching the schema-side navigator (`HoodieSchema.getNestedField`) and the
base-file (Parquet) path to resolve the synthetic accessors `.key_value.key`,
`.key_value.value` and `.list.element`. Such leaves therefore pass
`isColumnTypeSupported` (the resolved leaf is a scalar).
The column-stats V2 record path used by MOR inline log-append
(`HoodieTableMetadataUtil.collectColumnRangeFieldValueV2` ->
`HoodieRecord.getColumnValueAsJava` ->
`AvroRecordContext.getFieldValueFromIndexedRecord`) was never taught the same
navigation. It splits the field path on `.` and assumes every segment is a
`RECORD` field, so when it reaches the `MAP`/`ARRAY` schema it calls
`HoodieSchema.getField(...)` on it, which throws `IllegalStateException: Cannot
get field from schema type: MAP`.
The V1 path (table version 6) does not hit this:
`HoodieAvroUtils.getNestedFieldVal` returns `null` when a path segment is not a
record. Only the V2 record navigator was left asymmetric.
This PR makes `AvroRecordContext.getFieldValueFromIndexedRecord` return
`null` when a path segment cannot be resolved as a plain `RECORD` field,
instead of throwing:
- Return `null` when the current schema does not support fields (a
`MAP`/`ARRAY`/other non-record intermediate) or when the intermediate value is
`null`, mirroring `HoodieAvroUtils.getNestedFieldVal`.
- Guard the intermediate downcast so a non-record value (java `Map`/`List`)
degrades to `null` rather than throwing `ClassCastException`.
A map/array leaf is multi-valued per record and has no single value to fold
into a min/max, so returning `null` (no stats from the record path) is correct
and safe; statistics for such leaves are still collected from the base-file
(Parquet) footer path added in #17694.
### Impact
MOR tables at table version 9 with column stats configured on
map/array-nested leaves can be upserted (no longer crash on log-append). No
change for scalar or record-nested columns. Behavior now matches table version
6.
### Risk Level
low. The change is scoped to path segments that cannot be resolved as record
fields, which previously threw. Covered by new unit tests in
`TestAvroRecordContext` and validated end-to-end via Apache XTable's
`ITHudiConversionSource` (MOR source at table version 9 with column stats on
map/array leaves): the upsert that previously crashed with `Cannot get field
from schema type: MAP` now succeeds.
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
