vinishjail97 opened a new issue, #19125:
URL: https://github.com/apache/hudi/issues/19125
**Describe the problem you faced**
Building the column-stats index on a table version 9 (timeline layout V2)
**Merge-on-Read** table crashes during the inline log-append upsert when the
col-stats column list contains a leaf nested inside a `MAP` or `ARRAY` (e.g.
`my_map.key_value.value`, `my_array.list.element`):
```
org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType
UPDATE for partition :0
...
Caused by: java.lang.IllegalStateException: Cannot get field from schema
type: MAP
at
org.apache.hudi.common.schema.HoodieSchema.getField(HoodieSchema.java:1172)
at
org.apache.hudi.avro.AvroRecordContext.getFieldValueFromIndexedRecord(AvroRecordContext.java:90)
at
org.apache.hudi.common.model.HoodieAvroIndexedRecord.getColumnValueAsJava(HoodieAvroIndexedRecord.java:193)
at
org.apache.hudi.metadata.HoodieTableMetadataUtil.collectColumnRangeFieldValueV2(HoodieTableMetadataUtil.java:354)
at
org.apache.hudi.metadata.HoodieTableMetadataUtil.collectColumnRangeMetadata(HoodieTableMetadataUtil.java:296)
at
org.apache.hudi.io.HoodieInlineLogAppendHandle.collectColumnStats(HoodieInlineLogAppendHandle.java:182)
at
org.apache.hudi.io.HoodieAppendHandle.processAppendResult(HoodieAppendHandle.java:454)
```
**Root cause**
#17694 added support for column stats on primitives nested inside
`MAP`/`ARRAY` by teaching the **schema-side** navigator
(`HoodieSchema.getNestedField`) and the base-file (Parquet) path to resolve the
Parquet-style synthetic accessors `.key_value.key`, `.key_value.value` and
`.list.element`. Such leaves therefore pass `isColumnTypeSupported` (the
resolved leaf is a scalar).
The **column-stats V2 record path** used by MOR inline log-append
(`HoodieTableMetadataUtil.collectColumnRangeFieldValueV2` ->
`HoodieRecord.getColumnValueAsJava` ->
`AvroRecordContext.getFieldValueFromIndexedRecord`) was never taught the same
navigation. It splits the field path on `.` and assumes every segment is a
`RECORD` field, so when it reaches the `MAP`/`ARRAY` schema it calls
`HoodieSchema.getField(...)` on it, which throws `IllegalStateException: Cannot
get field from schema type: MAP`.
The V1 path (table version 6) does not hit this:
`HoodieAvroUtils.getNestedFieldVal` returns `null` when a path segment is not a
record, so version 6 tolerates these column-stats columns (they simply
contribute nothing from the record path, and get their statistics from the
Parquet base-file footer). Only the V2 record navigator was left asymmetric.
**To Reproduce**
1. Create a MOR table at table version 9 with column stats enabled and a
col-stats column list that includes a map/array leaf, e.g.
`my_map.key_value.value` or `my_array.list.element`.
2. Insert some records, then upsert records into the same file group (forces
a MOR log-append update).
3. The upsert fails with `HoodieUpsertException` caused by
`IllegalStateException: Cannot get field from schema type: MAP`.
This surfaces in Apache XTable when a MOR source table at table version 9
has column stats configured on map/array leaves.
**Expected behavior**
Collecting column stats on a MOR table at version 9 should succeed for
map/array-nested leaf columns, matching version 6 behavior. The record-path
value navigator should degrade to `null` for a path it cannot resolve to a
single value (statistics for such leaves are still collected from the
base-file/Parquet footer path, which is the coverage #17694 added), instead of
throwing.
**Environment Description**
* Hudi version : master (1.3.0-SNAPSHOT)
* Spark version : 3.5
* Table type : Merge-on-Read
* Table version : 9 (col-stats index V2)
**Additional context**
Fix mirrors `HoodieAvroUtils.getNestedFieldVal`:
`AvroRecordContext.getFieldValueFromIndexedRecord` returns `null` when a path
segment cannot be resolved as a plain `RECORD` field (a `MAP`/`ARRAY`
intermediate or a null intermediate) rather than calling `getField` on a
non-record schema. A map/array leaf is multi-valued per record and has no
single value to fold into a min/max, so `null` (no stats from the record path)
is the correct and safe result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
