lokeshj1703 opened a new issue, #19559:
URL: https://github.com/apache/hudi/issues/19559
## Problem
When a Hudi 1.x reader reads a Metadata Table (MDT) that was written by an
older Hudi release, merging log-file delta records against base-file records
throws an `ArrayIndexOutOfBoundsException`:
```
java.lang.ArrayIndexOutOfBoundsException: Index 6 out of bounds for length 6
at org.apache.avro.generic.GenericData$Record.put(GenericData.java:275)
at
org.apache.hudi.metadata.HoodieMetadataPayload.getInsertValue(HoodieMetadataPayload.java)
at
org.apache.hudi.metadata.HoodieMetadataPayload.combineAndGetUpdateValue(HoodieMetadataPayload.java)
at
org.apache.hudi.common.model.HoodieAvroRecordMerger.merge(HoodieAvroRecordMerger.java)
```
## Root Cause
`SecondaryIndexMetadata` was added to `HoodieMetadataRecord` in a later Hudi
release, growing the schema from 6 to 7 fields. MDT log blocks written by older
releases carry the 6-field writer schema in their block header.
When merging, `HoodieAvroRecordMerger` retrieves the writer schema from the
buffered record (`getSchemaFromBufferRecord`) and passes it to
`HoodieMetadataPayload.getInsertValue(schema)`. Inside `getInsertValue`, the
fast path uses an object-identity check:
```java
if (schema == null || schema == HOODIE_METADATA_AVRO_SCHEMA) {
// safe path – return HoodieMetadataRecord directly
} else {
// assumes HOODIE_META_COLUMNS are prepended; uses KEY_FIELD_OFFSET=5,
TYPE_FIELD_OFFSET=6
record.put(TYPE_FIELD_OFFSET, type); // index 6 on a 6-field schema →
AIOOBE
}
```
The old 6-field schema is a different `Schema` object from
`HOODIE_METADATA_AVRO_SCHEMA` (7 fields), so it fails the identity check and
falls into the `else` branch, which is designed for schemas with
`HOODIE_META_COLUMNS` prepended. Since the old schema has only 6 fields
(indices 0–5), writing to index 6 throws `ArrayIndexOutOfBoundsException`.
## Fix
Add a helper `isHoodieMetadataRecordSchema(Schema)` that recognises any
version of the `HoodieMetadataRecord` schema by Avro record name and namespace.
Route these older-version schemas to the same fast path as the current schema,
returning a `HoodieMetadataRecord` with the current 7-field schema (missing
`SecondaryIndexMetadata` is `null`).
This preserves backward compatibility: tables written by older Hudi readers
can be read safely by a 1.x reader without requiring a table upgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
