voonhous opened a new issue, #19823:
URL: https://github.com/apache/hudi/issues/19823
### Bug Description
`HoodieSchemaUtils.createDeleteLogSchema`
(`HoodieSchemas.createDeleteLogSchema` before #19809) resolves each ordering
field with `tableSchema.getField(name)`, an exact top-level lookup
(`HoodieSchema.java:1215`). A nested ordering field such as
`nested_record.level` is never a top-level name, so the call throws
`IllegalArgumentException: Ordering field nested_record.level not found in
table schema`.
Nested precombine/ordering fields are supported (HUDI-4051, #5517) and
`HoodieRecordUtils.getOrderingFieldNames` passes `hoodie.table.ordering.fields`
through verbatim, so on a table version 10 MOR table with `EVENT_TIME_ORDERING`
and a nested ordering field:
- write: `HoodieNativeLogAppendHandle.createLogWriterForAppend` ->
`HoodieNativeLogFormatWriter.appendDeleteRecord` -> `ensureDeleteFileWriter`
(`HoodieNativeLogFormatWriter.java:261`) throws on the first delete.
- read: `HoodieNativeLogDeleteBlock` (`:72`) and
`LsmFileIterators.createNativeDeleteLogIterator` (`:176`) throw for any native
delete log written under such a table config.
Present since #18987 / #19072; reachable by default since #19118 made native
logs the default for table version >= 10. Not a regression from #19809, which
moves the method unchanged. No in-repo test covers a nested ordering field on a
MOR table (`TestFileGroupReaderPartitionColumn.scala:192` uses
`nested_record.level` on COW only).
Minimal reproduction:
```java
HoodieSchema tableSchema = ...; // record with field nested_record: record {
level: long }
HoodieSchemaUtils.createDeleteLogSchema(tableSchema,
Collections.singletonList("nested_record.level"));
// IllegalArgumentException: Ordering field nested_record.level not found in
table schema
```
Fix direction: resolve the field via `HoodieSchema#getNestedField` and give
the delete-log column a flat name (Avro field names cannot contain `.`), then
align the lookups that use the dotted name against the flat delete record:
`HoodieNativeLogDeleteBlock.java:113,129` and
`LsmFileIterators.createNativeDeleteRecord` call `getOrderingValue(record,
deleteLogSchema, orderingFieldNames)`.
`HoodieNativeLogFormatWriter.createDeleteLogFieldValues` is positional and
needs no change.
### Environment
- Hudi master (`c59987a024cd`), table version 10, MOR, native log format
(default)
- `hoodie.record.merge.mode=EVENT_TIME_ORDERING` with a nested ordering
field, any engine
### Logs and Stack Trace
```
java.lang.IllegalArgumentException: Ordering field nested_record.level not
found in table schema
at
org.apache.hudi.common.schema.HoodieSchemaUtils.lambda$createDeleteLogSchema$...
at
org.apache.hudi.io.cdc.HoodieNativeLogFormatWriter.ensureDeleteFileWriter(HoodieNativeLogFormatWriter.java:261)
at
org.apache.hudi.io.cdc.HoodieNativeLogFormatWriter.appendDeleteRecord(HoodieNativeLogFormatWriter.java:169)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
