cshuo opened a new issue, #20330: URL: https://github.com/apache/hudi/issues/20330
Follow-up to https://github.com/apache/hudi/pull/20288. Two paths still convert deletes with non-null payloads into `HoodieEmptyRecord`: - **Spark `BaseSparkInternalRecordContext.constructHoodieRecord()`** bypasses engine-specific ordering conversion. After precombine, string/decimal ordering values fail inline delete-block serialization with `UnsupportedOperationException` for `Spark3HoodieUTF8String` or Spark `Decimal`. - **Avro `AvroRecordContext.constructFinalHoodieRecord()`** drops delete payloads even when its output will participate in subsequent log-scanner merges. With `DefaultHoodieRecordPayload`, the resulting empty payload triggers the early return in `preCombine()`, skipping the ordering comparison and allowing an older insert to resurrect the delete. This is payload-dependent, not a general failure of the built-in event-time merger. Local component-level probes reproduced both behaviors. For Avro, merging `delete60` with `insert55` preserves the delete before final-record reconstruction, but selects the insert after reconstruction through `HoodieEmptyRecord`, using `HoodiePreCombineAvroRecordMerger`. Full write/read and log-scanner integration coverage is still needed. ### Proposed fix Construct `HoodieEmptyRecord` only when `isDelete() && isEmpty()`, preserving the engine record and delete flag otherwise. Avro's regular `constructHoodieRecord()` already follows this rule. Optionally centralize this rule in `RecordContext`, retaining engine-specific empty-record construction. Empty payloads can carry event-time ordering, so `HoodieEmptyRecord` should not be restricted to commit-time deletes. Keep the operation/ordering-preserving key-copy fix from the merged PR. ### Validation - Spark string/decimal delete ordering through precombine and MOR inline-log write/read. - Avro log scanning with a payload-bearing `delete60` followed by `insert55`, ensuring the delete survives. - Empty tombstones retain operation and ordering. Converting engine-native ordering values for genuinely empty tombstones during subsequent log writes remains a separate concern. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
