nsivabalan commented on issue #14472: URL: https://github.com/apache/hudi/issues/14472#issuecomment-4795167300
This feature request appears to be addressed in current master. Two payload classes implement field-level merging across records with the same key: - `OverwriteNonDefaultsWithLatestAvroPayload` (added in [a1cff8a](https://github.com/apache/hudi/commit/a1cff8abae9d) / HUDI-1255, PR #2056) — picks the latest record by ordering, then merges non-default fields when writing to storage (`combineAndGetUpdateValue`/`mergeRecords`). - `PartialUpdateAvroPayload` (extends the above) — performs field-level merging directly in `preCombine` (`mergeOldRecord`/`mergeRecords`), so two same-key records in a batch are combined by picking each non-null field. Applying `PartialUpdateAvroPayload.preCombine` to the example in the issue: ``` before: (id=1, name=Karl, age=null, ts=0.0) + (id=1, name=null, age=18, ts=0.0) after: (id=1, name=Karl, age=18, ts=0.0) ``` If this matches the intent, the issue can likely be closed. If a different semantic (e.g., merging across more than two values, or treating zero-valued numerics as 'default' the way the original `OverwriteNonDefaultsWithLatestAvroPayload` does via `field.defaultVal()`) is needed, please clarify and we can keep it open. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
