rmahindra123 opened a new pull request, #19660:
URL: https://github.com/apache/hudi/pull/19660
### Change Logs
Debezium payloads order records with **two different mechanisms**:
- `preCombine` (intra-batch dedup) compares `orderingVal` — the value of the
**configured precombine field** captured at payload construction.
- `combineAndGetUpdateValue` (incoming vs stored record) drops the payload
`Properties` and always compares the **connector-hardcoded column**
(`_event_lsn` for Postgres, `_event_seq` for MySQL) via
`shouldPickCurrentRecord`. The 3-arg `combineAndGetUpdateValue(current, schema,
properties)` funnels into the props-less template, so `Properties` never
reaches the ordering decision.
The two agree only by convention (precombine set to the connector column).
Any table where precombine differs silently orders against storage by a column
the user never configured.
**Change:**
- `AbstractDebeziumAvroPayload` now extends `DefaultHoodieRecordPayload`
(which extends `OverwriteWithLatestAvroPayload`, so `instanceof` checks are
unaffected) and adds a `Properties`-aware `combineAndGetUpdateValue` template
(the 2-arg overload delegates with empty `Properties`) which dispatches the
ordering decision:
- ordering fields absent, **composite** (`length != 1` — multi-field
ordering, e.g. MySQL file+pos, is not supported), **or equal to the connector's
own column** → the connector-specific `shouldPickCurrentRecord` (unchanged,
original 3-arg signature);
- a single other configured field → the inherited
`DefaultHoodieRecordPayload.needUpdatingPersistedRecord`, called directly — no
duplicated comparison logic.
- Each subclass declares its column via a one-line
`getConnectorOrderingField()` (`_event_lsn` / `_event_seq`).
- The connector-column condition is what keeps MySQL correct with the common
configuration precombine = `_event_seq`: the seq's `"file.pos"` encoding needs
segment-wise numeric compare, and a plain `Comparable` compare is lexicographic
(`"2.11" > "10.111"` as strings).
- **Debezium semantics pinned against the new parent:**
`getInsertValue(schema, properties)` routes to the Debezium delete-op version
so `Default`'s event-time / `DELETE_KEY` variant cannot intercept; the
`preCombine` overrides (orderingVal compare, MySQL seq compare, delete
natural-order) are untouched; Postgres toast-column merging is unchanged and
now additionally covered by a test through the configured-ordering path.
Ties go to the incoming record in every path, matching the legacy strict
comparisons. Null persisted ordering value (e.g. bootstrapped rows) → incoming
wins. The connector path keeps the legacy `HoodieDebeziumAvroPayloadException`
when the insert record lacks its connector column (covered by
`testInvalidIncomingRecord`); the config-field path has
`DefaultHoodieRecordPayload`'s stock null semantics.
### Impact
| Table config | Before | After |
|---|---|---|
| precombine = connector column (common configuration) | connector ordering
| identical (numeric seq compare preserved for MySQL) |
| precombine = other field | `preCombine` by config field,
`combineAndGetUpdateValue` by connector column | both by config field |
| no ordering props passed | connector ordering | identical |
### Risk level (write none, low medium or high below)
low
Behavior-preserving for the common configuration (precombine = connector
column) and for callers that pass no ordering properties. Covered by 6 new unit
tests: config field overrides contradicting LSN/seq (incl. tie and
null-persisted cases), config = `_event_seq` keeps the numeric compare (proven
with unpadded seqs where lexicographic order inverts), no-config falls back to
the connector column, and toast-column merging through the configured-ordering
path. `TestPostgresDebeziumAvroPayload` 11/11, `TestMySqlDebeziumAvroPayload`
16/16, `TestDefaultHoodieRecordPayload` 23/23,
`TestOverwriteWithLatestAvroPayload` 2/2.
### Documentation Update
None — the payload honors `hoodie.payload.ordering.field` / precombine as
already documented for `DefaultHoodieRecordPayload`.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [ ] CI passed
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_016pZ8WDUGsW2bxfUbgjjUeR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
