lokeshj1703 opened a new pull request, #19348:
URL: https://github.com/apache/hudi/pull/19348
### Change Logs
For a table below table version 9 whose payload class is supplied **only via
the write config** (`hoodie.datasource.write.payload.class`) and is **not
persisted in the table properties**,
`HoodieTableConfig.getTableMergeProperties()` resolved the payload class from
the table config, so it never derived the Debezium/DMS delete markers
(`hoodie.payload.delete.field` / `hoodie.payload.delete.marker`) for the pre-v9
path.
As a result the FileGroupReader write path could not classify a
payload-level delete (e.g. a Postgres/MySQL Debezium
`_change_operation_type='d'` record) for a key **absent from the base file it
is merged into**. The record took the `previousRecord == null` branch of
`PayloadUpdateProcessor.handleNonDeletes`, which calls
`rewriteRecordWithNewSchema` and `.get()`s the `Option` the payload
deliberately emptied for a delete:
```
org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType
UPDATE for partition :0
Caused by: java.util.NoSuchElementException: No value present in Option
at org.apache.hudi.common.util.Option.get(Option.java:93)
at
org.apache.hudi.common.model.HoodieAvroRecord.rewriteRecordWithNewSchema(HoodieAvroRecord.java:178)
at
org.apache.hudi.common.table.read.UpdateProcessor$PayloadUpdateProcessor.handleNonDeletes(UpdateProcessor.java:146)
at
org.apache.hudi.io.FileGroupReaderBasedMergeHandle.doMerge(FileGroupReaderBasedMergeHandle.java:294)
```
**Fix:** `ConfigUtils.getMergeProps(props, tableConfig)` now resolves the
effective payload class from the reader/write `props` first (falling back to
the table config) and passes it into a new
`HoodieTableConfig.getTableMergeProperties(String payloadClass)` overload, so
the existing pre-v9 delete-marker derivation fires for these tables.
`hoodie.datasource.write.payload.class` is already present in the write props
on the FileGroupReader write path (`config.getProps()`), so no plumbing change
is needed at the call sites.
Behavior is unchanged for:
- tables that carry the payload class in the table config (resolution falls
back to it as before), and
- table version >= 9 (the pre-v9 block does not run).
### Impact
Fixes an upsert failure (`NoSuchElementException: No value present in
Option`) for Debezium/AWS-DMS CDC tables on the FileGroupReader write path when
the payload class is set only in the write config and the table has not been
upgraded to version 9. No config or format change.
### Risk level: low
Localized to the pre-v9 merge-property derivation; covered by a new
end-to-end test and existing `TestPayloadDeprecationFlow` / `TestConfigUtils`
coverage.
### Documentation Update
None.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added (end-to-end reproduction in
`TestPayloadDeprecationFlow`, covering the payload-class-absent case that
reproduces the crash and the payload-class-present control case)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
