Cq-study opened a new pull request, #680: URL: https://github.com/apache/doris-flink-connector/pull/680
# Proposed changes Issue Number: close #644 ## Problem Summary: When `sink.enable-delete=false`, CDC delete events are not filtered out. They are serialized with `__DORIS_DELETE_SIGN=0` (the before-image row) and sent to Doris, which effectively upserts the old row values. This causes useless serialization, network transfer and stream load processing on the Doris FE/BE side. ### What is changed In `JsonDebeziumDataChange#serialize`, when the op is `d` and delete is disabled, return `null` before serialization so the event is dropped early. `DorisWriter#writeOneDorisRecord` and `DorisBatchWriter#writeOneDorisRecord` already treat a null `DorisRecord` as a no-op, so no other change is needed. Add a unit test `testSerializeDeleteSkippedWhenDeleteDisabled` asserting that no `__DORIS_DELETE_SIGN=0` record is produced for delete events when delete is disabled. ### Why - Reduce network traffic and Doris FE/BE load for CDC pipelines that intentionally ignore deletes. - Avoid re-upserting the before-image of deleted rows, which is pointless work. ## Checklist(Required) 1. Does it affect the original behavior: Yes (only when `sink.enable-delete=false`: delete events are now dropped instead of being written back as `__DORIS_DELETE_SIGN=0` upserts of the before-image; the intent of disabling delete is preserved) 2. Has unit tests been added: Yes 3. Has document been added or modified: No 4. Does it need to update dependencies: No 5. Are there any changes that cannot be rolled back: No ## Further comments The change is scoped to the Debezium JSON CDC path (`JsonDebeziumDataChange`). The Table/SQL `RowDataSerializer` path is untouched. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
