[
https://issues.apache.org/jira/browse/CAMEL-26117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on CAMEL-26117 started by Claus Ibsen.
-------------------------------------------
> camel-debezium - with the ExtractNewRecordState SMT the body and the
> operation header are always null, and mongodb updates have a null body
> -------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-26117
> URL: https://issues.apache.org/jira/browse/CAMEL-26117
> Project: Camel
> Issue Type: Bug
> Components: camel-debezium
> Reporter: Claus Ibsen
> Assignee: Claus Ibsen
> Priority: Major
> Fix For: 4.24.0
>
>
> Found while adding the catalog body types of the six debezium components
> (db2, mongodb, mysql, oracle, postgres, sqlserver) (CAMEL-24961, PR #28418).
> h4. 1. With the ExtractNewRecordState / ExtractNewDocumentState SMT the body
> is always null
> With {{additionalProperties.transforms=unwrap}} and the Debezium
> ExtractNewRecordState (or ExtractNewDocumentState for mongodb) SMT, the
> record value is the flattened row Struct, which has no {{after}} or {{op}}
> field. {{DebeziumEndpoint.extractValueFromValueStruct}} (main 2a465aa4814c,
> line 138) catches the {{DataException}} and returns null, so the body
> ({{extractBodyValueFromValueStruct}}, line 84) and the
> {{CamelDebeziumOperation}} header are null and the row is lost. This is in
> the shared base code, so all six connectors have it. Checked on
> 4.24.0-SNAPSHOT with postgres 17 (pgoutput) via Testcontainers.
> Proposed: when the value Struct is not a Debezium envelope (no {{after}}
> field in its schema), use the Struct itself as the body.
> h4. 2. mongodb: an update has a null body with captureMode=change_streams
> {{DebeziumMongodbEndpoint.extractBodyValueFromValueStruct}} (main
> 2a465aa4814c, line 89) falls back to the {{patch}} field when {{after}} is
> null, but Debezium 2 and later has no {{patch}} field, so this fallback is
> dead code. With {{captureMode=change_streams}} (no full document lookup) an
> update event has a null body, and the {{updateDescription}} field of the
> event (the changed fields) is not exposed in a header either. Checked on
> 4.24.0-SNAPSHOT with a mongo 7 replica set via Testcontainers.
> Proposed: drop the {{patch}} fallback and expose {{updateDescription}} (as
> the body when there is no {{after}}, or in a header).
> h4. 3. DebeziumTypeConverter.toMap javadoc is wrong, and Structs inside lists
> are not converted
> The javadoc of {{DebeziumTypeConverter.toMap}} (main 2a465aa4814c, line 32)
> says it only works with flat fields and does not handle nested structures,
> but the method recurses into nested Structs (line 45). It does not convert
> Structs inside a list or an array, so those stay Kafka Connect Structs in the
> Map. Read in the code, not run.
> Proposed: fix the javadoc and also convert the Structs held in lists and
> arrays.
> _Claude Code on behalf of davsclaus_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
