Claus Ibsen created CAMEL-26117:
-----------------------------------
Summary: camel-debezium - with the ExtractNewRecordState SMT the
body and the operation header are always null, and mongodb updates have a null
body
Key: CAMEL-26117
URL: https://issues.apache.org/jira/browse/CAMEL-26117
Project: Camel
Issue Type: Bug
Components: camel-debezium
Reporter: Claus Ibsen
Fix For: 4.24.0
Found while adding the catalog body types of the six debezium components (db2,
mongodb, mysql, oracle, postgres, sqlserver) (CAMEL-24961, PR #28418).
h4. 1. With the ExtractNewRecordState / ExtractNewDocumentState SMT the body is
always null
With {{additionalProperties.transforms=unwrap}} and the Debezium
ExtractNewRecordState (or ExtractNewDocumentState for mongodb) SMT, the record
value is the flattened row Struct, which has no {{after}} or {{op}} field.
{{DebeziumEndpoint.extractValueFromValueStruct}} (main 2a465aa4814c, line 138)
catches the {{DataException}} and returns null, so the body
({{extractBodyValueFromValueStruct}}, line 84) and the
{{CamelDebeziumOperation}} header are null and the row is lost. This is in the
shared base code, so all six connectors have it. Checked on 4.24.0-SNAPSHOT
with postgres 17 (pgoutput) via Testcontainers.
Proposed: when the value Struct is not a Debezium envelope (no {{after}} field
in its schema), use the Struct itself as the body.
h4. 2. mongodb: an update has a null body with captureMode=change_streams
{{DebeziumMongodbEndpoint.extractBodyValueFromValueStruct}} (main 2a465aa4814c,
line 89) falls back to the {{patch}} field when {{after}} is null, but Debezium
2 and later has no {{patch}} field, so this fallback is dead code. With
{{captureMode=change_streams}} (no full document lookup) an update event has a
null body, and the {{updateDescription}} field of the event (the changed
fields) is not exposed in a header either. Checked on 4.24.0-SNAPSHOT with a
mongo 7 replica set via Testcontainers.
Proposed: drop the {{patch}} fallback and expose {{updateDescription}} (as the
body when there is no {{after}}, or in a header).
h4. 3. DebeziumTypeConverter.toMap javadoc is wrong, and Structs inside lists
are not converted
The javadoc of {{DebeziumTypeConverter.toMap}} (main 2a465aa4814c, line 32)
says it only works with flat fields and does not handle nested structures, but
the method recurses into nested Structs (line 45). It does not convert Structs
inside a list or an array, so those stay Kafka Connect Structs in the Map. Read
in the code, not run.
Proposed: fix the javadoc and also convert the Structs held in lists and arrays.
_Claude Code on behalf of davsclaus_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
