LeonYoah opened a new pull request, #13278: URL: https://github.com/apache/seatunnel/pull/13278
This change contributes to Seatunnel. ### Purpose of this pull request Close #10512. Oracle CDC does not deliver CLOB, NCLOB, BLOB, or XMLTYPE. Snapshot JDBC dropped those values. Incremental events either skipped them or, with `debezium.lob.enabled = true`, wrote `__debezium_unavailable_value` into the sink when the statement did not change the LOB column. This PR captures those types on the Oracle CDC source (snapshot + LogMiner). LogMiner LOB/XML handling is ported from Debezium 3.7.1.Final onto SeaTunnel’s Debezium 1.9.8 / Java 8 overlay. Overlay types keep the upstream names; SeaTunnel-only deltas are tagged `[SeaTunnel]`. - Snapshot reads CLOB / NCLOB / BLOB / XMLTYPE through JDBC. - Incremental events mine LOB/XML redo when `lob.enabled = true`. - An unchanged LOB on UPDATE, and LOB columns on DELETE / UPDATE_BEFORE, stay the unavailable placeholder unless `lob.reselect.enabled = true` looks the current value up by primary key (`AS OF SCN`). - `EMPTY_CLOB()` / `EMPTY_BLOB()` become empty values. SQL `NULL` stays null. This change does not modify the Oracle JDBC sink. `CREATE TABLE ... OF XMLTYPE` object tables are not supported. ### Does this PR introduce _any_ user-facing change? Yes. New Oracle-CDC options (defaults keep previous behavior): | Option | Default | Meaning | |---|---|---| | `lob.enabled` | false (omit leaves `debezium.lob.enabled` unchanged) | Capture CLOB, NCLOB, BLOB, and XMLTYPE in incremental events | | `lob.reselect.enabled` | false | On INSERT / UPDATE_AFTER, replace the unavailable placeholder with a primary-key lookup. Requires `lob.enabled = true` and a primary key | | `lob.unavailable-value.handling` | `warn_and_keep` | When a placeholder remains: `warn_and_keep` / `null` / `fail`. `null` and `fail` require `lob.enabled = true` | XMLTYPE is STRING. Copy `xdb` and `xmlparserv2` (same version as `ojdbc8`) next to the driver. Docs: `docs/en/connectors/source/Oracle-CDC.md` and `docs/zh/connectors/source/Oracle-CDC.md`. If XML parsing fails after those jars are added, start the engine with `-Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl`. ### How was this patch tested? Unit tests: `./mvnw -pl :connector-cdc-oracle test` → **105 run, 0 fail, 1 skip** (`OracleJdbcLobXeTest` needs `-Doracle.xe.it=true`). `./mvnw -pl :connector-cdc-base -Dtest=IncrementalSourceReaderTest test` → **19 run, 0 fail**. 19c LogMiner E2E used `seatunnelhub/oracle-19.3.0-ee:non-cdb` (Zeta), jobs `oraclecdc_to_oracle_lob.conf` / `oraclecdc_to_oracle_xml.conf` (`lob.enabled = true`, `lob.reselect.enabled = true`, `lob.unavailable-value.handling = "null"`). Source vs sink compared until equal. | Scenario | Phase | Type | Environment | Result | |---|---|---|---|---| | Snapshot CLOB / NCLOB / BLOB | snapshot | CLOB, NCLOB, BLOB | 19c EE E2E | Pass | | Varchar-only UPDATE (placeholder re-select) | streaming | CLOB / XMLTYPE | 19c EE E2E | Pass | | Streaming INSERT | streaming | CLOB, NCLOB, BLOB, XMLTYPE | 19c EE E2E | Pass | | Out-of-line CLOB (~5KB, `DBMS_LOB.WRITEAPPEND`) | streaming | CLOB | 19c EE E2E | Pass | | Emoji in CLOB / NCLOB | streaming | CLOB, NCLOB | 19c EE E2E | Pass | | Apostrophe in CLOB / NCLOB | streaming | CLOB, NCLOB | 19c EE E2E | Pass | | PNG BLOB | streaming | BLOB | 19c EE E2E | Pass | | SQL NULL CLOB | streaming | CLOB | 19c EE E2E | Pass | | `EMPTY_CLOB()` / `EMPTY_BLOB()` | streaming | CLOB, BLOB | 19c EE E2E | Pass | | DELETE | streaming | row delete | 19c EE E2E | Pass | | Snapshot XMLTYPE as STRING | snapshot | XMLTYPE | 19c EE E2E | Pass | | Streaming XML insert / update | streaming | XMLTYPE | 19c EE E2E | Pass | | Apostrophe in XML | streaming | XMLTYPE | 19c EE E2E | Pass | | NULL XMLTYPE | streaming | XMLTYPE | 19c EE E2E | Pass | | ~8KB XML (XML_BEGIN / XML_WRITE / XML_END) | streaming | XMLTYPE | 19c EE E2E | Pass | | XML_BEGIN / XML_WRITE assembly | unit | XMLTYPE redo | `XmlBeginParserTest`, `XmlWriteAssemblyTest` | Pass | | HEXTORAW decode | unit | BLOB / XML bytes | `TransactionCommitConsumerTest` | Pass | | Re-select SQL + flashback fallback | unit | CLOB / XMLTYPE | `OracleLobReselectSqlTest` | Pass | | Placeholder `warn_and_keep` / `null` / `fail` | unit | LOB handler | `OracleLobUnavailableValueHandlerTest` | Pass | | JDBC CLOB / BLOB / XMLTYPE converters | unit | snapshot JDBC | `OracleJdbcLobValueConvertersTest`, `OracleXmlTypeJdbcTest` | Pass | | Option validation (`lob.enabled` vs reselect / handling) | unit | config | `OracleIncrementalSourceFactoryTest` | Pass | | Reader close of AutoCloseable deserialize schema | unit | CDC base | `IncrementalSourceReaderTest` (19 tests) | Pass | | Snapshot CLOB / NCLOB / BLOB / XMLTYPE (emoji, PNG, MP4) | snapshot | JDBC converters | XE 11g (`gvenzl/oracle-xe:11-full`), `-Doracle.xe.it=true` | Pass | | Snapshot CLOB / NCLOB / BLOB / XMLTYPE (emoji, PNG, MP4) | snapshot | JDBC converters | XE 21c (`gvenzl/oracle-xe:21-slim-faststart`), `-Doracle.xe.it=true` | Pass | Default `mvn test` skips the two XE rows so CI does not pull Oracle XE. ### Check list * [x] New jars: `xdb` / `xmlparserv2` are `provided` and are not bundled * [x] Docs updated (`docs/en`, `docs/zh`) * [x] `incompatible-changes.md` (defaults unchanged) * [x] Existing connector: plugin-mapping / dist / labeler / plugin_config not required -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
