LeonYoah opened a new pull request, #12736:
URL: https://github.com/apache/seatunnel/pull/12736
### Purpose of this pull request
Close #10512.
Oracle CDC loses or corrupts large-object data. Snapshot drops
CLOB/NCLOB/BLOB values, streaming LOB inserts arrive as null, unchanged LOB
columns reach the sink as `__debezium_unavailable_value`, non-BMP characters
(emoji) are dropped while LOB fragments are rebuilt, and XMLTYPE columns are
not captured. This PR fixes those paths end to end (source and Oracle JDBC
sink) in 5 commits:
1. **[Fix][CDC-Base] Fix TIMESTAMP WITH TIME ZONE offset parsing**:
`TemporalConversions` parses Debezium timestamps that carry a numeric offset
(`ISO_OFFSET_DATE_TIME`), not only `Z`. Tests added.
2. **[Fix][Connector-V2][JDBC] Fix Oracle sink LOB writing**:
`OracleJdbcRowConverter` uses `createClob`/`createNClob`/`createBlob` for long
values, so ojdbc no longer splits surrogate pairs at chunk boundaries. It also
binds empty LOBs as empty locators instead of NULL. New `StatementBoundLobs`
and freeing in `FieldNamedPreparedStatement`. Tests added.
3. **[Fix][Connector-V2][Oracle-CDC] Read real LOB values in snapshot
phase**: `OracleJdbcLobValueConverters` materializes JDBC Clob/Blob values
during snapshot regardless of `lob.enabled`. Tests added.
4. **[Feature][Connector-V2][Oracle-CDC] Support LOB in incremental phase**:
- top-level `lob.enabled` option, checked by `OracleLobOptionValidator`
- `OracleLobUnavailableValueHandler`, which re-selects by primary key `AS
OF SCN ?`
- LobFragment truncation by code point / UTF-16 amount
- `EMPTY_CLOB()`/`EMPTY_BLOB()` become empty values
- `TransactionCommitConsumer.mergeEvents` keeps a later same-transaction
SQL NULL
- en/zh docs and e2e
5. **[Feature][Connector-V2][Oracle-CDC] Support XMLTYPE**: port of DBZ-3605
(XML_BEGIN/XML_WRITE/XML_END) to the Debezium 1.9.8 LogMiner fork.
`XmlUnderConstruction` decodes UTF-8 once over all chunks, adds the
`xmlparserv2`/`xdb` dependencies (provided scope), plus docs and tests.
### Does this PR introduce _any_ user-facing change?
Yes. There are new Oracle-CDC source options:
| Option | Default | Meaning |
|---|---|---|
| `lob.enabled` | false (unset leaves `debezium.lob.enabled` unchanged) |
Pass-through to Debezium `lob.enabled`; mines CLOB/NCLOB/BLOB/XMLTYPE redo |
| `lob.reselect.enabled` | false | Re-selects LOB columns that hold the
unavailable placeholder by primary key (`AS OF SCN`) on INSERT/UPDATE_AFTER;
DELETE/UPDATE_BEFORE placeholders become null |
| `lob.unavailable-value.handling` | `warn_and_keep` | `warn_and_keep` /
`null` / `fail` when a placeholder remains |
The defaults change nothing: they issue no extra Oracle queries and leave
rows as before. Setting `lob.reselect.enabled=true` or handling `null`/`fail`
while LOB mining is off is rejected when the job is submitted. Snapshot now
returns real LOB values. The Oracle JDBC sink now writes long CLOBs and empty
LOBs correctly. XMLTYPE needs `xdb` and `xmlparserv2` next to `ojdbc8`. Docs
are updated in en and zh.
### How was this patch tested?
- **Unit tests**: added for TemporalConversions, OracleJdbcRowConverter,
OracleJdbcLobValueConverters, TransactionCommitConsumer (LOB fragments, null
merge), XmlBeginParser and XML write assembly, LOB reselect SQL and handler,
and option validation. connector-cdc-base, connector-cdc-oracle and
connector-jdbc unit tests pass.
- **E2E**: `OracleCDCIT` gained
`testOracleCdcLobSnapshotStreamingAndOutOfLineInsert` (snapshot, streaming,
out-of-line LOBs, emoji boundary, empty and NULL LOBs). A full `OracleCDCIT`
run on this branch head passed 48/48, 0 failures, 0 errors: Flink 1.13.6 /
1.15.3 / 1.18.0 / 1.20.1 and SeaTunnel Zeta (LOB test 5/5).
- **Stability runs** against Oracle 19c with continuous load:
- 6 tables in 2 schemas with mixed LOB/non-LOB DML: 0 mismatches, 0
placeholders, 0 U+FFFD.
- Emoji/SMP and XMLTYPE workload, about 15 minutes: clean, source equals
sink.
**Known limitations**
- Not supported: tables whose row type is XMLTYPE, `XMLTYPE STORE AS CLOB`,
and DDL that adds an XMLTYPE column.
- Already present before this PR, not changed here: the JDBC sink MERGE
(upsert) path can raise ORA-01461 for large CLOB values.
### Check list
* [x] If any new Jar binary package adding in your PR, please add License
Notice according [New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
(xmlparserv2 is `provided` scope and is not bundled)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the
following files are updated: (not applicable, existing connector)
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
