ss666 opened a new pull request, #11108:
URL: https://github.com/apache/seatunnel/pull/11108
### Purpose of this pull request
Closes #11044. Adds include/exclude filtering for CDC schema change event
types, so users can control
which schema changes propagate downstream.
- New source options `schema-changes.include` / `schema-changes.exclude`.
Wired into MySQL / Oracle / SqlServer CDC.
- Filtering lives in `connector-cdc-base`, applied after events are
normalized to the canonical model and
**before** downstream emission (before checkpoint/barrier coordination and
sink-side schema evolution).
- Deterministic precedence: (1) if `include` is non-empty, only included
types are eligible; (2) then
`exclude` is applied; (3) `exclude` wins on conflict. Unknown names fail
fast with a message listing valid
names; names are trimmed, case-insensitive, de-duplicated.
### Does this PR introduce _any_ user-facing change?
Yes.
- Two new optional CDC source options `schema-changes.include` /
`schema-changes.exclude`
(default empty = no filtering, fully backward compatible).
- en/zh docs for MySQL / Oracle / SqlServer CDC updated with a
canonical-name table and a HOCON example.
- Known limitation: excluding `drop.column` keeps the column in the sink
schema while the source stops supplying it.
- a **nullable** retained column is written as `NULL` (works);
- a **NOT NULL** retained column has its `NULL` write rejected by the
sink and the job fails. Deciding what
to write is *data-handling behavior*, out of #11044's event-type-filtering
scope. See the follow-up below.
### How was this patch tested?
- Format: `./mvnw spotless:apply`
- Unit: `SchemaChangeEventFilterTest` — 13 cases (include-only /
exclude-only / precedence / group alias /
unknown-name fail-fast / normalization + dedup / mapping completeness).
`Tests run: 13, Failures: 0, Errors: 0`.
- E2E: `MysqlCDCWithSchemaChangeIT#testMysqlCdcSchemaChangeEventTypeFilter`
— add.column propagates, excluded
drop.column stays in the sink, concurrent INSERT/UPDATE/DELETE still reach
the sink. `Tests run: 1, Failures: 0, Errors: 0`.
- Compile: `./mvnw clean install -pl
:connector-cdc-base,:connector-cdc-mysql -am -DskipTests`.
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according [New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR. (N/A — backward compatible, options default
to empty)
* [ ] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
— N/A (no new connector)
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
— N/A
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
— N/A
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
— done (`MysqlCDCWithSchemaChangeIT`)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
— N/A
### Follow up
The NOT NULL limitation above is out of #11044's scope (it is *data-handling
behavior*, not event-type
filtering), but it is still necessary to make exclude(drop.column) usable
for NOT NULL columns. Either of the two mechanisms below might fix it and could
be proposed as a dedicated follow-up:
- **Option A — relax the retained column to nullable (derived schema
change).** When `exclude` suppresses
the `drop.column` of a NOT NULL column, emit a derived "modify column →
nullable" event downstream so the
retained sink column legitimately accepts `NULL`. This changes the
*constraint*, not the data, and is mostly contained in `connector-cdc-base`. It
actually makes `exclude(drop.column)` work for NOT NULL
columns instead of merely failing more politely.
- **Option B — opt-in, user-declared fill value.** Add a per-column option
(e.g.
`schema-changes.drop-column.fill`) the user explicitly sets for a
retained-but-no-longer-supplied NOT NULL
column. Because the value is user-declared, it is *not* silent fabrication
— the user owns the semantics.
When unset, the job still fails fast.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
