Jubin Soni created FLINK-41498:
----------------------------------
Summary: OceanBase pipeline sink ignores lenient mode and
exclude.schema.changes, dropping and truncating target tables
Key: FLINK-41498
URL: https://issues.apache.org/jira/browse/FLINK-41498
Project: Flink
Issue Type: Bug
Components: Flink CDC
Affects Versions: cdc-3.7.0, cdc-3.8.0
Reporter: Jubin Soni
h2. Bug / gap
OceanBaseMetadataApplier does not override
MetadataApplier#setAcceptedSchemaEvolutionTypes, #acceptsSchemaEvolutionType or
#getSupportedSchemaEvolutionTypes. The interface defaults discard the
configured event types and accept every schema change type.
SchemaDerivator#normalizeSchemaChangeEvents filters schema change events only
through metadataApplier.acceptsSchemaEvolutionType(...), so for the OceanBase
sink nothing is filtered. With the default schema.change.behavior (lenient),
the parser excludes drop.table and truncate.table, but upstream DROP TABLE /
TRUNCATE TABLE events are still forwarded to
OceanBaseMetadataApplier#applySchemaChange, which calls catalog.dropTable /
catalog.truncateTable on the target table. The same happens when a user
explicitly sets sink.exclude.schema.changes: [drop.table, truncate.table], or
uses include.schema.changes.
h2. Reproduction
Pipeline (default lenient behavior, no include/exclude options):
{code}
source:
type: mysql
…
sink:
type: oceanbase
url: jdbc:mysql://<host>:2881/test
username: root@test
password: ‘'
{code}
Resolving the accepted types with YamlPipelineDefinitionParser, passing them to
OceanBaseMetadataApplier#setAcceptedSchemaEvolutionTypes, and running
SchemaDerivator#normalizeSchemaChangeEvents on a TruncateTableEvent and a
DropTableEvent (released 3.7.0-1.20 jars; the four files involved are unchanged
on master):
{code}
resolved accepted types = [ADD_COLUMN, ALTER_COLUMN_TYPE, CREATE_TABLE,
DROP_COLUMN, RENAME_COLUMN, ALTER_TABLE_COMMENT]
acceptsSchemaEvolutionType(DROP_TABLE) = true
acceptsSchemaEvolutionType(TRUNCATE_TABLE) = true
events forwarded to applySchemaChange =
[TruncateTableEvent{tableId=app_db.orders},
DropTableEvent{tableId=app_db.orders}]
{code}
Expected:
{code}
acceptsSchemaEvolutionType(DROP_TABLE) = false
acceptsSchemaEvolutionType(TRUNCATE_TABLE) = false
events forwarded to applySchemaChange = []
{code}
h2. Why it matters
The schema evolution docs state that in lenient mode (the default)
"TruncateTableEvent and DropTableEvent will not be sent to downstream to avoid
unexpected data loss". For the OceanBase sink this does not hold: a TRUNCATE
TABLE or DROP TABLE on the MySQL source deletes the data or the whole table in
OceanBase, and per-event-type exclusion cannot prevent it.
h2. Related work
FLINK-41107 (PR #4599, fixed in cdc-3.8.0) fixed the same gap for
HudiMetadataApplier and MaxComputeMetadataApplier. OceanBaseMetadataApplier has
the same gap and was not covered. All other pipeline sinks (Doris, StarRocks,
Paimon, Iceberg, Fluss, Kafka, Values) already override these methods. Searched
JIRA for "oceanbase", "acceptsSchemaEvolutionType",
"setAcceptedSchemaEvolutionTypes", "exclude.schema.changes", and Flink CDC
issues mentioning MetadataApplier / drop.table / truncate.table, and GitHub PRs
for OceanBaseMetadataApplier and the same method names, on 2026-11-14: no
existing ticket or PR for OceanBase.
h2. Proposed fix
Keep the enabled schema evolution types in OceanBaseMetadataApplier
(initialised to its supported types), override setAcceptedSchemaEvolutionTypes
/ acceptsSchemaEvolutionType to honour them, and override
getSupportedSchemaEvolutionTypes with the types the applier actually
implements, following DorisMetadataApplier and the FLINK-41107 change.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
