Xin Chen created FLINK-41107:
--------------------------------
Summary: schema.change.behavior: lenient or
exclude.schema.changes is ignored by Hudi and MaxCompute sinks because their
MetadataApplier implementations do not honor accepted schema evolution types
Key: FLINK-41107
URL: https://issues.apache.org/jira/browse/FLINK-41107
Project: Flink
Issue Type: Bug
Components: Flink CDC
Affects Versions: cdc-3.7.0, cdc-3.6.0
Reporter: Xin Chen
HudiMetadataApplier and MaxComputeMetadataApplier do not honor the schema
evolution event types configured through include.schema.changes and
exclude.schema.changes.
In the Pipeline runtime, the resolved allowed schema change event types are
passed to each sink through:
{code:java}
MetadataApplier#setAcceptedSchemaEvolutionTypes(...){code}
Before forwarding an event to the sink, SchemaDerivator checks:
{code:java}
metadataApplier.acceptsSchemaEvolutionType(schemaChangeEvent.getType()){code}
However, both Hudi and MaxCompute metadata appliers rely on the default methods
in MetadataApplier:
{code:java}
default MetadataApplier setAcceptedSchemaEvolutionTypes(
Set<SchemaChangeEventType> schemaEvolutionTypes) {
return this;
}
default boolean acceptsSchemaEvolutionType(
SchemaChangeEventType schemaChangeEventType) {
return true;
}
{code}
As a result, the configured accepted event types are discarded and every schema
change event is accepted.
This makes configurations such as the following ineffective:
{code:java}
sink:
exclude.schema.changes:
- truncate.table
- drop.table{code}
or
{code:java}
schema.change.behavior: lenient{code}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
