[
https://issues.apache.org/jira/browse/FLINK-41107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiqian Yu reassigned FLINK-41107:
---------------------------------
Assignee: Xin Chen
> schema.change.behavior: lenient or exclude.schema.changes is ignored by Hudi
> and MaxCompute sinks because their MetadataApplier implementations do not
> honor accepted schema evolution types
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-41107
> URL: https://issues.apache.org/jira/browse/FLINK-41107
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
> Affects Versions: cdc-3.6.0, cdc-3.7.0
> Reporter: Xin Chen
> Assignee: Xin Chen
> Priority: Major
>
> HudiMetadataApplier and MaxComputeMetadataApplier do not honor the schema
> evolution event types configured through include.schema.changes and
> exclude.schema.changes.
> In the Pipeline runtime, the resolved allowed schema change event types are
> passed to each sink through:
>
> {code:java}
> MetadataApplier#setAcceptedSchemaEvolutionTypes(...){code}
>
>
> Before forwarding an event to the sink, SchemaDerivator checks:
>
> {code:java}
> metadataApplier.acceptsSchemaEvolutionType(schemaChangeEvent.getType()){code}
>
> However, both Hudi and MaxCompute metadata appliers rely on the default
> methods in MetadataApplier:
>
> {code:java}
> default MetadataApplier setAcceptedSchemaEvolutionTypes(
> Set<SchemaChangeEventType> schemaEvolutionTypes) {
> return this;
> }
> default boolean acceptsSchemaEvolutionType(
> SchemaChangeEventType schemaChangeEventType) {
> return true;
> }
> {code}
>
> As a result, the configured accepted event types are discarded and every
> schema change event is accepted.
> This makes configurations such as the following ineffective:
>
> {code:java}
> sink:
> exclude.schema.changes:
> - truncate.table
> - drop.table{code}
> or
> {code:java}
> schema.change.behavior: lenient{code}
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
