[
https://issues.apache.org/jira/browse/FLINK-40349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongshun Wang reassigned FLINK-40349:
-------------------------------------
Assignee: Hongshun Wang
> Add Fluss sink schema compatibility option to avoid silent data loss
> --------------------------------------------------------------------
>
> Key: FLINK-40349
> URL: https://issues.apache.org/jira/browse/FLINK-40349
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
> Affects Versions: cdc-3.5.0
> Reporter: Hongshun Wang
> Assignee: Hongshun Wang
> Priority: Major
> Fix For: cdc-3.6.0
>
>
>
> h2. {{Motivation}}
> {{Currently, the Fluss sink supports schema mapping between the input schema
> and the target Fluss table schema. This behavior is useful because Fluss only
> supports limited schema evolution today, so the upstream and downstream
> schemas may not always be exactly the same.}}
> {{}}
> {{However, the current default behavior may silently drop data when the
> upstream input schema contains columns that do not exist in the downstream
> Fluss table schema. For example, if the upstream schema is wider than the
> target Fluss table schema, the extra input columns can be ignored during
> sink-side schema mapping, and users may not notice that data is being lost.
> We should provide an explicit sink option to control how strictly the sink
> validates schema compatibility.}}
> {{}}
> h2.
> h2. {{Proposal}}
>
> {{Introduce a new Flink sink option, tentatively named:
> schema-validation-mode.}}
> {{}}
> The option controls the allowed relationship between the input schema and the
> target Fluss table schema.
> Supported values:
> * {{permissive: }}
> Default value. Keeps the current behavior for backward compatibility.
> The sink may perform schema mapping, projection, and padding based on the
> target Fluss table schema. This allows existing jobs to continue working as
> before.
> *
> {{target-superset:}}
> The target Fluss table schema must contain all fields from the input schema
> by field name.
> This mode allows the target schema to be wider than the input schema, which
> is useful for schema evolution such as adding nullable columns to the Fluss
> table. However, it rejects the case where the input schema contains columns
> that do not exist in the target Fluss table schema, preventing silent data
> loss.
> *
> {{strict: }}
> The input schema and target Fluss table schema must be strictly identical.
> The validation should check field count, field names, field order, and field
> types. If any mismatch is found, the sink should fail during initialization
> with a clear validation error.
> {{{}{}}}{{{}{}}}
> {{{}{}}}{{{}{}}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
