Xiqian Yu created FLINK-38393:
---------------------------------
Summary: Pipeline transform schema consistency check should be
loosen for unreachable rules
Key: FLINK-38393
URL: https://issues.apache.org/jira/browse/FLINK-38393
Project: Flink
Issue Type: Bug
Components: Flink CDC
Affects Versions: cdc-3.4.0
Reporter: Xiqian Yu
Consider this transform block:
{code:java}
transform:
- { source-table: db.tbl1, filter: 'id > 0', projection: 'id, col1' }
- { source-table: db.tbl1, filter: 'id <= 0', projection: 'id, col2, col3' }
{code}
This should be correctly rejected, since there might be two incompatible
schemas deduced, depending on if the filter condition is satisfied or not.
However, consider a slightly different example:
{code:java}
transform:
- { source-table: db.tbl1, projection: 'id, col1' }
- { source-table: db.tbl2, projection: 'id, col2' }
- ...
- { source-table: db.\.*, projection: 'id, fallback' } {code}
One may write a wildcard matching rule at the end, as a fallback rule.
However, this will be rejected in CDC 3.4.0, because Table "db.tbl1" matches
two rules with incompatible schema, though the latter one is unreachable.
We may loose the validation check here to avoid breaking existing use cases.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
