[
https://issues.apache.org/jira/browse/FLINK-41007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18127555#comment-18127555
]
sepuri sai krishna commented on FLINK-41007:
--------------------------------------------
I would like to work on this. Could someone assign the ticket to me?
The fix direction is a choice between rejecting the combination at planning
time and documenting
that the clause is ignored, so I would rather agree on that first than guess.
Happy to implement
either one.
> ON CONFLICT clause is silently ignored when
> table.exec.sink.upsert-materialize is NONE
> --------------------------------------------------------------------------------------
>
> Key: FLINK-41007
> URL: https://issues.apache.org/jira/browse/FLINK-41007
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
> Affects Versions: 2.3.0, 2.4.0
> Reporter: sepuri sai krishna
> Priority: Major
>
> With {{table.exec.sink.upsert-materialize}} set to {{NONE}}, an {{ON
> CONFLICT}} clause is accepted
> by the planner and then has no effect. {{DO ERROR}} does not fail on a
> primary key conflict,
> {{DO NOTHING}} does not keep the first record, and {{DO DEDUPLICATE}} does
> not deduplicate.
> h3. Reproducer
> Same query and sink, only {{table.exec.sink.upsert-materialize}} differs.
> {code:sql}
> CREATE TABLE src (a INT, b BIGINT, ts TIMESTAMP_LTZ(3), WATERMARK FOR ts AS
> ts)
> WITH ('connector' = 'datagen');
> CREATE TABLE snk (id INT, cnt BIGINT, PRIMARY KEY (id) NOT ENFORCED)
> WITH ('connector' = 'blackhole');
> INSERT INTO snk SELECT MAX(a), COUNT(*) FROM src GROUP BY b ON CONFLICT DO
> ERROR;
> {code}
> With {{AUTO}}, the default:
> {noformat}
> Sink(table=[...snk], fields=[EXPR$0, EXPR$1], upsertMaterialize=[true],
> conflictStrategy=[ERROR])
> +- Calc(select=[EXPR$0, EXPR$1])
> +- GroupAggregate(groupBy=[b], select=[b, MAX(a) AS EXPR$0, COUNT(*) AS
> EXPR$1])
> {noformat}
> With {{NONE}}:
> {noformat}
> Sink(table=[...snk], fields=[EXPR$0, EXPR$1], conflictStrategy=[ERROR])
> +- Calc(select=[EXPR$0, EXPR$1])
> +- GroupAggregate(groupBy=[b], select=[b, MAX(a) AS EXPR$0, COUNT(*) AS
> EXPR$1])
> {noformat}
> The strategy is still recorded on the sink, but there is no operator left in
> the plan that could
> apply it, so the clause is a no-op and conflicts pass through unresolved.
> h3. The clause is still validated under NONE
> {{DO ERROR}} and {{DO NOTHING}} keep their watermark requirement under
> {{NONE}}. With a source
> that has no watermark, {{AUTO}} and {{NONE}} both fail with the identical
> message:
> {noformat}
> ON CONFLICT DO ERROR requires all source tables to define watermarks, but the
> following source(s)
> do not: default_catalog.default_database.src. Please add a WATERMARK
> declaration to these tables.
> {noformat}
> So a user is required to add watermarks to satisfy a clause that will not be
> applied.
> h3. Expected
> Either the combination is rejected at planning time, the way an {{ON
> CONFLICT}} clause is already
> rejected for a sink that cannot use it, or the documentation states that the
> clause has no effect
> when the materializer is disabled.
> h3. Affects
> {{master}} and {{release-2.3}}.
> Related to FLINK-40924, which covers the {{FORCE}} side of the same
> validation, and to FLINK-40899.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
