Rion Williams created FLINK-41258:
-------------------------------------
Summary: Streaming join miscounts matches on DELETEs of records
that are not stored
Key: FLINK-41258
URL: https://issues.apache.org/jira/browse/FLINK-41258
Project: Flink
Issue Type: Bug
Components: Table SQL / Runtime
Reporter: Rion Williams
In {{StreamingJoinOperator}}, with an upsert input:
* A DELETE of a record that was never inserted, such as a tombstone, is still
joined. When the outer side's join key contains the unique key of the other
side, the outer record's number of matches can go negative. Since
{{isNewMatch}} doesn't increase a non-zero count, the record then never gets
its null padding back or matches correctly.
* A key-only DELETE of a record that is not stored is joined with its null
fields. A NULL-tolerant non-equi condition (for example {{r.v IS NULL OR ...}})
can then match it and retract rows it never produced.
h3. Proposed approach
As in FLINK-40842 for the semi/anti join: for inputs whose changelog mode is
known to be upsert, skip such DELETEs, or treat them as matching nothing. Plans
compiled before 2.4 keep the current handling, because a missing record there
may have expired under state TTL.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
