Gustavo de Morais created FLINK-41562:
-----------------------------------------

             Summary: Declare insert-only output for DEDUPLICATE when it can 
only emit inserts, and keep the rowtime attribute
                 Key: FLINK-41562
                 URL: https://issues.apache.org/jira/browse/FLINK-41562
             Project: Flink
          Issue Type: Sub-task
          Components: Table SQL / Planner
            Reporter: Gustavo de Morais


{{DEDUPLICATE}} declares updating output, but some calls can only ever emit +I. 
The planner should declare insert-only output for them, so append-only sinks 
and event-time operators can consume the result without {{TO_CHANGELOG}}.

*When the output is insert-only:*
* {{keep => 'first'}} without {{sequence_by}}: in arrival order, the first 
record is never replaced.
* PARTITION BY covers every input column (exact duplicates): later records are 
identical, and an identical record emits nothing. Holds for both {{keep}} 
values.

{{state_ttl}} expiry is fine: an expired key emits +I again.

*Example:* exact duplicates left in a sink after reprocessing. Today this needs 
{{ROW_NUMBER()}} plus {{TO_CHANGELOG}}.
{code:sql}
INSERT INTO trades_dedup   -- append-only sink
SELECT * FROM DEDUPLICATE(
    input => TABLE(trades_with_dups) PARTITION BY (trade_id, ticker, quantity, 
price)
)
{code}

Observation regarding rowtime/watermarks: the output is a subset of the input, 
in the same order with unchanged timestamps, so the input's watermarks stay 
valid. Options:
# The planner keeps the input's rowtime attribute on the output column and 
forwards watermarks. No new argument.
# Allow {{on_time}} only in these cases. Simpler in the PTF framework, but more 
things for the user to understand.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to