Gustavo de Morais created FLINK-41549:
-----------------------------------------

             Summary: FLIP-621: Add built-in TO_APPEND PTF
                 Key: FLINK-41549
                 URL: https://issues.apache.org/jira/browse/FLINK-41549
             Project: Flink
          Issue Type: New Feature
          Components: Table SQL / API, Table SQL / Planner
            Reporter: Gustavo de Morais
            Assignee: Gustavo de Morais


Many queries produce an updating result (upsert sources, deduplication, 
aggregations, joins), but the place they write to only accepts appends: an 
append-only Kafka topic, an append-only sink, or an operator that requires 
insert-only input. Since {{TO_CHANGELOG}} (FLIP-564) was released, one use case 
keeps coming up: write inserts and updates to an append-only log and drop 
deletes. {{TO_CHANGELOG}} can do this, but it requires an {{op_mapping}} that 
drops deletes plus a projection that removes the op column.

This FLIP adds a built-in {{TO_APPEND}} process table function that converts an 
updating table into an append-only table in one call. INSERT and UPDATE_AFTER 
rows are emitted as INSERT rows, UPDATE_BEFORE and DELETE rows are dropped, and 
the output keeps the input schema. The function is stateless and has row 
semantics. With FLINK-41371 and FLINK-41516, an upsert input with an upsert key 
is consumed as emitted, without a {{ChangelogNormalize}}.

{code:sql}
-- Input: upsert table 'customers' with primary key 'id'
-- +U[id:'c1', city:'Berlin']
-- +U[id:'c1', city:'Munich']
-- -D[id:'c1']

SELECT * FROM TO_APPEND(input => TABLE customers)

-- Output (append-only):
-- +I[id:'c1', city:'Berlin']
-- +I[id:'c1', city:'Munich']
{code}

The Table API exposes the same function as {{Table.toAppend()}}.

FLIP: 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-621%3A+Add+built-in+TO_APPEND+PTF
Discussion thread: 
https://lists.apache.org/thread/g5gytjmlfhglokl7zpcfqd6xh70snjb8
Vote thread: https://lists.apache.org/thread/5lof8ms2s2ycn090ozjzvz1jqfcs7z7m



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to