Hi all. Seems like reasonable changes to remove the 'reset_ttl_on_duplicate` parameter from DEDUPLICATE_WINDOW
I will update the FLIP accordingly. Thanks for the feedback Kind Regards Vas. ________________________________ From: Timo Walther via dev <[email protected]> Sent: 11 November 2026 08:50 To: [email protected] <[email protected]> Cc: Timo Walther <[email protected]> Subject: Re: [DISCUSS] FLIP-622: Add built-in DEDUPLICATE and DEDUPLICATE_WINDOW PTFs Hi Vas, thanks for proposing this FLIP. We had a lot of offline discussion for this design document, but I'm quite happy about the outcome. DEDUPLICATE is one of the most required transformations in stream processing and having a dedicated and highly optimized PTF for this will make many use cases easier. Big +1 for this FLIP. Reg Gustavo's feedback: I agree we can drop reset_ttl_on_duplicate parameter and make it the default for DEDUPLICATE and skip if for DEDUPLICATE_WINDOW. Also regarding the other open questions: nullability sorting is defined by Flink's generated comparators already. And reg the changelog mode, where ever we can declare append we should. Thanks, Timo On 10.11.26 16:40, Gustavo de Morais wrote: > Hi Vas, > > The FLIP and the examples look great. I really like the separation between > the updating and the event-time worlds. I think with this we have a clear, > easy-to-use deduplication feature that willl help people easily perform one > of the most common tasks we see. > > Regarding one of the open points, I think we can drop the > `reset_ttl_on_duplicate` parameter for DEDUPLICATE_WINDOW. Since this is > window-based event-time processing, most of the state is taken care of by > windows closing. The additional `state_ttl` is just a last-resort cleanup > mechanism for windows that never close, so it should be enough on its own > without an extra parameter to configure it. I'd be interested to hear how > useful others find it for DEDUPLICATE. > > Kind regards, > Gustavo > > On Tue, 10 Nov 2026 at 16:14, Vas Shabu <[email protected]> wrote: > >> Hi all, >> >> I'd like to propose FLIP-622: Add built-in DEDUPLICATE and >> DEDUPLICATE_WINDOW PTFs [1] for discussion. It supersedes FLIP-614 [2]. >> While working on DEDUPLICATE_KEEP_LAST, we found a cleaner design: one >> function per output mode, with first or last as an argument. >> DEDUPLICATE_KEEP_FIRST is unreleased, so we want to raise this before >> anything ships. >> >> Today deduplication is written as a ROW_NUMBER() over-window. That pattern >> is planned as deduplication only when ordered by a time attribute and >> becomes a general Top-N when ordered by a version column. This FLIP adds >> two built-in PTFs, each with keep => 'first' | 'last'. >> >> DEDUPLICATE keeps one row per key. Its output is updating, and it accepts >> insert-only and updating input: >> >> SELECT * FROM DEDUPLICATE( >> input => TABLE(clicks) PARTITION BY event_id, >> keep => 'first' >> ); >> >> DEDUPLICATE_WINDOW emits one final row per key and event-time window, and >> its output is insert-only: >> >> SELECT * FROM DEDUPLICATE_WINDOW( >> input => TABLE(clicks) PARTITION BY event_id, >> keep => 'first', >> duration => INTERVAL '5' MINUTES, >> on_time => DESCRIPTOR(ts) >> ); >> >> With this proposal, we aim to improve user experience by introducing a >> user-friendly feature to perform one of the most common tasks in Flink. >> Looking forward to your feedback and thoughts. >> >> Kind regards, >> Vas Shabu >> >> >> [1] >> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fspaces%2FFLINK%2Fpages%2F462816464%2FFLIP-622%2BAdd%2Bbuilt-in%2BDEDUPLICATE%2Band%2BDEDUPLICATE_WINDOW%2BPTFs&data=05%7C02%7C%7C55857733197443665f0a08df3fd0d1b6%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C639299838231015447%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2rVJSoQjMLtTgx920nY5dKNooN2a0JFcxJACnQ7Xat0%3D&reserved=0<https://cwiki.apache.org/confluence/spaces/FLINK/pages/462816464/FLIP-622+Add+built-in+DEDUPLICATE+and+DEDUPLICATE_WINDOW+PTFs> >> [2] >> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fspaces%2FFLINK%2Fpages%2F451975182%2FFLIP-614%2BAdd%2Bbuilt-in%2BDEDUPLICATE_KEEP_FIRST%2BPTF&data=05%7C02%7C%7C55857733197443665f0a08df3fd0d1b6%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C639299838231071868%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=MJ6VG%2BCzIrRhuYMuY3R7VIj9QWNDvavlDcmn351ZJf0%3D&reserved=0<https://cwiki.apache.org/confluence/spaces/FLINK/pages/451975182/FLIP-614+Add+built-in+DEDUPLICATE_KEEP_FIRST+PTF> >> >
