[
https://issues.apache.org/jira/browse/FLINK-19896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17224037#comment-17224037
]
Jun Zhang commented on FLINK-19896:
-----------------------------------
Put aside the semantic or definition of "deduplicate", I think collecting the
first n rows is easier to implement if viewed as kind of deduplication than
top-n. The generic top-n implementation would involve sort key comparison and
row retraction which are not needed for first-n-rows case. Actually we only
need to store a counter in the state, much like a boolean flag in the first row
case.
> Support first-n-rows deduplication in the Deduplicate operator
> --------------------------------------------------------------
>
> Key: FLINK-19896
> URL: https://issues.apache.org/jira/browse/FLINK-19896
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner, Table SQL / Runtime
> Reporter: Jun Zhang
> Priority: Major
>
> Currently Deduplicate operator only supports first-row deduplication (ordered
> by proc-time). In scenario of first-n-rows deduplication, the planner has to
> resort to Rank operator. However, Rank operator is less efficient than
> Deduplicate due to larger state and more state access.
> This issue proposes to extend DeduplicateKeepFirstRowFunction to support
> first-n-rows deduplication. And the original first-row deduplication would be
> a special case of first-n-rows deduplication.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
