[
https://issues.apache.org/jira/browse/FLINK-12294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17336492#comment-17336492
]
Flink Jira Bot commented on FLINK-12294:
----------------------------------------
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Kafka connector, work with grouping partitions
> ----------------------------------------------
>
> Key: FLINK-12294
> URL: https://issues.apache.org/jira/browse/FLINK-12294
> Project: Flink
> Issue Type: New Feature
> Components: API / DataStream, Connectors / Kafka, Runtime / Task
> Reporter: Sergey
> Priority: Major
> Labels: performance, stale-major
> Attachments: KeyGroupAssigner.java, KeyGroupRangeAssignment.java
>
>
> Additional flag (with default false value) controlling whether topic
> partitions already grouped by the key. Exclude unnecessary shuffle/resorting
> operation when this parameter set to true. As an example, say we have
> client's payment transaction in a kafka topic. We grouping by clientId
> (transaction with the same clientId goes to one kafka topic partition) and
> the task is to find max transaction per client in sliding windows. In terms
> of map\reduce there is no needs to shuffle data between all topic consumers,
> may be it`s worth to do within each consumer to gain some speedup due to
> increasing number of executors within each partition data. With N messages
> (in partition) instead of N*ln(N) (current realization with
> shuffle/resorting) it will be just N operations. For windows with thousands
> events - the tenfold gain of execution speed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
