Yingjie Cao created FLINK-25796:
-----------------------------------
Summary: Avoid record copy for result partition of sort-shuffle if
there are enough buffers for better performance
Key: FLINK-25796
URL: https://issues.apache.org/jira/browse/FLINK-25796
Project: Flink
Issue Type: Sub-task
Components: Runtime / Network
Reporter: Yingjie Cao
Fix For: 1.15.0
Currently, for result partition of sort-shuffle, there is extra record copy
overhead Introduced by clustering records by subpartition index. For small
records, this overhead can cause even 20% performance regression. This ticket
aims to solve the problem.
In fact, the hash-based implementation is a nature way to achieve the goal of
sorting records by partition index. However, it incurs some serious weaknesses.
For example, when there is no enough buffers or there is data skew, it can
waste buffers and influence compression efficiency which can cause performance
regression.
This ticket tries to solve the issue by dynamically switching between the two
implementations, that is, if there are enough buffers, the hash-based
implementation will be used and if there is no enough buffers, the sort-based
implementation will be used.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
