Zhijiang created FLINK-16645: -------------------------------- Summary: Limit the maximum backlogs in subpartitions for data skew case Key: FLINK-16645 URL: https://issues.apache.org/jira/browse/FLINK-16645 Project: Flink Issue Type: Sub-task Components: Runtime / Network Reporter: Zhijiang Fix For: 1.11.0
In the case of data skew, most of the buffers in partition's LocalBufferPool are probably requested away and accumulated in certain subpartition, which would increase in-flight data to slow down the barrier alignment. We can set up a proper config to control how many backlogs are allowed for one subpartition. If one subpartition reaches this threshold, it will make the buffer pool unavailable which blocks task processing continuously. Then we can reduce the in-flight data for speeding up checkpoint process a bit and not impact on the performance. -- This message was sent by Atlassian Jira (v8.3.4#803005)