[jira] [Commented] (FLINK-31386) Fix the potential deadlock issue of blocking shuffle
[ https://issues.apache.org/jira/browse/FLINK-31386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17715425#comment-17715425 ] Yingjie Cao commented on FLINK-31386: - Cherry picked to 1.16 via 4e9516aa855cd5262a8574ecce60768553f0e7cf. > Fix the potential deadlock issue of blocking shuffle > > > Key: FLINK-31386 > URL: https://issues.apache.org/jira/browse/FLINK-31386 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.16.0, 1.16.1 >Reporter: Yingjie Cao >Assignee: Yingjie Cao >Priority: Blocker > Labels: pull-request-available > Fix For: 1.17.0, 1.16.2 > > > Currently, the SortMergeResultPartition may allocate more network buffers > than the guaranteed size of the LocalBufferPool. As a result, some result > partitions may need to wait other result partitions to release the > over-allocated network buffers to continue. However, the result partitions > which have allocated more than guaranteed buffers relies on the processing of > input data to trigger data spilling and buffer recycling. The input data > further relies on batch reading buffers used by the > SortMergeResultPartitionReadScheduler which may already taken by those > blocked result partitions which are waiting for buffers. Then deadlock > occurs. We can easily fix this deadlock by reserving the guaranteed buffers > on initializing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31386) Fix the potential deadlock issue of blocking shuffle
[ https://issues.apache.org/jira/browse/FLINK-31386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17698953#comment-17698953 ] Yingjie Cao commented on FLINK-31386: - Merged into 1.17 to unblock release. Will pick to master latter. > Fix the potential deadlock issue of blocking shuffle > > > Key: FLINK-31386 > URL: https://issues.apache.org/jira/browse/FLINK-31386 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Reporter: Yingjie Cao >Assignee: Yingjie Cao >Priority: Blocker > Labels: pull-request-available > Fix For: 1.17.0 > > > Currently, the SortMergeResultPartition may allocate more network buffers > than the guaranteed size of the LocalBufferPool. As a result, some result > partitions may need to wait other result partitions to release the > over-allocated network buffers to continue. However, the result partitions > which have allocated more than guaranteed buffers relies on the processing of > input data to trigger data spilling and buffer recycling. The input data > further relies on batch reading buffers used by the > SortMergeResultPartitionReadScheduler which may already taken by those > blocked result partitions which are waiting for buffers. Then deadlock > occurs. We can easily fix this deadlock by reserving the guaranteed buffers > on initializing. -- This message was sent by Atlassian Jira (v8.20.10#820010)