[ https://issues.apache.org/jira/browse/FLINK-33879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jiang Xin updated FLINK-33879: ------------------------------ Description: Currently, the Hybrid Shuffle can work with the memory tier and disk tier together, however, in the following scenario the result partition would stop working. Suppose we have a shuffle task with 2 sub-partitions. The LocalBufferPool has 15 buffers, the memory tier can use at most 15-(2*(2+1)+1) = 8 buffers accroding to `TieredStorageMemoryManagerImpl#getMaxNonReclaimableBuffers`. If the memory tier uses up all 8 buffers and the input channel doesn't consume them because of some problem, the disk tier can still work with 1 reserved buffer. However, if a redistribution happens now and the pool size is decreased to less than 8, then the BufferAccumulator can not request buffers anymore, and thus the result partition stops working as well. was: Currently, the Hybrid Shuffle can work with Memory Tier and Disk Tier together, however, in the following scenirio the result partition would stop working. Suppose we have a shuffle task with 2 sub partitions. The LocalBufferPool has 15 buffers, the memory tier can use at most 15-(2*(2+1)+1) = 8 buffers. If the memory tier used up all 8 buffers and the input channel doesn't consume them because of some problem, the disk tier can still work with 1 reserved buffer. However, if a redistribution happens now and the pool size is decrease to less than 8, then the BufferAccumulator can not request buffers any more, so > Hybrid Shuffle may hang during redistribution > --------------------------------------------- > > Key: FLINK-33879 > URL: https://issues.apache.org/jira/browse/FLINK-33879 > Project: Flink > Issue Type: Bug > Components: Runtime / Network > Reporter: Jiang Xin > Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > Currently, the Hybrid Shuffle can work with the memory tier and disk tier > together, however, in the following scenario the result partition would stop > working. > Suppose we have a shuffle task with 2 sub-partitions. The LocalBufferPool has > 15 buffers, the memory tier can use at most 15-(2*(2+1)+1) = 8 buffers > accroding to `TieredStorageMemoryManagerImpl#getMaxNonReclaimableBuffers`. If > the memory tier uses up all 8 buffers and the input channel doesn't consume > them because of some problem, the disk tier can still work with 1 reserved > buffer. However, if a redistribution happens now and the pool size is > decreased to less than 8, then the BufferAccumulator can not request buffers > anymore, and thus the result partition stops working as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)