Yuxin Tan created FLINK-32709: --------------------------------- Summary: Modify segment size to improve memory utilization for Hybrid Shuffle Key: FLINK-32709 URL: https://issues.apache.org/jira/browse/FLINK-32709 Project: Flink Issue Type: Bug Components: Runtime / Network Affects Versions: 1.18.0 Reporter: Yuxin Tan
Currently, each subpartition in Disk/Remote has a segment size of 8M. When writing segments to the Disk tier with a parallelism of 1000, only shuffle data exceeding 1000 * 8M can be written to the Memory tier again. However, for most shuffles, the data volume size falls below this limit, significantly impacting Memory tier utilization. For better performance, it is necessary to address this issue to improve the memory tier utilization. -- This message was sent by Atlassian Jira (v8.20.10#820010)