Weijie Guo created FLINK-28623:
----------------------------------
Summary: Optimize the use of off heap memory by blocking and
hybrid shuffle reader
Key: FLINK-28623
URL: https://issues.apache.org/jira/browse/FLINK-28623
Project: Flink
Issue Type: Improvement
Components: Runtime / Network
Reporter: Weijie Guo
Currently, each FileReader(PartitionFileReader or HsSubpartitionFileReaderImpl
) will internally allocate a headerbuffer with the size of 8B. Beside,
PartitionFileReader also has a 12B indexEntryBuf. Because FileReader is of
subpartition granularity, If the parallelism becomes very big, and there are
many slots on each TM, the memory occupation will even reach the MB level. In
fact, all FileReader of the same ResultPartition read data in a single thread,
so we only need to allocate a headerbuffer to a ResultPartition to optimize
this phenomenon.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
