[
https://issues.apache.org/jira/browse/FLINK-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yingjie Cao updated FLINK-20740:
--------------------------------
Summary: Use managed memory (network memory) to avoid direct memory OOM
error for sort-merge shuffle (introduce a separated buffer pool) (was: Use
managed memory (network memory) to avoid direct memory OOM error for sort-merge
shuffle)
> Use managed memory (network memory) to avoid direct memory OOM error for
> sort-merge shuffle (introduce a separated buffer pool)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-20740
> URL: https://issues.apache.org/jira/browse/FLINK-20740
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
> Affects Versions: 1.12.0
> Reporter: Yingjie Cao
> Priority: Major
> Fix For: 1.13.0
>
>
> Currently, sort-merge blocking shuffle uses some unmanaged memory for data
> writing and reading, which means users must increase the size of direct
> memory, otherwise, one may encounter direct memory OOM error, which is really
> bad for usability.
> As discussed in the mailing list, a separated buffer pool will be introduced
> to be used by the shuffle reading phase, this buffer pool is part of Flink
> framework and has different life cycle with the tasks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
