[
https://issues.apache.org/jira/browse/FLINK-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yingjie Cao updated FLINK-20740:
--------------------------------
Description:
Currently, sort-merge blocking shuffle uses some unmanaged memory for data
writing and reading, which means users must increase the size of direct memory,
otherwise, one may encounter direct memory OOM error, which is really bad for
usability.
As discussed in the mailing list, a separated buffer pool will be introduced to
be used by the shuffle reading phase, this buffer pool is part of Flink
framework and has different life cycle with the tasks.
was:Currently, sort-merge blocking shuffle uses some unmanaged memory for
data writing and reading, which means users must increase the size of direct
memory, otherwise, one may encounter direct memory OOM error, which is really
bad for usability.
> Use managed memory (network memory) to avoid direct memory OOM error for
> sort-merge shuffle
> -------------------------------------------------------------------------------------------
>
> Key: FLINK-20740
> URL: https://issues.apache.org/jira/browse/FLINK-20740
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
> Affects Versions: 1.12.0
> Reporter: Yingjie Cao
> Priority: Major
> Fix For: 1.13.0
>
>
> Currently, sort-merge blocking shuffle uses some unmanaged memory for data
> writing and reading, which means users must increase the size of direct
> memory, otherwise, one may encounter direct memory OOM error, which is really
> bad for usability.
> As discussed in the mailing list, a separated buffer pool will be introduced
> to be used by the shuffle reading phase, this buffer pool is part of Flink
> framework and has different life cycle with the tasks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
