Yingjie Cao created FLINK-12329:
-----------------------------------
Summary: Netty thread deadlock bug of the SpilledSubpartitionView
Key: FLINK-12329
URL: https://issues.apache.org/jira/browse/FLINK-12329
Project: Flink
Issue Type: Bug
Components: Runtime / Network
Affects Versions: 1.8.0
Reporter: Yingjie Cao
Fix For: 1.9.0
The Netty thread may be blocked when using the blocking batch mode of FLINK. In
my opinion, the combination of several designs, including request buffer
blocking, zero coy (the buffer is recycled only after it is sent out to
network), limited number of buffers (only two buffers and not configurable) and
backlog data is not ready (must request buffer and read, pipeline mode dose not
have the problem), leads to this bug.
The flowing processing flow of Netty thread can block itself (note that
writeAndFlush dose not mean the buffer is send out to the network).
1. request and read the first buffer -> write and flush the first buffer ->
send the first buffer to network -> request and read the second buffer ->
write and flush the first buffer -> no credit -> add credit -> request and
read buffer -> blocking (the second buffer is not sent out)
2. no credit -> add credit -> request and read buffer -> write and flush the
buffer -> no credit -> add credit -> request and read buffer -> blocking (the
previous read buffer is not sent out)
How to reproduce?
The bug is easy to be reproduced, two vertices with a blocking edge can
reproduce it. Large parallelism, small number of slots per TM and large data
volume make it easy to reproduce the bug.Setting the parallelism to 100, the
number of slots per TM to 1 and more than 10M data per subpartition will be ok.
How to fix?
The new mmappartition implementation can fix this problem because the number of
buffers is not limited and the data is not loaded until sent.
The bug can be also fixed based on the old implementation. Firstly, the buffer
request should not be blocking. Besides, the NetworkSequenceViewReader should
enqueue as available reader when it is available for read and is not registered
currently.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
