Hassan Eslami created GIRAPH-1137:
-------------------------------------
Summary: Remove channel probing from Netty worker thread for
credit-based flow-control
Key: GIRAPH-1137
URL: https://issues.apache.org/jira/browse/GIRAPH-1137
Project: Giraph
Issue Type: Bug
Reporter: Hassan Eslami
Assignee: Hassan Eslami
In credit-based flow-control, sometimes, client threads (one type of Netty
worker threads used in Giraph) try to send requests to other workers. This is
bad practice for Netty and can cause Netty to mark the execution as
deadlock-prone (an example exception shown below). Client threads should only
be responsible for sending ACK/NACK messages in response to requests, and they
should do so by reuseing the channel from which they received the request. In
the current implementation, client threads may try to send unsent/cached
requests in credit-based flow control. Sending such requests should be
delegated to other threads.
WARN 2017-03-08 06:06:22,104 [netty-client-worker-3] ....
io.netty.util.concurrent.BlockingOperationException:
DefaultChannelPromise@2c455378(incomplete)
at
io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:383)
at
io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157)
at io.netty.util.concurrent.DefaultPromise.await0(DefaultPromise.java:343)
at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:259)
at
org.apache.giraph.utils.ProgressableUtils$ChannelFutureWaitable.waitFor(ProgressableUtils.java:461)
at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:214)
at
org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:180)
at
org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:165)
at
org.apache.giraph.utils.ProgressableUtils.awaitChannelFuture(ProgressableUtils.java:132)
at org.apache.giraph.comm.netty.NettyClient.getNextChannel(NettyClient.java:715)
at
org.apache.giraph.comm.netty.NettyClient.writeRequestToChannel(NettyClient.java:799)
at org.apache.giraph.comm.netty.NettyClient.doSend(NettyClient.java:789)
at
org.apache.giraph.comm.flow_control.CreditBasedFlowControl.trySendCachedRequests(CreditBasedFlowControl.java:515)
at
org.apache.giraph.comm.flow_control.CreditBasedFlowControl.messageAckReceived(CreditBasedFlowControl.java:485)
at
org.apache.giraph.comm.netty.NettyClient.messageReceived(NettyClient.java:840)
at
org.apache.giraph.comm.netty.handler.ResponseClientHandler.channelRead(ResponseClientHandler.java:87)
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
at
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:153)
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
at
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
at
org.apache.giraph.comm.netty.InboundByteCounter.channelRead(InboundByteCounter.java:89)
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
at
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:785)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:126)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:485)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:452)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:346)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
