[jira] [Commented] (GIRAPH-1137) Remove channel probing from Netty worker thread for credit-based flow-control
[ https://issues.apache.org/jira/browse/GIRAPH-1137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15933911#comment-15933911 ] ASF GitHub Bot commented on GIRAPH-1137: Github user dlogothetis commented on the issue: https://github.com/apache/giraph/pull/26 Tests done: - Snapshot tests. - Performance on large PageRank benchmark remains the same. - Performance on internal prod job remains the same. > Remove channel probing from Netty worker thread for credit-based flow-control > - > > Key: GIRAPH-1137 > URL: https://issues.apache.org/jira/browse/GIRAPH-1137 > Project: Giraph > Issue Type: Bug >Reporter: Hassan Eslami >Assignee: Hassan Eslami > > In credit-based flow-control, sometimes, client threads (one type of Netty > worker threads used in Giraph) try to send requests to other workers. This is > bad practice for Netty and can cause Netty to mark the execution as > deadlock-prone (an example exception shown below). Client threads should only > be responsible for sending ACK/NACK messages in response to requests, and > they should do so by reuseing the channel from which they received the > request. In the current implementation, client threads may try to send > unsent/cached requests in credit-based flow control. Sending such requests > should be delegated to other threads. > WARN 2017-03-08 06:06:22,104 [netty-client-worker-3] > io.netty.util.concurrent.BlockingOperationException: > DefaultChannelPromise@2c455378(incomplete) > at > io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:383) > at > io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157) > at io.netty.util.concurrent.DefaultPromise.await0(DefaultPromise.java:343) > at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:259) > at > org.apache.giraph.utils.ProgressableUtils$ChannelFutureWaitable.waitFor(ProgressableUtils.java:461) > at > org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:214) > at > org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:180) > at > org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:165) > at > org.apache.giraph.utils.ProgressableUtils.awaitChannelFuture(ProgressableUtils.java:132) > at > org.apache.giraph.comm.netty.NettyClient.getNextChannel(NettyClient.java:715) > at > org.apache.giraph.comm.netty.NettyClient.writeRequestToChannel(NettyClient.java:799) > at org.apache.giraph.comm.netty.NettyClient.doSend(NettyClient.java:789) > at > org.apache.giraph.comm.flow_control.CreditBasedFlowControl.trySendCachedRequests(CreditBasedFlowControl.java:515) > at > org.apache.giraph.comm.flow_control.CreditBasedFlowControl.messageAckReceived(CreditBasedFlowControl.java:485) > at > org.apache.giraph.comm.netty.NettyClient.messageReceived(NettyClient.java:840) > at > org.apache.giraph.comm.netty.handler.ResponseClientHandler.channelRead(ResponseClientHandler.java:87) > at > io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338) > at > io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324) > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:153) > at > io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338) > at > io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324) > at > org.apache.giraph.comm.netty.InboundByteCounter.channelRead(InboundByteCounter.java:89) > at > io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338) > at > io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324) > at > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:785) > at > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:126) > at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:485) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:452) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:346) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (GIRAPH-1137) Remove channel probing from Netty worker thread for credit-based flow-control
[ https://issues.apache.org/jira/browse/GIRAPH-1137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15933077#comment-15933077 ] ASF GitHub Bot commented on GIRAPH-1137: Github user dlogothetis commented on the issue: https://github.com/apache/giraph/pull/26 There are a couple of checkstyle errors: giraph/giraph-core/target/munged/main/org/apache/giraph/comm/flow_control/CreditBasedFlowControl.java:48:8: Unused import - java.util.concurrent.LinkedBlockingQueue. giraph/giraph-core/target/munged/main/org/apache/giraph/comm/flow_control/CreditBasedFlowControl.java:172:3: Missing a Javadoc comment. > Remove channel probing from Netty worker thread for credit-based flow-control > - > > Key: GIRAPH-1137 > URL: https://issues.apache.org/jira/browse/GIRAPH-1137 > Project: Giraph > Issue Type: Bug >Reporter: Hassan Eslami >Assignee: Hassan Eslami > > In credit-based flow-control, sometimes, client threads (one type of Netty > worker threads used in Giraph) try to send requests to other workers. This is > bad practice for Netty and can cause Netty to mark the execution as > deadlock-prone (an example exception shown below). Client threads should only > be responsible for sending ACK/NACK messages in response to requests, and > they should do so by reuseing the channel from which they received the > request. In the current implementation, client threads may try to send > unsent/cached requests in credit-based flow control. Sending such requests > should be delegated to other threads. > WARN 2017-03-08 06:06:22,104 [netty-client-worker-3] > io.netty.util.concurrent.BlockingOperationException: > DefaultChannelPromise@2c455378(incomplete) > at > io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:383) > at > io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157) > at io.netty.util.concurrent.DefaultPromise.await0(DefaultPromise.java:343) > at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:259) > at > org.apache.giraph.utils.ProgressableUtils$ChannelFutureWaitable.waitFor(ProgressableUtils.java:461) > at > org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:214) > at > org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:180) > at > org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:165) > at > org.apache.giraph.utils.ProgressableUtils.awaitChannelFuture(ProgressableUtils.java:132) > at > org.apache.giraph.comm.netty.NettyClient.getNextChannel(NettyClient.java:715) > at > org.apache.giraph.comm.netty.NettyClient.writeRequestToChannel(NettyClient.java:799) > at org.apache.giraph.comm.netty.NettyClient.doSend(NettyClient.java:789) > at > org.apache.giraph.comm.flow_control.CreditBasedFlowControl.trySendCachedRequests(CreditBasedFlowControl.java:515) > at > org.apache.giraph.comm.flow_control.CreditBasedFlowControl.messageAckReceived(CreditBasedFlowControl.java:485) > at > org.apache.giraph.comm.netty.NettyClient.messageReceived(NettyClient.java:840) > at > org.apache.giraph.comm.netty.handler.ResponseClientHandler.channelRead(ResponseClientHandler.java:87) > at > io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338) > at > io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324) > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:153) > at > io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338) > at > io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324) > at > org.apache.giraph.comm.netty.InboundByteCounter.channelRead(InboundByteCounter.java:89) > at > io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338) > at > io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324) > at > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:785) > at > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:126) > at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:485) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:452) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:346) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
