Hi Henning, The best place to continue discussion would be to file an issue in GitHub. This sounds like a bug or at least a place for improvement.
https://github.com/basho/riak-java-client/issues > How many active, busy connections does Riak KV support? You're correct that "it depends" is the right answer. In doing some benchmarks with the .NET client, I found that there was little benefit to the maximum number of connections exceeding the ring size in the cluster. This is probably specific to the benchmarks I was doing at the time, too. The best option is always to simulate your workload, tweak settings, and benchmark. -- Luke Bakken Engineer lbak...@basho.com On Wed, Jul 13, 2016 at 7:03 AM, Henning Verbeek <hankipa...@gmail.com> wrote: > I'm still struggling with a BlockingOperationException thrown by > riak-java-client 2.0.6, which occurs when I put heavy load on Riak KV. > Since https://github.com/basho/riak-java-client/issues/523 is fixed, > this happens only in - what I assume is - an overload-scenario. > > The exception: > > 2016-07-13 14:41:12.789 localhost [nioEventLoopGroup-2-2] ERROR > com.basho.riak.client.core.RiakNode - Operation onException() channel: > id:237445453 localhost:8087 {} > io.netty.util.concurrent.BlockingOperationException: > DefaultChannelPromise@77ccd827(incomplete) > at > io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:390) > at > io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157) > at > io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:251) > at > io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:129) > at > io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:28) > at > com.basho.riak.client.core.RiakNode.doGetConnection(RiakNode.java:697) > at > com.basho.riak.client.core.RiakNode.getConnection(RiakNode.java:656) > at com.basho.riak.client.core.RiakNode.execute(RiakNode.java:587) > at > com.basho.riak.client.core.DefaultNodeManager.executeOnNode(DefaultNodeManager.java:91) > at > com.basho.riak.client.core.RiakCluster.execute(RiakCluster.java:322) > at > com.basho.riak.client.core.RiakCluster.execute(RiakCluster.java:240) > at > com.basho.riak.client.api.commands.kv.StoreValue.executeAsync(StoreValue.java:117) > at > com.basho.riak.client.api.commands.kv.UpdateValue$1.handle(UpdateValue.java:182) > at > com.basho.riak.client.api.commands.ListenableFuture.notifyListeners(ListenableFuture.java:78) > at > com.basho.riak.client.api.commands.CoreFutureAdapter.handle(CoreFutureAdapter.java:120) > at > com.basho.riak.client.core.FutureOperation.fireListeners(FutureOperation.java:176) > at > com.basho.riak.client.core.FutureOperation.setComplete(FutureOperation.java:224) > at com.basho.riak.client.core.RiakNode.onSuccess(RiakNode.java:878) > at > com.basho.riak.client.core.netty.RiakResponseHandler.channelRead(RiakResponseHandler.java:30) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304) > at > io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:276) > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:263) > at > io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:103) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304) > at > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846) > at > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) > at > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112) > at > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) > at java.lang.Thread.run(Thread.java:745) > > Then shortly thereafter: > > 2016-07-13 14:41:12.820 localhost [nioEventLoopGroup-2-2] ERROR > com.basho.riak.client.core.RiakNode - Write failed on RiakNode > localhost:8087 id: 237445453; cause: {} > java.nio.channels.ClosedChannelException: null > 2016-07-13 14:41:12.843 localhost [nioEventLoopGroup-2-2] ERROR > com.basho.riak.client.core.RiakNode - Write failed on RiakNode > localhost:8087 id: 237445453; cause: {} > java.nio.channels.ClosedChannelException: null > > The result is a hanging thread, never returning from the call to > `RiakClient.execute()`. It would be great, if in such a case, the > client threw a RiakException (as javadoc says). > > But in the meantime, I'm trying to avoid getting into this scenario in > the first place, by sizing the maxConnections of RiakConnector and my > applications frontend threadpools appropriately. My current > observation is this: If I set RiakConnector.withMaxConnections() too > small for the given load (let's say 100), I get the > BlockingOperationException, although according to my (potentially > unprecise connection monitoring), the threshold is never reached. > Setting maxConnections to a higher value (let's say 160) lets the > loadtest run through fine, with around 110 active connections to Riak. > > It seems that in order to avoid the BlockingOperationException, I must > set the maxConnections limit higher than what I'll need. Maybe I > should skip maxConnections altogether then, and leave it at 0? But > then, how do I protect against an overload scenario? How many active, > busy connections does Riak KV support? I'm pretty sure the answer to > that is "it depends...". > > Thanks for your input. > Cheers, > Henning > -- > My other signature is a regular expression. > http://www.pray4snow.de > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
