Thanks, Luke! https://github.com/basho/riak-java-client/issues/638
On Thu, Jul 14, 2016 at 2:42 PM, Luke Bakken <lbak...@basho.com> wrote: > Hi Henning, > > The best place to continue discussion would be to file an issue in > GitHub. This sounds like a bug or at least a place for improvement. > > https://github.com/basho/riak-java-client/issues > >> How many active, busy connections does Riak KV support? > > You're correct that "it depends" is the right answer. In doing some > benchmarks with the .NET client, I found that there was little benefit > to the maximum number of connections exceeding the ring size in the > cluster. This is probably specific to the benchmarks I was doing at > the time, too. The best option is always to simulate your workload, > tweak settings, and benchmark. > > -- > Luke Bakken > Engineer > lbak...@basho.com > > > On Wed, Jul 13, 2016 at 7:03 AM, Henning Verbeek <hankipa...@gmail.com> wrote: >> I'm still struggling with a BlockingOperationException thrown by >> riak-java-client 2.0.6, which occurs when I put heavy load on Riak KV. >> Since https://github.com/basho/riak-java-client/issues/523 is fixed, >> this happens only in - what I assume is - an overload-scenario. >> >> The exception: >> >> 2016-07-13 14:41:12.789 localhost [nioEventLoopGroup-2-2] ERROR >> com.basho.riak.client.core.RiakNode - Operation onException() channel: >> id:237445453 localhost:8087 {} >> io.netty.util.concurrent.BlockingOperationException: >> DefaultChannelPromise@77ccd827(incomplete) >> at >> io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:390) >> at >> io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157) >> at >> io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:251) >> at >> io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:129) >> at >> io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:28) >> at >> com.basho.riak.client.core.RiakNode.doGetConnection(RiakNode.java:697) >> at >> com.basho.riak.client.core.RiakNode.getConnection(RiakNode.java:656) >> at com.basho.riak.client.core.RiakNode.execute(RiakNode.java:587) >> at >> com.basho.riak.client.core.DefaultNodeManager.executeOnNode(DefaultNodeManager.java:91) >> at >> com.basho.riak.client.core.RiakCluster.execute(RiakCluster.java:322) >> at >> com.basho.riak.client.core.RiakCluster.execute(RiakCluster.java:240) >> at >> com.basho.riak.client.api.commands.kv.StoreValue.executeAsync(StoreValue.java:117) >> at >> com.basho.riak.client.api.commands.kv.UpdateValue$1.handle(UpdateValue.java:182) >> at >> com.basho.riak.client.api.commands.ListenableFuture.notifyListeners(ListenableFuture.java:78) >> at >> com.basho.riak.client.api.commands.CoreFutureAdapter.handle(CoreFutureAdapter.java:120) >> at >> com.basho.riak.client.core.FutureOperation.fireListeners(FutureOperation.java:176) >> at >> com.basho.riak.client.core.FutureOperation.setComplete(FutureOperation.java:224) >> at com.basho.riak.client.core.RiakNode.onSuccess(RiakNode.java:878) >> at >> com.basho.riak.client.core.netty.RiakResponseHandler.channelRead(RiakResponseHandler.java:30) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304) >> at >> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:276) >> at >> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:263) >> at >> io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:103) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304) >> at >> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846) >> at >> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) >> at >> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) >> at >> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) >> at >> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) >> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) >> at >> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112) >> at >> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) >> at java.lang.Thread.run(Thread.java:745) >> >> Then shortly thereafter: >> >> 2016-07-13 14:41:12.820 localhost [nioEventLoopGroup-2-2] ERROR >> com.basho.riak.client.core.RiakNode - Write failed on RiakNode >> localhost:8087 id: 237445453; cause: {} >> java.nio.channels.ClosedChannelException: null >> 2016-07-13 14:41:12.843 localhost [nioEventLoopGroup-2-2] ERROR >> com.basho.riak.client.core.RiakNode - Write failed on RiakNode >> localhost:8087 id: 237445453; cause: {} >> java.nio.channels.ClosedChannelException: null >> >> The result is a hanging thread, never returning from the call to >> `RiakClient.execute()`. It would be great, if in such a case, the >> client threw a RiakException (as javadoc says). >> >> But in the meantime, I'm trying to avoid getting into this scenario in >> the first place, by sizing the maxConnections of RiakConnector and my >> applications frontend threadpools appropriately. My current >> observation is this: If I set RiakConnector.withMaxConnections() too >> small for the given load (let's say 100), I get the >> BlockingOperationException, although according to my (potentially >> unprecise connection monitoring), the threshold is never reached. >> Setting maxConnections to a higher value (let's say 160) lets the >> loadtest run through fine, with around 110 active connections to Riak. >> >> It seems that in order to avoid the BlockingOperationException, I must >> set the maxConnections limit higher than what I'll need. Maybe I >> should skip maxConnections altogether then, and leave it at 0? But >> then, how do I protect against an overload scenario? How many active, >> busy connections does Riak KV support? I'm pretty sure the answer to >> that is "it depends...". >> >> Thanks for your input. >> Cheers, >> Henning >> -- >> My other signature is a regular expression. >> http://www.pray4snow.de >> >> _______________________________________________ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com -- My other signature is a regular expression. http://www.pray4snow.de _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
