Query on creating multiple livy sessions in parallel

[Rao, Abhishek (Nokia - IN/Bangalore)]

Hi,

We're trying to create multiple livy sessions in parallel and then using them. 
But when we try to create the sessions continuously, we're seeing that few 
sessions are entering to dead state. We see the below exception in the logs.

18/02/27 10:30:20 WARN RSCClient: Client RPC channel closed unexpectedly.
18/02/27 10:30:20 WARN RSCClient: Error stopping RPC.
io.netty.util.concurrent.BlockingOperationException: 
DefaultChannelPromise@7a828ea3(uncancellable)
               at 
io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:390)
               at 
io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157)
               at 
io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:251)
               at 
io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:129)
               at 
io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:28)
               at 
io.netty.util.concurrent.DefaultPromise.sync(DefaultPromise.java:218)
               at 
io.netty.channel.DefaultChannelPromise.sync(DefaultChannelPromise.java:117)
               at 
io.netty.channel.DefaultChannelPromise.sync(DefaultChannelPromise.java:28)
               at com.cloudera.livy.rsc.rpc.Rpc.close(Rpc.java:307)
               at com.cloudera.livy.rsc.RSCClient.stop(RSCClient.java:225)
               at 
com.cloudera.livy.rsc.RSCClient$2$1.onSuccess(RSCClient.java:122)
               at 
com.cloudera.livy.rsc.RSCClient$2$1.onSuccess(RSCClient.java:116)
               at 
com.cloudera.livy.rsc.Utils$2.operationComplete(Utils.java:108)
               at 
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
               at 
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567)
               at 
io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
               at 
io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
               at 
io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:956)
               at 
io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:608)
               at 
io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:586)
               at 
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71)
               at 
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:158)
               at 
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
               at 
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
               at 
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
               at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
               at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
               at java.lang.Thread.run(Thread.java:748)
18/02/27 10:30:20 DEBUG RSCClient: Disconnected from context 
dad7c668-3c09-4ad2-9810-28f684c5ec49, shutdown = false.

However, when we create the sessions one after the other (Create session 1 
after session 0 is in Idle state), it works fine.
We wanted to know if there is any known restriction in livy for creating 
multiple sessions in parallel.

Thanks & Regards,
Abhishek