How many sessions are you creating? Have you tried to throttle down session creation? What is the value for `livy.server.session.max-creation` in your setup? Also check that you are not running out of resources (particularly memory) on the Livy server node. Each session creation starts a new JVM process which can easily take a lot of memory.
Thanks, Meisam On Mon, Mar 19, 2018 at 8:15 PM Saisai Shao <sai.sai.s...@gmail.com> wrote: > This might be a BUG. If possible can you please create a JIRA to track this > issue. Thanks! > > Best, > Jerry > > 2018-03-19 20:18 GMT+08:00 Rao, Abhishek (Nokia - IN/Bangalore) < > abhishek....@nokia.com>: > > > Hi, > > > > We're trying to create multiple livy sessions in parallel and then using > > them. But when we try to create the sessions continuously, we're seeing > > that few sessions are entering to dead state. We see the below exception > in > > the logs. > > > > 18/02/27 10:30:20 WARN RSCClient: Client RPC channel closed unexpectedly. > > 18/02/27 10:30:20 WARN RSCClient: Error stopping RPC. > > io.netty.util.concurrent.BlockingOperationException: > > DefaultChannelPromise@7a828ea3(uncancellable) > > at io.netty.util.concurrent.DefaultPromise.checkDeadLock( > > DefaultPromise.java:390) > > at io.netty.channel.DefaultChannelPromise.checkDeadLock( > > DefaultChannelPromise.java:157) > > at io.netty.util.concurrent.DefaultPromise.await( > > DefaultPromise.java:251) > > at io.netty.channel.DefaultChannelPromise.await( > > DefaultChannelPromise.java:129) > > at io.netty.channel.DefaultChannelPromise.await( > > DefaultChannelPromise.java:28) > > at io.netty.util.concurrent.DefaultPromise.sync( > > DefaultPromise.java:218) > > at io.netty.channel.DefaultChannelPromise.sync( > > DefaultChannelPromise.java:117) > > at io.netty.channel.DefaultChannelPromise.sync( > > DefaultChannelPromise.java:28) > > at com.cloudera.livy.rsc.rpc.Rpc.close(Rpc.java:307) > > at > com.cloudera.livy.rsc.RSCClient.stop(RSCClient.java:225) > > at com.cloudera.livy.rsc.RSCClient$2$1.onSuccess( > > RSCClient.java:122) > > at com.cloudera.livy.rsc.RSCClient$2$1.onSuccess( > > RSCClient.java:116) > > at com.cloudera.livy.rsc.Utils$2. > > operationComplete(Utils.java:108) > > at > io.netty.util.concurrent.DefaultPromise.notifyListener0( > > DefaultPromise.java:680) > > at > io.netty.util.concurrent.DefaultPromise.notifyListeners( > > DefaultPromise.java:567) > > at io.netty.util.concurrent.DefaultPromise.trySuccess( > > DefaultPromise.java:406) > > at io.netty.channel.DefaultChannelPromise.trySuccess( > > DefaultChannelPromise.java:82) > > at io.netty.channel.AbstractChannel$CloseFuture. > > setClosed(AbstractChannel.java:956) > > at io.netty.channel.AbstractChannel$ > > AbstractUnsafe.doClose0(AbstractChannel.java:608) > > at io.netty.channel.AbstractChannel$AbstractUnsafe.close( > > AbstractChannel.java:586) > > at io.netty.channel.nio.AbstractNioByteChannel$ > > NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71) > > at io.netty.channel.nio.AbstractNioByteChannel$ > > NioByteUnsafe.read(AbstractNioByteChannel.java:158) > > at io.netty.channel.nio.NioEventLoop.processSelectedKey( > > NioEventLoop.java:511) > > at io.netty.channel.nio.NioEventLoop. > > processSelectedKeysOptimized(NioEventLoop.java:468) > > at io.netty.channel.nio.NioEventLoop.processSelectedKeys( > > NioEventLoop.java:382) > > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop. > > java:354) > > at io.netty.util.concurrent.SingleThreadEventExecutor$2. > > run(SingleThreadEventExecutor.java:111) > > at java.lang.Thread.run(Thread.java:748) > > 18/02/27 10:30:20 DEBUG RSCClient: Disconnected from context > > dad7c668-3c09-4ad2-9810-28f684c5ec49, shutdown = false. > > > > However, when we create the sessions one after the other (Create session > 1 > > after session 0 is in Idle state), it works fine. > > We wanted to know if there is any known restriction in livy for creating > > multiple sessions in parallel. > > > > Thanks & Regards, > > Abhishek > > > > >
