RE: Query on creating multiple livy sessions in parallel
Hi Meisam, Comments below. How many sessions are you creating? Abhi> We had tried creating around 5-8 sessions when we start seeing the below exception. Have you tried to throttle down session creation? Abhi> We do not have any issue if we create sessions one after the other. What is the value for `livy.server.session.max-creation` in your setup? Abhi> We're not setting any value. So it should be using default. I guess it is 100. Also check that you are not running out of resources (particularly memory) on the Livy server node. Each session creation starts a new JVM process which can easily take a lot of memory. Abhi> We did not see any out of memory errors during our tests. But just out of curiosity, we tried to create sessions in an infinite loop. After creating around 35 sessions, we’ve started seeing out of memory errors. Hi Jerry, We've raised the ticket LIVY-453 for this. Thanks & Regards, Abhishek -Original Message- From: Meisam Fathi [mailto:meisam.fa...@gmail.com] Sent: Wednesday, March 21, 2018 5:41 AM To: dev@livy.incubator.apache.org Subject: Re: Query on creating multiple livy sessions in parallel How many sessions are you creating? Have you tried to throttle down session creation? What is the value for `livy.server.session.max-creation` in your setup? Also check that you are not running out of resources (particularly memory) on the Livy server node. Each session creation starts a new JVM process which can easily take a lot of memory. Thanks, Meisam On Mon, Mar 19, 2018 at 8:15 PM Saisai Shao mailto:sai.sai.s...@gmail.com>> wrote: > This might be a BUG. If possible can you please create a JIRA to track > this issue. Thanks! > > Best, > Jerry > > 2018-03-19 20:18 GMT+08:00 Rao, Abhishek (Nokia - IN/Bangalore) < > abhishek@nokia.com<mailto:abhishek@nokia.com>>: > > > Hi, > > > > We're trying to create multiple livy sessions in parallel and then > > using them. But when we try to create the sessions continuously, > > we're seeing that few sessions are entering to dead state. We see > > the below exception > in > > the logs. > > > > 18/02/27 10:30:20 WARN RSCClient: Client RPC channel closed unexpectedly. > > 18/02/27 10:30:20 WARN RSCClient: Error stopping RPC. > > io.netty.util.concurrent.BlockingOperationException: > > DefaultChannelPromise@7a828ea3(uncancellable) > >at > > io.netty.util.concurrent.DefaultPromise.checkDeadLock( > > DefaultPromise.java:390) > >at > > io.netty.channel.DefaultChannelPromise.checkDeadLock( > > DefaultChannelPromise.java:157) > >at io.netty.util.concurrent.DefaultPromise.await( > > DefaultPromise.java:251) > >at io.netty.channel.DefaultChannelPromise.await( > > DefaultChannelPromise.java:129) > >at io.netty.channel.DefaultChannelPromise.await( > > DefaultChannelPromise.java:28) > >at io.netty.util.concurrent.DefaultPromise.sync( > > DefaultPromise.java:218) > >at io.netty.channel.DefaultChannelPromise.sync( > > DefaultChannelPromise.java:117) > >at io.netty.channel.DefaultChannelPromise.sync( > > DefaultChannelPromise.java:28) > >at com.cloudera.livy.rsc.rpc.Rpc.close(Rpc.java:307) > >at > com.cloudera.livy.rsc.RSCClient.stop(RSCClient.java:225) > >at com.cloudera.livy.rsc.RSCClient$2$1.onSuccess( > > RSCClient.java:122) > >at com.cloudera.livy.rsc.RSCClient$2$1.onSuccess( > > RSCClient.java:116) > >at com.cloudera.livy.rsc.Utils$2. > > operationComplete(Utils.java:108) > >at > io.netty.util.concurrent.DefaultPromise.notifyListener0( > > DefaultPromise.java:680) > >at > io.netty.util.concurrent.DefaultPromise.notifyListeners( > > DefaultPromise.java:567) > >at > > io.netty.util.concurrent.DefaultPromise.trySuccess( > > DefaultPromise.java:406) > >at io.netty.channel.DefaultChannelPromise.trySuccess( > > DefaultChannelPromise.java:82) > >at io.netty.channel.AbstractChannel$CloseFuture. > > setClosed(AbstractChannel.java:956) > >at io.netty.channel.AbstractChannel$ > > AbstractUnsafe.doClose0(AbstractChannel.java:608) > >at > > io.netty.channel.AbstractChannel$AbstractUnsafe.close( > > AbstractChannel.java:586) > >at io.netty.channel.nio.AbstractNioByteChannel$ > > NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71) > >at io.netty.channe
Re: Query on creating multiple livy sessions in parallel
How many sessions are you creating? Have you tried to throttle down session creation? What is the value for `livy.server.session.max-creation` in your setup? Also check that you are not running out of resources (particularly memory) on the Livy server node. Each session creation starts a new JVM process which can easily take a lot of memory. Thanks, Meisam On Mon, Mar 19, 2018 at 8:15 PM Saisai Shao wrote: > This might be a BUG. If possible can you please create a JIRA to track this > issue. Thanks! > > Best, > Jerry > > 2018-03-19 20:18 GMT+08:00 Rao, Abhishek (Nokia - IN/Bangalore) < > abhishek@nokia.com>: > > > Hi, > > > > We're trying to create multiple livy sessions in parallel and then using > > them. But when we try to create the sessions continuously, we're seeing > > that few sessions are entering to dead state. We see the below exception > in > > the logs. > > > > 18/02/27 10:30:20 WARN RSCClient: Client RPC channel closed unexpectedly. > > 18/02/27 10:30:20 WARN RSCClient: Error stopping RPC. > > io.netty.util.concurrent.BlockingOperationException: > > DefaultChannelPromise@7a828ea3(uncancellable) > >at io.netty.util.concurrent.DefaultPromise.checkDeadLock( > > DefaultPromise.java:390) > >at io.netty.channel.DefaultChannelPromise.checkDeadLock( > > DefaultChannelPromise.java:157) > >at io.netty.util.concurrent.DefaultPromise.await( > > DefaultPromise.java:251) > >at io.netty.channel.DefaultChannelPromise.await( > > DefaultChannelPromise.java:129) > >at io.netty.channel.DefaultChannelPromise.await( > > DefaultChannelPromise.java:28) > >at io.netty.util.concurrent.DefaultPromise.sync( > > DefaultPromise.java:218) > >at io.netty.channel.DefaultChannelPromise.sync( > > DefaultChannelPromise.java:117) > >at io.netty.channel.DefaultChannelPromise.sync( > > DefaultChannelPromise.java:28) > >at com.cloudera.livy.rsc.rpc.Rpc.close(Rpc.java:307) > >at > com.cloudera.livy.rsc.RSCClient.stop(RSCClient.java:225) > >at com.cloudera.livy.rsc.RSCClient$2$1.onSuccess( > > RSCClient.java:122) > >at com.cloudera.livy.rsc.RSCClient$2$1.onSuccess( > > RSCClient.java:116) > >at com.cloudera.livy.rsc.Utils$2. > > operationComplete(Utils.java:108) > >at > io.netty.util.concurrent.DefaultPromise.notifyListener0( > > DefaultPromise.java:680) > >at > io.netty.util.concurrent.DefaultPromise.notifyListeners( > > DefaultPromise.java:567) > >at io.netty.util.concurrent.DefaultPromise.trySuccess( > > DefaultPromise.java:406) > >at io.netty.channel.DefaultChannelPromise.trySuccess( > > DefaultChannelPromise.java:82) > >at io.netty.channel.AbstractChannel$CloseFuture. > > setClosed(AbstractChannel.java:956) > >at io.netty.channel.AbstractChannel$ > > AbstractUnsafe.doClose0(AbstractChannel.java:608) > >at io.netty.channel.AbstractChannel$AbstractUnsafe.close( > > AbstractChannel.java:586) > >at io.netty.channel.nio.AbstractNioByteChannel$ > > NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71) > >at io.netty.channel.nio.AbstractNioByteChannel$ > > NioByteUnsafe.read(AbstractNioByteChannel.java:158) > >at io.netty.channel.nio.NioEventLoop.processSelectedKey( > > NioEventLoop.java:511) > >at io.netty.channel.nio.NioEventLoop. > > processSelectedKeysOptimized(NioEventLoop.java:468) > >at io.netty.channel.nio.NioEventLoop.processSelectedKeys( > > NioEventLoop.java:382) > >at io.netty.channel.nio.NioEventLoop.run(NioEventLoop. > > java:354) > >at io.netty.util.concurrent.SingleThreadEventExecutor$2. > > run(SingleThreadEventExecutor.java:111) > >at java.lang.Thread.run(Thread.java:748) > > 18/02/27 10:30:20 DEBUG RSCClient: Disconnected from context > > dad7c668-3c09-4ad2-9810-28f684c5ec49, shutdown = false. > > > > However, when we create the sessions one after the other (Create session > 1 > > after session 0 is in Idle state), it works fine. > > We wanted to know if there is any known restriction in livy for creating > > multiple sessions in parallel. > > > > Thanks & Regards, > > Abhishek > > > > >
Re: Query on creating multiple livy sessions in parallel
This might be a BUG. If possible can you please create a JIRA to track this issue. Thanks! Best, Jerry 2018-03-19 20:18 GMT+08:00 Rao, Abhishek (Nokia - IN/Bangalore) < abhishek@nokia.com>: > Hi, > > We're trying to create multiple livy sessions in parallel and then using > them. But when we try to create the sessions continuously, we're seeing > that few sessions are entering to dead state. We see the below exception in > the logs. > > 18/02/27 10:30:20 WARN RSCClient: Client RPC channel closed unexpectedly. > 18/02/27 10:30:20 WARN RSCClient: Error stopping RPC. > io.netty.util.concurrent.BlockingOperationException: > DefaultChannelPromise@7a828ea3(uncancellable) >at io.netty.util.concurrent.DefaultPromise.checkDeadLock( > DefaultPromise.java:390) >at io.netty.channel.DefaultChannelPromise.checkDeadLock( > DefaultChannelPromise.java:157) >at io.netty.util.concurrent.DefaultPromise.await( > DefaultPromise.java:251) >at io.netty.channel.DefaultChannelPromise.await( > DefaultChannelPromise.java:129) >at io.netty.channel.DefaultChannelPromise.await( > DefaultChannelPromise.java:28) >at io.netty.util.concurrent.DefaultPromise.sync( > DefaultPromise.java:218) >at io.netty.channel.DefaultChannelPromise.sync( > DefaultChannelPromise.java:117) >at io.netty.channel.DefaultChannelPromise.sync( > DefaultChannelPromise.java:28) >at com.cloudera.livy.rsc.rpc.Rpc.close(Rpc.java:307) >at com.cloudera.livy.rsc.RSCClient.stop(RSCClient.java:225) >at com.cloudera.livy.rsc.RSCClient$2$1.onSuccess( > RSCClient.java:122) >at com.cloudera.livy.rsc.RSCClient$2$1.onSuccess( > RSCClient.java:116) >at com.cloudera.livy.rsc.Utils$2. > operationComplete(Utils.java:108) >at io.netty.util.concurrent.DefaultPromise.notifyListener0( > DefaultPromise.java:680) >at io.netty.util.concurrent.DefaultPromise.notifyListeners( > DefaultPromise.java:567) >at io.netty.util.concurrent.DefaultPromise.trySuccess( > DefaultPromise.java:406) >at io.netty.channel.DefaultChannelPromise.trySuccess( > DefaultChannelPromise.java:82) >at io.netty.channel.AbstractChannel$CloseFuture. > setClosed(AbstractChannel.java:956) >at io.netty.channel.AbstractChannel$ > AbstractUnsafe.doClose0(AbstractChannel.java:608) >at io.netty.channel.AbstractChannel$AbstractUnsafe.close( > AbstractChannel.java:586) >at io.netty.channel.nio.AbstractNioByteChannel$ > NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71) >at io.netty.channel.nio.AbstractNioByteChannel$ > NioByteUnsafe.read(AbstractNioByteChannel.java:158) >at io.netty.channel.nio.NioEventLoop.processSelectedKey( > NioEventLoop.java:511) >at io.netty.channel.nio.NioEventLoop. > processSelectedKeysOptimized(NioEventLoop.java:468) >at io.netty.channel.nio.NioEventLoop.processSelectedKeys( > NioEventLoop.java:382) >at io.netty.channel.nio.NioEventLoop.run(NioEventLoop. > java:354) >at io.netty.util.concurrent.SingleThreadEventExecutor$2. > run(SingleThreadEventExecutor.java:111) >at java.lang.Thread.run(Thread.java:748) > 18/02/27 10:30:20 DEBUG RSCClient: Disconnected from context > dad7c668-3c09-4ad2-9810-28f684c5ec49, shutdown = false. > > However, when we create the sessions one after the other (Create session 1 > after session 0 is in Idle state), it works fine. > We wanted to know if there is any known restriction in livy for creating > multiple sessions in parallel. > > Thanks & Regards, > Abhishek > >
