[ https://issues.apache.org/jira/browse/LIVY-426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279720#comment-16279720 ]
Jialiang Tan commented on LIVY-426: ----------------------------------- Also I found out that when Livy has this problem on one user, it tends to kill other users sessions as well, and the same thing is no error message. > Livy stops session quietly without error message > ------------------------------------------------ > > Key: LIVY-426 > URL: https://issues.apache.org/jira/browse/LIVY-426 > Project: Livy > Issue Type: Bug > Reporter: Jialiang Tan > > Livy stops interactive sessions quietly without any error messages. In YARN > log it only shows RECEIVED SIGNAL TERM which I think indicates Livy wants to > terminate the driver in yarn. In Livy log it only gives Client RPC channel > closed unexpectedly. No other information found. > Spark Version: 2.0.1 > Scala Version: 2.11.8 > Livy Version: 0.4.0-incubating > Zeppelin Version: 0.7.2 > Hadoop Version: 2.7.3 > YARN log: > {code:java} > 17/12/06 00:39:23 INFO storage.BlockManagerMaster: Removed 3 successfully in > removeExecutor > 17/12/06 00:39:23 INFO spark.ExecutorAllocationManager: Existing executor 3 > has been removed (new total is 0) > 17/12/06 00:46:07 ERROR yarn.ApplicationMaster: RECEIVED SIGNAL TERM > 17/12/06 00:46:07 INFO spark.SparkContext: Invoking stop() from shutdown hook > 17/12/06 00:46:07 INFO server.ServerConnector: Stopped > ServerConnector@690ca6ae{HTTP/1.1}{0.0.0.0:0} > 17/12/06 00:46:07 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@5959d99{/stages/stage/kill,null,UNAVAILABLE} > 17/12/06 00:46:07 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@5daa4dcf{/api,null,UNAVAILABLE} > 17/12/06 00:46:07 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@1f30b9f2{/,null,UNAVAILABLE} > 17/12/06 00:46:07 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@1a36836{/static,null,UNAVAILABLE} > 17/12/06 00:46:07 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@56c1c4df{/executors/threadDump/json,null,UNAVAILABLE} > 17/12/06 00:46:07 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@5ea39f64{/executors/threadDump,null,UNAVAILABLE} > 17/12/06 00:46:07 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@5be8acbc{/executors/json,null,UNAVAILABLE} > 17/12/06 00:46:07 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@17a813b2{/executors,null,UNAVAILABLE} > 17/12/06 00:46:07 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@48780245{/environment/json,null,UNAVAILABLE} > 17/12/06 00:46:07 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@66bc6e53{/environment,null,UNAVAILABLE} > 17/12/06 00:46:07 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@7b06e14a{/storage/rdd/json,null,UNAVAILABLE} > 17/12/06 00:46:07 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@58f0ffff{/storage/rdd,null,UNAVAILABLE} > 17/12/06 00:46:07 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@4cfc52cb{/storage/json,null,UNAVAILABLE} > 17/12/06 00:46:07 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@731dd75e{/storage,null,UNAVAILABLE} > 17/12/06 00:46:07 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@1716986b{/stages/pool/json,null,UNAVAILABLE} > 17/12/06 00:46:07 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@54380417{/stages/pool,null,UNAVAILABLE} > 17/12/06 00:46:07 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@5b75d33{/stages/stage/json,null,UNAVAILABLE} > 17/12/06 00:46:07 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@24b81ae5{/stages/stage,null,UNAVAILABLE} > 17/12/06 00:46:07 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@68356b10{/stages/json,null,UNAVAILABLE} > 17/12/06 00:46:07 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@4422af95{/stages,null,UNAVAILABLE} > 17/12/06 00:46:07 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@39431dbc{/jobs/job/json,null,UNAVAILABLE} > 17/12/06 00:46:07 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@162b5d5e{/jobs/job,null,UNAVAILABLE} > 17/12/06 00:46:07 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@a84d7e6{/jobs/json,null,UNAVAILABLE} > 17/12/06 00:46:07 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@76d179b8{/jobs,null,UNAVAILABLE} > 17/12/06 00:46:07 INFO ui.SparkUI: Stopped Spark web UI at > http://10.28.24.141:33403 > {code} > And Livy log: > {code:java} > 17/12/06 00:46:07 WARN rsc.RSCClient: Client RPC channel closed unexpectedly. > 17/12/06 00:46:07 WARN rsc.RSCClient: Error stopping RPC. > io.netty.util.concurrent.BlockingOperationException: > DefaultChannelPromise@74e40000(uncancellable) > at > io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:390) > at > io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157) > at > io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:251) > at > io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:129) > at > io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:28) > at > io.netty.util.concurrent.DefaultPromise.sync(DefaultPromise.java:218) > at > io.netty.channel.DefaultChannelPromise.sync(DefaultChannelPromise.java:117) > at > io.netty.channel.DefaultChannelPromise.sync(DefaultChannelPromise.java:28) > at org.apache.livy.rsc.rpc.Rpc.close(Rpc.java:307) > at org.apache.livy.rsc.RSCClient.stop(RSCClient.java:232) > at org.apache.livy.rsc.RSCClient$2$1.onSuccess(RSCClient.java:129) > at org.apache.livy.rsc.RSCClient$2$1.onSuccess(RSCClient.java:123) > at org.apache.livy.rsc.Utils$2.operationComplete(Utils.java:108) > at > io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680) > at > io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567) > at > io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406) > at > io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82) > at > io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:956) > at > io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:608) > at > io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:586) > at > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71) > at > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:158) > at > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Unknown Source) > {code} > There is not enough debug information for this behavior and the behavior > happens occasionally. Sometimes the session got stopped immediately upon > startup. Other times the session can run properly for hours before it hit > this. I have changed some configurations in Livy, not sure if that is related: > livy.conf: > livy.spark.master = yarn > livy.spark.deploy-mode = cluster > livy.server.session.timeout = 24h > livy.impersonation.enabled = true > livy.file.local-dir-whitelist = /usr/lib/sparkExternalLibs > livy.repl.enable-hive-context = true > livy.server.yarn.app-lookup-timeout = 600s > livy-client.conf: > livy.rsc.server.connect.timeout = 360s > livy.rsc.client.connect.timeout = 120s > (I initially suspect it might be due to some rpc timeout issue and hence > increased the timeout, but nothing changed) > Not sure if anyone encountered similar issues and any workaround? -- This message was sent by Atlassian JIRA (v6.4.14#64029)