[ https://issues.apache.org/jira/browse/LIVY-436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16340823#comment-16340823 ]
Maziyar PANAHI edited comment on LIVY-436 at 1/26/18 10:01 AM: --------------------------------------------------------------- Hi Jeff, As I mention in the report, I did check the YARN app logs and there is nothing that indicates the session should be closed. In fact, I really don't know what kind of error/warning should close RPC channel unexpectedly. Do you think this is not related to Livy rather than the YARN/Spark2 issue? Update: The only think that indicates the session is closed is the error I can see in our Zeppelin notebook/logs (users report this error often and they have to re-run their code): {code:java} Job 20180125-121150_180855190 is finished, status: ERROR, exception: null, result: %text org.springframework.web.client.HttpServerErrorException: 500 Internal Server Error {code} was (Author: maziyar): Hi Jeff, As I mention in the report, I did check the YARN app logs and there is nothing that indicates the session should be closed. In fact, I really don't know what kind of error/warning should close RPC channel unexpectedly. Do you think this is not related to Livy rather than the YARN/Spark2 issue? > Client RPC channel closed unexpectedly > -------------------------------------- > > Key: LIVY-436 > URL: https://issues.apache.org/jira/browse/LIVY-436 > Project: Livy > Issue Type: Bug > Affects Versions: 0.4.0 > Environment: Cloudera CDH 5.14 > Spark 2.2 / Scala > Apache Livy 0.4.0 > Apache Zeppelin 0.7.3 > Ubuntu 16.04 LTS > Reporter: Maziyar PANAHI > Priority: Major > > Hi, > We are getting this error time to time and we have to re-create the session > and start the execution of our codes from the beginning. However, it is > really hard to find out why the error happens since it's not always the same > code nor the Spark commands. The session is suddenly closed without any WARN > or ERROR in different time with different code. > I checked the logs from YARN and Spark and there is nothing that indicates > the session was closed because of any error or warning before hand. > I have Livy server and Apache Zeppelin on the same machine so the IP address > is the same for both of them. > I am just wondering, what could cause this error in a multi-user environment. > (number of connections, open files, heap size, etc.). Is it the connection > unstable or is there some configs that could help preventing this sudden > death of sessions. > > Livy conf on Zeppelin: > > {noformat} > livy.spark.driver.cores 4 > livy.spark.driver.memory 4g > livy.spark.dynamicAllocation.enabled true > livy.spark.executor.cores 5 > livy.spark.executor.memory 5g > zeppelin.interpreter.output.limit 102400 > zeppelin.livy.concurrentSQL false > zeppelin.livy.displayAppInfo false > zeppelin.livy.pull_status.interval.millis 1000 > zeppelin.livy.session.create_timeout 120 > zeppelin.livy.spark.sql.maxResult 10000 > zeppelin.spark.printREPLOutput true > {noformat} > > {code:java} > 18/01/26 10:08:59 WARN RSCClient: Client RPC channel closed unexpectedly. > 18/01/26 10:08:59 WARN Rpc: [id: 0x94c07360, /Livy-IP-Address:57846 :> > executor-hostname/executor-IP-Address:10000] CLOSE() > 18/01/26 10:08:59 WARN RSCClient: Error stopping RPC. > io.netty.util.concurrent.BlockingOperationException: > DefaultChannelPromise@1062b60(uncancellable) > at > io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:390) > at > io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157) > at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:251) > at > io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:129) > at > io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:28) > at io.netty.util.concurrent.DefaultPromise.sync(DefaultPromise.java:218) > at > io.netty.channel.DefaultChannelPromise.sync(DefaultChannelPromise.java:117) > at io.netty.channel.DefaultChannelPromise.sync(DefaultChannelPromise.java:28) > at org.apache.livy.rsc.rpc.Rpc.close(Rpc.java:307) > at org.apache.livy.rsc.RSCClient.stop(RSCClient.java:232) > at org.apache.livy.rsc.RSCClient$2$1.onSuccess(RSCClient.java:129) > at org.apache.livy.rsc.RSCClient$2$1.onSuccess(RSCClient.java:123) > at org.apache.livy.rsc.Utils$2.operationComplete(Utils.java:108) > at > io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680) > at > io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567) > at > io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406) > at > io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82) > at > io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:956) > at > io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:608) > at > io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:586) > at > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71) > at > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:158) > at > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:748) > 18/01/26 10:08:59 WARN Rpc: [id: 0x94c07360,/Livy-IP-Address:57846 :> > executor-hostname/executor-IP-Address:10000] INACTIVE > 18/01/26 10:08:59 WARN Rpc: [id: 0x94c07360,/Livy-IP-Address:57846 :> > executor-hostname/executor-IP-Address:10000] UNREGISTERED > 18/01/26 10:09:09 ERROR SessionServlet$: internal error > java.lang.IllegalStateException: Session is in state dead > at > org.apache.livy.server.interactive.InteractiveSession.ensureRunning(InteractiveSession.scala:568) > at > org.apache.livy.server.interactive.InteractiveSession.executeStatement(InteractiveSession.scala:490) > at > org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$11$$anonfun$apply$6.apply(InteractiveSessionServlet.scala:124) > at > org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$11$$anonfun$apply$6.apply(InteractiveSessionServlet.scala:123) > at > org.apache.livy.server.interactive.SessionHeartbeatNotifier$$anonfun$withModifyAccessSession$1.apply(SessionHeartbeat.scala:76) > at > org.apache.livy.server.interactive.SessionHeartbeatNotifier$$anonfun$withModifyAccessSession$1.apply(SessionHeartbeat.scala:74) > at > org.apache.livy.server.SessionServlet.doWithSession(SessionServlet.scala:221) > at > org.apache.livy.server.SessionServlet.withModifyAccessSession(SessionServlet.scala:212) > at > org.apache.livy.server.interactive.InteractiveSessionServlet.org$apache$livy$server$interactive$SessionHeartbeatNotifier$$super$withModifyAccessSession(InteractiveSessionServlet.scala:40) > at > org.apache.livy.server.interactive.SessionHeartbeatNotifier$class.withModifyAccessSession(SessionHeartbeat.scala:74) > at > org.apache.livy.server.interactive.InteractiveSessionServlet.withModifyAccessSession(InteractiveSessionServlet.scala:40) > at > org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$11.apply(InteractiveSessionServlet.scala:123) > at > org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$11.apply(InteractiveSessionServlet.scala:122) > at > org.apache.livy.server.JsonServlet.org$apache$livy$server$JsonServlet$$doAction(JsonServlet.scala:113) > at > org.apache.livy.server.JsonServlet$$anonfun$jpost$1.apply(JsonServlet.scala:75) > at > org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$liftAction(ScalatraBase.scala:270) > at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265) > at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265) > at org.scalatra.ApiFormats$class.withRouteMultiParams(ApiFormats.scala:178) > at > org.apache.livy.server.JsonServlet.withRouteMultiParams(JsonServlet.scala:39) > at org.scalatra.ScalatraBase$class.invoke(ScalatraBase.scala:264) > at org.scalatra.ScalatraServlet.invoke(ScalatraServlet.scala:49) > at > org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.apply(ScalatraBase.scala:240) > at > org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.apply(ScalatraBase.scala:238) > at scala.Option.flatMap(Option.scala:170) > at > org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.scala:238) > at > org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.scala:237) > at scala.collection.immutable.Stream.flatMap(Stream.scala:446) > at org.scalatra.ScalatraBase$class.runRoutes(ScalatraBase.scala:237) > at org.scalatra.ScalatraServlet.runRoutes(ScalatraServlet.scala:49) > at org.scalatra.ScalatraBase$class.runActions$1(ScalatraBase.scala:163) > at > org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply$mcV$sp(ScalatraBase.scala:175) > at > org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBase.scala:175) > at > org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBase.scala:175) > at > org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$cradleHalt(ScalatraBase.scala:193) > at org.scalatra.ScalatraBase$class.executeRoutes(ScalatraBase.scala:175) > at org.scalatra.ScalatraServlet.executeRoutes(ScalatraServlet.scala:49) > at > org.scalatra.ScalatraBase$$anonfun$handle$1.apply$mcV$sp(ScalatraBase.scala:113) > at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala:113) > at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala:113) > at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) > at org.scalatra.DynamicScope$class.withResponse(DynamicScope.scala:80) > at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet.scala:49) > at > org.scalatra.DynamicScope$$anonfun$withRequestResponse$1.apply(DynamicScope.scala:60) > at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) > at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71) > at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49) > at org.scalatra.DynamicScope$class.withRequestResponse(DynamicScope.scala:59) > at org.scalatra.ScalatraServlet.withRequestResponse(ScalatraServlet.scala:49) > at org.scalatra.ScalatraBase$class.handle(ScalatraBase.scala:111) > at > org.scalatra.ScalatraServlet.org$scalatra$servlet$ServletBase$$super$handle(ScalatraServlet.scala:49) > at org.scalatra.servlet.ServletBase$class.handle(ServletBase.scala:43) > at > org.apache.livy.server.SessionServlet.org$scalatra$MethodOverride$$super$handle(SessionServlet.scala:39) > at org.scalatra.MethodOverride$class.handle(MethodOverride.scala:28) > at > org.apache.livy.server.SessionServlet.org$scalatra$GZipSupport$$super$handle(SessionServlet.scala:39) > at > org.scalatra.GZipSupport$$anonfun$handle$1.apply$mcV$sp(GZipSupport.scala:36) > at org.scalatra.GZipSupport$$anonfun$handle$1.apply(GZipSupport.scala:19) > at org.scalatra.GZipSupport$$anonfun$handle$1.apply(GZipSupport.scala:19) > at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) > at org.scalatra.DynamicScope$class.withResponse(DynamicScope.scala:80) > at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet.scala:49) > at > org.scalatra.DynamicScope$$anonfun$withRequestResponse$1.apply(DynamicScope.scala:60) > at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) > at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71) > at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49) > at org.scalatra.DynamicScope$class.withRequestResponse(DynamicScope.scala:59) > at org.scalatra.ScalatraServlet.withRequestResponse(ScalatraServlet.scala:49) > [7557/9847] > at org.scalatra.GZipSupport$class.handle(GZipSupport.scala:18) > at > org.apache.livy.server.interactive.InteractiveSessionServlet.org$scalatra$servlet$FileUploadSupport$$super$handle(InteractiveSessionServlet.scala:40) > at > org.scalatra.servlet.FileUploadSupport$class.handle(FileUploadSupport.scala:93) > at > org.apache.livy.server.interactive.InteractiveSessionServlet.handle(InteractiveSessionServlet.scala:40) > at org.scalatra.ScalatraServlet.service(ScalatraServlet.scala:54) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) > at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) > at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) > at org.eclipse.jetty.server.Server.handle(Server.java:499) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) > at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) > at java.lang.Thread.run(Thread.java:748) > 18/01/26 10:09:11 ERROR SessionServlet$: internal error > java.lang.IllegalStateException: Session is in state dead > at > org.apache.livy.server.interactive.InteractiveSession.ensureRunning(InteractiveSession.scala:568) > at > org.apache.livy.server.interactive.InteractiveSession.executeStatement(InteractiveSession.scala:490) > at > org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$11$$anonfun$apply$6.apply(InteractiveSessionServlet.scala:124) > at > org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$11$$anonfun$apply$6.apply(InteractiveSessionServlet.scala:123) > at > org.apache.livy.server.interactive.SessionHeartbeatNotifier$$anonfun$withModifyAccessSession$1.apply(SessionHeartbeat.scala:76) > at > org.apache.livy.server.interactive.SessionHeartbeatNotifier$$anonfun$withModifyAccessSession$1.apply(SessionHeartbeat.scala:74) > at > org.apache.livy.server.SessionServlet.doWithSession(SessionServlet.scala:221) > at > org.apache.livy.server.SessionServlet.withModifyAccessSession(SessionServlet.scala:212) > at > org.apache.livy.server.interactive.InteractiveSessionServlet.org$apache$livy$server$interactive$SessionHeartbeatNotifier$$super$withModifyAccessSession(InteractiveSessionServlet.scala:40) > at > org.apache.livy.server.interactive.SessionHeartbeatNotifier$class.withModifyAccessSession(SessionHeartbeat.scala:74) > at > org.apache.livy.server.interactive.InteractiveSessionServlet.withModifyAccessSession(InteractiveSessionServlet.scala:40) > at > org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$11.apply(InteractiveSessionServlet.scala:123) > at > org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$11.apply(InteractiveSessionServlet.scala:122) > at > org.apache.livy.server.JsonServlet.org$apache$livy$server$JsonServlet$$doAction(JsonServlet.scala:113) > at > org.apache.livy.server.JsonServlet$$anonfun$jpost$1.apply(JsonServlet.scala:75) > at > org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$liftAction(ScalatraBase.scala:270) > at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265) > at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265) > at org.scalatra.ApiFormats$class.withRouteMultiParams(ApiFormats.scala:178) > at > org.apache.livy.server.JsonServlet.withRouteMultiParams(JsonServlet.scala:39) > at org.scalatra.ScalatraBase$class.invoke(ScalatraBase.scala:264) > at org.scalatra.ScalatraServlet.invoke(ScalatraServlet.scala:49) > at > org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.apply(ScalatraBase.scala:240) > at > org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.apply(ScalatraBase.scala:238) > at scala.Option.flatMap(Option.scala:170) > at > org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.scala:238) > at > org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.scala:237) > at scala.collection.immutable.Stream.flatMap(Stream.scala:446) > at org.scalatra.ScalatraBase$class.runRoutes(ScalatraBase.scala:237) > at org.scalatra.ScalatraServlet.runRoutes(ScalatraServlet.scala:49) > at org.scalatra.ScalatraBase$class.runActions$1(ScalatraBase.scala:163) > at > org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply$mcV$sp(ScalatraBase.scala:175) > at > org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBase.scala:175) > at > org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBase.scala:175) > at > org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$cradleHalt(ScalatraBase.scala:193) > at org.scalatra.ScalatraBase$class.executeRoutes(ScalatraBase.scala:175) > at org.scalatra.ScalatraServlet.executeRoutes(ScalatraServlet.scala:49) > at > org.scalatra.ScalatraBase$$anonfun$handle$1.apply$mcV$sp(ScalatraBase.scala:113) > at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala:113) > at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala:113) > at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) > at org.scalatra.DynamicScope$class.withResponse(DynamicScope.scala:80) > at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet.scala:49) > at > org.scalatra.DynamicScope$$anonfun$withRequestResponse$1.apply(DynamicScope.scala:60) > at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) > at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71) > at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49) > at org.scalatra.DynamicScope$class.withRequestResponse(DynamicScope.scala:59) > at org.scalatra.ScalatraServlet.withRequestResponse(ScalatraServlet.scala:49) > at org.scalatra.ScalatraBase$class.handle(ScalatraBase.scala:111) > at > org.scalatra.ScalatraServlet.org$scalatra$servlet$ServletBase$$super$handle(ScalatraServlet.scala:49) > at org.scalatra.servlet.ServletBase$class.handle(ServletBase.scala:43) > at > org.apache.livy.server.SessionServlet.org$scalatra$MethodOverride$$super$handle(SessionServlet.scala:39) > at org.scalatra.MethodOverride$class.handle(MethodOverride.scala:28) > at > org.apache.livy.server.SessionServlet.org$scalatra$GZipSupport$$super$handle(SessionServlet.scala:39) > at > org.scalatra.GZipSupport$$anonfun$handle$1.apply$mcV$sp(GZipSupport.scala:36) > at org.scalatra.GZipSupport$$anonfun$handle$1.apply(GZipSupport.scala:19) > at org.scalatra.GZipSupport$$anonfun$handle$1.apply(GZipSupport.scala:19) > at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) > at org.scalatra.DynamicScope$class.withResponse(DynamicScope.scala:80) > at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet.scala:49) > at > org.scalatra.DynamicScope$$anonfun$withRequestResponse$1.apply(DynamicScope.scala:60) > at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) > at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71) > at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49) > at org.scalatra.DynamicScope$class.withRequestResponse(DynamicScope.scala:59) > at org.scalatra.ScalatraServlet.withRequestResponse(ScalatraServlet.scala:49) > at org.scalatra.GZipSupport$class.handle(GZipSupport.scala:18) > at > org.apache.livy.server.interactive.InteractiveSessionServlet.org$scalatra$servlet$FileUploadSupport$$super$handle(InteractiveSessionServlet.scala:40) > at > org.scalatra.servlet.FileUploadSupport$class.handle(FileUploadSupport.scala:93) > at > org.apache.livy.server.interactive.InteractiveSessionServlet.handle(InteractiveSessionServlet.scala:40) > at org.scalatra.ScalatraServlet.service(ScalatraServlet.scala:54) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) > at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) > at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) > at org.eclipse.jetty.server.Server.handle(Server.java:499) > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) > at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) > at java.lang.Thread.run(Thread.java:748) > {code} > > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)