>From this line : Removing executor app-20141015142644-0125/0 because it is EXITED I would guess that you need to examine the executor log to see why the executor actually exited. My guess would be that the executor cannot connect back to your driver. But check the log from the executor. It should be in SPARK_HOME/work/app-id/executor_id/stderr on the worker box, I believe.
On Wed, Oct 15, 2014 at 8:56 AM, Mehdi Singer <mehdi.sin...@lampiris.be> wrote: > Hi, > > > > I have a Spark standalone example application which is working fine. > > I'm now trying to integrate this application into a J2EE application, > deployed on JBoss 7.1.1 and accessed via a web service. The JBoss server is > installed on my local machine (Windows 7) and the master Spark is remote > (Linux). > > The example simply executes a count on my RDD. > > When I call the webservice I'm getting the following error at JBoss side > when executing the count: > > > > 11:48:10,232 ERROR > [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/el2-etrm-spark].[ws]] > (http--127.0.0.1-8082-3) "Servlet.service()" pour la servlet ws a généré > une exception: java.lang.RuntimeException: > org.apache.cxf.interceptor.Fault: Job cancelled because SparkContext was > shut down > > at > org.apache.cxf.interceptor.AbstractFaultChainInitiatorObserver.onMessage(AbstractFaultChainInitiatorObserver.java:116) > [cxf-api-2.6.9.jar:2.6.9] > > at > org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:322) > [cxf-api-2.6.9.jar:2.4.3] > > at > org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) > [cxf-api-2.6.9.jar:2.6.9] > > at > org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:211) > [cxf-bundle-2.6.2.jar:2.6.2] > > at > org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:213) > [cxf-bundle-2.6.2.jar:2.6.2] > > at > org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:154) > [cxf-bundle-2.6.2.jar:2.6.2] > > at > org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:130) > [cxf-bundle-2.6.2.jar:2.6.2] > > at > org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:221) > [cxf-bundle-2.6.2.jar:2.6.2] > > at > org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:146) > [cxf-bundle-2.6.2.jar:2.6.2] > > at > javax.servlet.http.HttpServlet.service(HttpServlet.java:734) > [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] > > at > org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:197) > [cxf-bundle-2.6.2.jar:2.6.2] > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) > [jbossweb-7.0.13.Final.jar:] > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > [jbossweb-7.0.13.Final.jar:] > > at > org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.doFilterInternal(OpenEntityManagerInViewFilter.java:180) > [spring-orm-3.2.3.RELEASE.jar:3.2.3.RELEASE] > > at > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) > [spring-web-3.2.3.RELEASE.jar:3.2.3.RELEASE] > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) > [jbossweb-7.0.13.Final.jar:] > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > [jbossweb-7.0.13.Final.jar:] > > at > org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:186) > [spring-security-web-3.1.3.RELEASE.jar:3.1.3.RELEASE] > > at > org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) > [spring-security-web-3.1.3.RELEASE.jar:3.1.3.RELEASE] > > at > org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) > [spring-web-3.2.3.RELEASE.jar:3.2.3.RELEASE] > > at > org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259) > [spring-web-3.2.3.RELEASE.jar:3.2.3.RELEASE] > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) > [jbossweb-7.0.13.Final.jar:] > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > [jbossweb-7.0.13.Final.jar:] > > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) > [jbossweb-7.0.13.Final.jar:] > > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) > [jbossweb-7.0.13.Final.jar:] > > at > org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) > [jboss-as-jpa-7.1.1.Final.jar:7.1.1.Final] > > at > org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) > [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] > > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) > [jbossweb-7.0.13.Final.jar:] > > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > [jbossweb-7.0.13.Final.jar:] > > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > [jbossweb-7.0.13.Final.jar:] > > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) > [jbossweb-7.0.13.Final.jar:] > > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) > [jbossweb-7.0.13.Final.jar:] > > at > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) > [jbossweb-7.0.13.Final.jar:] > > at > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) > [jbossweb-7.0.13.Final.jar:] > > at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25] > > Caused by: org.apache.cxf.interceptor.Fault: Job cancelled because > SparkContext was shut down > > at > org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:162) > [cxf-api-2.6.9.jar:2.6.9] > > at > org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:128) > [cxf-api-2.6.9.jar:2.6.9] > > at > org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:167) > [cxf-bundle-2.6.2.jar:2.6.2] > > at > org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:94) > [cxf-bundle-2.6.2.jar:2.6.2] > > at > org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57) > [cxf-api-2.6.9.jar:2.6.9] > > at > org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93) > [cxf-api-2.6.9.jar:2.6.9] > > at > org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:262) > [cxf-api-2.6.9.jar:2.4.3] > > ... 33 more > > Caused by: org.apache.spark.SparkException: Job cancelled because > SparkContext was shut down > > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:694) > [spark-core_2.10-1.1.0.jar:1.1.0] > > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:693) > [spark-core_2.10-1.1.0.jar:1.1.0] > > at > scala.collection.mutable.HashSet.foreach(HashSet.scala:79) > [scala-library-2.10.4.jar:] > > at > org.apache.spark.scheduler.DAGScheduler.cleanUpAfterSchedulerStop(DAGScheduler.scala:693) > [spark-core_2.10-1.1.0.jar:1.1.0] > > at > org.apache.spark.scheduler.DAGSchedulerEventProcessActor.postStop(DAGScheduler.scala:1399) > [spark-core_2.10-1.1.0.jar:1.1.0] > > at > akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:201) > [akka-actor_2.10-2.2.3-shaded-protobuf.jar:] > > at > akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:163) > [akka-actor_2.10-2.2.3-shaded-protobuf.jar:] > > at akka.actor.ActorCell.terminate(ActorCell.scala:338) > [akka-actor_2.10-2.2.3-shaded-protobuf.jar:] > > at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:431) > [akka-actor_2.10-2.2.3-shaded-protobuf.jar:] > > at akka.actor.ActorCell.systemInvoke(ActorCell.scala:447) > [akka-actor_2.10-2.2.3-shaded-protobuf.jar:] > > at > akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:262) > [akka-actor_2.10-2.2.3-shaded-protobuf.jar:] > > at akka.dispatch.Mailbox.run(Mailbox.scala:218) > [akka-actor_2.10-2.2.3-shaded-protobuf.jar:] > > at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) > [akka-actor_2.10-2.2.3-shaded-protobuf.jar:] > > at > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > [scala-library-2.10.4.jar:] > > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > [scala-library-2.10.4.jar:] > > at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > [scala-library-2.10.4.jar:] > > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > [scala-library-2.10.4.jar:] > > > > On Spark master side I'm getting the following output: > > > > 14/10/15 11:48:10 INFO Master: akka.tcp://sparkDriver@mymachinename:51834 > got disassociated, removing it. > > 14/10/15 11:48:10 INFO Master: akka.tcp://sparkDriver@mymachinename:51834 > got disassociated, removing it. > > 14/10/15 11:48:10 INFO LocalActorRef: Message > [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from > Actor[akka://sparkMaster/deadLetters] to > Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.21.6.4%3A51850-649#254460547] > was not delivered. [215] dead letters encountered. This logging can be > turned off or adjusted with configuration settings 'akka.log-dead-letters' > and 'akka.log-dead-letters-during-shutdown'. > > 14/10/15 11:48:10 ERROR EndpointWriter: AssociationError > [akka.tcp://sparkMaster@sparkmastermachinename:7077] -> > [akka.tcp://sparkDriver@mymachinename:51834]: Error [Association failed > with [akka.tcp://sparkDriver@mymachinename:51834]] [ > > akka.remote.EndpointAssociationException: Association failed with > [akka.tcp://sparkDriver@mymachinename:51834] > > Caused by: > akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: > Connection refused: mymachinename/10.21.6.4:51834 > > ] > > 14/10/15 11:48:10 INFO Master: akka.tcp://sparkDriver@mymachinename:51834 > got disassociated, removing it. > > 14/10/15 11:48:10 ERROR EndpointWriter: AssociationError > [akka.tcp://sparkMaster@sparkmastermachinename:7077] -> > [akka.tcp://sparkDriver@mymachinename:51834]: Error [Association failed > with [akka.tcp://sparkDriver@mymachinename:51834]] [ > > akka.remote.EndpointAssociationException: Association failed with > [akka.tcp://sparkDriver@mymachinename:51834] > > Caused by: > akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: > Connection refused: mymachinename/10.21.6.4:51834 > > ] > > 14/10/15 11:48:10 INFO Master: akka.tcp://sparkDriver@mymachinename:51834 > got disassociated, removing it. > > 14/10/15 11:48:10 ERROR EndpointWriter: AssociationError > [akka.tcp://sparkMaster@sparkmastermachinename:7077] -> > [akka.tcp://sparkDriver@mymachinename:51834]: Error [Association failed > with [akka.tcp://sparkDriver@mymachinename:51834]] [ > > akka.remote.EndpointAssociationException: Association failed with > [akka.tcp://sparkDriver@mymachinename:51834] > > Caused by: > akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: > Connection refused: mymachinename/10.21.6.4:51834 > > ] > > 14/10/15 11:48:10 INFO Master: akka.tcp://sparkDriver@mymachinename:51834 > got disassociated, removing it. > > > > I've excluded the "Connection refused" messages from the possible reasons > for my problem as I have the same message when I execute my example as > standalone application... > > > > For information, here's the Spark master output when I execute the example > in standalone: > > > > 14/10/15 14:26:44 INFO Master: Registering app ts-aggregation > > 14/10/15 14:26:44 INFO Master: Registered app ts-aggregation with ID > app-20141015142644-0125 > > 14/10/15 14:26:44 INFO Master: Launching executor > app-20141015142644-0125/0 on worker > worker-20141015100053-sparkmastermachinename.lampiris.biz-39058 > > 14/10/15 14:26:44 INFO Master: Launching executor > app-20141015142644-0125/1 on worker > worker-20141015100053-sparkmastermachinename.lampiris.biz-46115 > > 14/10/15 14:26:48 INFO Master: Removing executor app-20141015142644-0125/0 > because it is EXITED > > 14/10/15 14:26:48 INFO Master: Removing executor app-20141015142644-0125/1 > because it is EXITED > > 14/10/15 14:26:49 INFO Master: akka.tcp://sparkDriver@mymachinename:58525 > got disassociated, removing it. > > 14/10/15 14:26:49 INFO Master: Removing app app-20141015142644-0125 > > 14/10/15 14:26:49 INFO LocalActorRef: Message > [akka.remote.transport.AssociationHandle$Disassociated] from > Actor[akka://sparkMaster/deadLetters] to > Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.21.6.4%3A58543-653#1905465071] > was not delivered. [216] dead letters encountered. This logging can be > turned off or adjusted with configuration settings 'akka.log-dead-letters' > and 'akka.log-dead-letters-during-shutdown'. > > 14/10/15 14:26:49 INFO LocalActorRef: Message > [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from > Actor[akka://sparkMaster/deadLetters] to > Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.21.6.4%3A58543-653#1905465071] > was not delivered. [217] dead letters encountered. This logging can be > turned off or adjusted with configuration settings 'akka.log-dead-letters' > and 'akka.log-dead-letters-during-shutdown'. > > 14/10/15 14:26:49 INFO Master: akka.tcp://sparkDriver@mymachinename:58525 > got disassociated, removing it. > > 14/10/15 14:26:49 ERROR EndpointWriter: AssociationError > [akka.tcp://sparkMaster@sparkmastermachinename:7077] -> > [akka.tcp://sparkDriver@mymachinename:58525]: Error [Association failed > with [akka.tcp://sparkDriver@mymachinename:58525]] [ > > akka.remote.EndpointAssociationException: Association failed with > [akka.tcp://sparkDriver@mymachinename:58525] > > Caused by: > akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: > Connection refused: mymachinename/10.21.6.4:58525 > > ] > > 14/10/15 14:26:49 INFO Master: akka.tcp://sparkDriver@mymachinename:58525 > got disassociated, removing it. > > 14/10/15 14:26:49 ERROR EndpointWriter: AssociationError > [akka.tcp://sparkMaster@sparkmastermachinename:7077] -> > [akka.tcp://sparkDriver@mymachinename:58525]: Error [Association failed > with [akka.tcp://sparkDriver@mymachinename:58525]] [ > > akka.remote.EndpointAssociationException: Association failed with > [akka.tcp://sparkDriver@mymachinename:58525] > > Caused by: > akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: > Connection refused: mymachinename/10.21.6.4:58525 > > ] > > 14/10/15 14:26:49 INFO Master: akka.tcp://sparkDriver@mymachinename:58525 > got disassociated, removing it. > > 14/10/15 14:26:49 ERROR EndpointWriter: AssociationError > [akka.tcp://sparkMaster@sparkmastermachinename:7077] -> > [akka.tcp://sparkDriver@mymachinename:58525]: Error [Association failed > with [akka.tcp://sparkDriver@mymachinename:58525]] [ > > akka.remote.EndpointAssociationException: Association failed with > [akka.tcp://sparkDriver@mymachinename:58525] > > Caused by: > akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: > Connection refused: mymachinename/10.21.6.4:58525 > > ] > > 14/10/15 14:26:49 INFO Master: akka.tcp://sparkDriver@mymachinename:58525 > got disassociated, removing it. > > > > Does anyone have any clue on what may cause this error? > > > > Thank you for your help ! > > > > Regards > > >
