Can someone help me out ! That what actually i am doing wrong ! The Spark UI shows that multiple apps are getting submitted , but I am submitting only single application on Spark and All the applications are in WAITING State except the main one !
On Mon, Nov 7, 2016 at 12:45 PM, Shivansh Srivastava <shiva...@knoldus.com> wrote: > > > This is the stackTrace that I am getting while running the application: > > 16/11/03 11:25:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: > Launching task 233 on executor id: 4 hostname: 10.178.149.243. > 16/11/03 11:25:45 WARN TaskSetManager: Lost task 1.0 in stage 11.0 > (TID 217, 10.178.149.243): java.util.NoSuchElementException: None.get > at scala.None$.get(Option.scala:347) > at scala.None$.get(Option.scala:345) > at org.apache.spark.storage.BlockInfoManager. > releaseAllLocksForTask(BlockInfoManager.scala:343) > at org.apache.spark.storage.BlockManager.releaseAllLocksForTask( > BlockManager.scala:644) > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:281) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > 16/11/03 11:25:45 INFO TaskSetManager: Lost task 14.0 in stage 11.0 > (TID 225) on executor 10.178.149.243: java.util.NoSuchElementException > (None.get) [duplicate 1] > 16/11/03 11:25:45 INFO TaskSetManager: Starting task 14.1 in stage > 11.0 (TID 234, 10.178.149.243, partition 14, NODE_LOCAL, 8828 bytes) > 16/11/03 11:25:45 INFO TaskSetManager: Lost task 22.0 in stage 11.0 > (TID 232) on executor 10.178.149.243: java.util.NoSuchElementException > (None.get) [duplicate 2] > 16/11/03 11:25:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: > Launching task 234 on executor id: 4 hostname: 10.178.149.243. > 16/11/03 11:25:45 INFO TaskSetManager: Starting task 22.1 in stage > 11.0 (TID 235, 10.178.149.243, partition 22, NODE_LOCAL, 9066 bytes) > 16/11/03 11:25:45 INFO TaskSetManager: Lost task 24.0 in stage 11.0 > (TID 233) on executor 10.178.149.243: java.util.NoSuchElementException > (None.get) [duplicate 3] > 16/11/03 11:25:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: > Launching task 235 on executor id: 4 hostname: 10.178.149.243. > 16/11/03 11:25:45 INFO TaskSetManager: Starting task 24.1 in stage > 11.0 (TID 236, 10.178.149.243, partition 24, NODE_LOCAL, 9185 bytes) > 16/11/03 11:25:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: > Launching task 236 on executor id: 4 hostname: 10.178.149.243. > 16/11/03 11:25:45 INFO TaskSetManager: Lost task 22.1 in stage 11.0 > (TID 235) on executor 10.178.149.243: java.util.NoSuchElementException > (None.get) [duplicate 4] > 16/11/03 11:25:45 INFO TaskSetManager: Starting task 22.2 in stage > 11.0 (TID 237, 10.178.149.243, partition 22, NODE_LOCAL, 9066 bytes) > 16/11/03 11:25:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: > Launching task 237 on executor id: 4 hostname: 10.178.149.243. > 16/11/03 11:25:45 INFO TaskSetManager: Lost task 14.1 in stage 11.0 > (TID 234) on executor 10.178.149.243: java.util.NoSuchElementException > (None.get) [duplicate 5] > 16/11/03 11:25:45 INFO TaskSetManager: Starting task 14.2 in stage > 11.0 (TID 238, 10.178.149.243, partition 14, NODE_LOCAL, 8828 bytes) > 16/11/03 11:25:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: > Launching task 238 on executor id: 4 hostname: 10.178.149.243. > 16/11/03 11:25:45 INFO TaskSetManager: Lost task 24.1 in stage 11.0 > (TID 236) on executor 10.178.149.243: java.util.NoSuchElementException > (None.get) [duplicate 6] > 16/11/03 11:25:45 INFO TaskSetManager: Starting task 24.2 in stage > 11.0 (TID 239, 10.178.149.243, partition 24, NODE_LOCAL, 9185 bytes) > 16/11/03 11:25:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: > Launching task 239 on executor id: 4 hostname: 10.178.149.243. > 16/11/03 11:25:45 INFO TaskSetManager: Lost task 22.2 in stage 11.0 > (TID 237) on executor 10.178.149.243: java.util.NoSuchElementException > (None.get) [duplicate 7] > 16/11/03 11:25:45 INFO TaskSetManager: Starting task 22.3 in stage > 11.0 (TID 240, 10.178.149.243, partition 22, NODE_LOCAL, 9066 bytes) > 16/11/03 11:25:45 INFO TaskSetManager: Lost task 14.2 in stage 11.0 > (TID 238) on executor 10.178.149.243: java.util.NoSuchElementException > (None.get) [duplicate 8] > 16/11/03 11:25:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: > Launching task 240 on executor id: 4 hostname: 10.178.149.243. > 16/11/03 11:25:45 INFO TaskSetManager: Starting task 14.3 in stage > 11.0 (TID 241, 10.178.149.243, partition 14, NODE_LOCAL, 8828 bytes) > 16/11/03 11:25:45 INFO TaskSetManager: Lost task 24.2 in stage 11.0 > (TID 239) on executor 10.178.149.243: java.util.NoSuchElementException > (None.get) [duplicate 9] > 16/11/03 11:25:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: > Launching task 241 on executor id: 4 hostname: 10.178.149.243. > 16/11/03 11:25:45 INFO TaskSetManager: Starting task 24.3 in stage > 11.0 (TID 242, 10.178.149.243, partition 24, NODE_LOCAL, 9185 bytes) > 16/11/03 11:25:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: > Launching task 242 on executor id: 4 hostname: 10.178.149.243. > 16/11/03 11:25:45 INFO TaskSetManager: Lost task 22.3 in stage 11.0 > (TID 240) on executor 10.178.149.243: java.util.NoSuchElementException > (None.get) [duplicate 10] > 16/11/03 11:25:45 ERROR TaskSetManager: Task 22 in stage 11.0 failed 4 > times; aborting job > 16/11/03 11:25:45 INFO TaskSetManager: Starting task 0.0 in stage 12.0 > (TID 243, 10.178.149.243, partition 0, NODE_LOCAL, 10016 bytes) > 16/11/03 11:25:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: > Launching task 243 on executor id: 4 hostname: 10.178.149.243. > 16/11/03 11:25:45 INFO TaskSetManager: Lost task 14.3 in stage 11.0 > (TID 241) on executor 10.178.149.243: java.util.NoSuchElementException > (None.get) [duplicate 11] > 16/11/03 11:25:45 INFO TaskSchedulerImpl: Cancelling stage 12 > 16/11/03 11:25:45 INFO TaskSchedulerImpl: Stage 12 was cancelled > 16/11/03 11:25:45 INFO TaskSetManager: Starting task 0.0 in stage 14.0 > (TID 244, 10.178.149.243, partition 0, NODE_LOCAL, 7638 bytes) > 16/11/03 11:25:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: > Launching task 244 on executor id: 4 hostname: 10.178.149.243. > 16/11/03 11:25:45 INFO TaskSetManager: Lost task 24.3 in stage 11.0 > (TID 242) on executor 10.178.149.243: java.util.NoSuchElementException > (None.get) [duplicate 12] > 16/11/03 11:25:45 INFO DAGScheduler: ShuffleMapStage 12 (show at > RNFBackTagger.scala:97) failed in 0.112 s > 16/11/03 11:25:45 INFO TaskSchedulerImpl: Cancelling stage 14 > 16/11/03 11:25:45 INFO TaskSchedulerImpl: Stage 14 was cancelled > 16/11/03 11:25:45 INFO DAGScheduler: ShuffleMapStage 14 (show at > RNFBackTagger.scala:97) failed in 0.104 s > 16/11/03 11:25:45 INFO TaskSchedulerImpl: Cancelling stage 11 > 16/11/03 11:25:45 INFO TaskSchedulerImpl: Stage 11 was cancelled > 16/11/03 11:25:45 INFO DAGScheduler: ShuffleMapStage 11 (show at > RNFBackTagger.scala:97) failed in 0.126 s > 16/11/03 11:25:45 WARN TaskSetManager: Lost task 0.0 in stage 12.0 > (TID 243, 10.178.149.243): java.util.NoSuchElementException: None.get > at scala.None$.get(Option.scala:347) > at scala.None$.get(Option.scala:345) > at org.apache.spark.storage.BlockInfoManager. > releaseAllLocksForTask(BlockInfoManager.scala:343) > at org.apache.spark.storage.BlockManager.releaseAllLocksForTask( > BlockManager.scala:644) > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:281) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > 16/11/03 11:25:45 INFO DAGScheduler: Job 7 failed: show at > RNFBackTagger.scala:97, took 0.141681 s > 16/11/03 11:25:45 INFO TaskSchedulerImpl: Removed TaskSet 12.0, whose > tasks have all completed, from pool > Exception in thread "main" org.apache.spark.SparkException: Job > aborted due to stage failure: Task 22 in stage 11.0 failed 4 times, most > recent failure: Lost task 22.3 in stage 11.0 (TID 240, 10.178.149.243): > java.util.NoSuchElementException: None.get > at scala.None$.get(Option.scala:347) > at scala.None$.get(Option.scala:345) > at org.apache.spark.storage.BlockInfoManager. > releaseAllLocksForTask(BlockInfoManager.scala:343) > at org.apache.spark.storage.BlockManager.releaseAllLocksForTask( > BlockManager.scala:644) > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:281) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > Driver stacktrace: > at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$ > scheduler$DAGScheduler$$failJobAndIndependentStages( > DAGScheduler.scala:1450) > at org.apache.spark.scheduler.DAGScheduler$$anonfun$ > abortStage$1.apply(DAGScheduler.scala:1438) > at org.apache.spark.scheduler.DAGScheduler$$anonfun$ > abortStage$1.apply(DAGScheduler.scala:1437) > at scala.collection.mutable.ResizableArray$class.foreach( > ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach( > ArrayBuffer.scala:48) > at org.apache.spark.scheduler.DAGScheduler.abortStage( > DAGScheduler.scala:1437) > at org.apache.spark.scheduler.DAGScheduler$$anonfun$ > handleTaskSetFailed$1.apply(DAGScheduler.scala:811) > at org.apache.spark.scheduler.DAGScheduler$$anonfun$ > handleTaskSetFailed$1.apply(DAGScheduler.scala:811) > at scala.Option.foreach(Option.scala:257) > at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed( > DAGScheduler.scala:811) > at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop. > doOnReceive(DAGScheduler.scala:1659) > at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop. > onReceive(DAGScheduler.scala:1618) > at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop. > onReceive(DAGScheduler.scala:1607) > at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) > at org.apache.spark.scheduler.DAGScheduler.runJob( > DAGScheduler.scala:632) > at org.apache.spark.SparkContext.runJob(SparkContext.scala:1871) > at org.apache.spark.SparkContext.runJob(SparkContext.scala:1884) > at org.apache.spark.SparkContext.runJob(SparkContext.scala:1897) > at org.apache.spark.sql.execution.SparkPlan. > executeTake(SparkPlan.scala:347) > at org.apache.spark.sql.execution.CollectLimitExec. > executeCollect(limit.scala:39) > at org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$ > Dataset$$execute$1$1.apply(Dataset.scala:2183) > at org.apache.spark.sql.execution.SQLExecution$. > withNewExecutionId(SQLExecution.scala:57) > at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset. > scala:2532) > at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$ > execute$1(Dataset.scala:2182) > at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$ > collect(Dataset.scala:2189) > at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset. > scala:1925) > at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset. > scala:1924) > at org.apache.spark.sql.Dataset.withTypedCallback(Dataset. > scala:2562) > at org.apache.spark.sql.Dataset.head(Dataset.scala:1924) > at org.apache.spark.sql.Dataset.take(Dataset.scala:2139) > at org.apache.spark.sql.Dataset.showString(Dataset.scala:239) > at org.apache.spark.sql.Dataset.show(Dataset.scala:526) > at com.knoldus.xml.RNFBackTagger$.main(RNFBackTagger.scala:97) > at com.knoldus.xml.RNFBackTagger.main(RNFBackTagger.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$ > deploy$SparkSubmit$$runMain(SparkSubmit.scala:729) > at org.apache.spark.deploy.SparkSubmit$.doRunMain$1( > SparkSubmit.scala:185) > at org.apache.spark.deploy.SparkSubmit$.submit( > SparkSubmit.scala:210) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit. > scala:124) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.util.NoSuchElementException: None.get > at scala.None$.get(Option.scala:347) > at scala.None$.get(Option.scala:345) > at org.apache.spark.storage.BlockInfoManager. > releaseAllLocksForTask(BlockInfoManager.scala:343) > at org.apache.spark.storage.BlockManager.releaseAllLocksForTask( > BlockManager.scala:644) > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:281) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > 16/11/03 11:25:45 WARN JobProgressListener: Task start for unknown > stage 12 > 16/11/03 11:25:45 WARN TaskSetManager: Lost task 0.0 in stage 14.0 > (TID 244, 10.178.149.243): java.util.NoSuchElementException: None.get > at scala.None$.get(Option.scala:347) > at scala.None$.get(Option.scala:345) > at org.apache.spark.storage.BlockInfoManager. > releaseAllLocksForTask(BlockInfoManager.scala:343) > at org.apache.spark.storage.BlockManager.releaseAllLocksForTask( > BlockManager.scala:644) > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:281) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > 16/11/03 11:25:45 INFO TaskSchedulerImpl: Removed TaskSet 14.0, whose > tasks have all completed, from pool > 16/11/03 11:25:45 INFO SparkContext: Invoking stop() from shutdown hook > 16/11/03 11:25:45 WARN JobProgressListener: Task start for unknown > stage 14 > 16/11/03 11:25:45 INFO SerialShutdownHooks: Successfully executed > shutdown hook: Clearing session cache for C* connector > 16/11/03 11:25:45 INFO TaskSetManager: Finished task 5.0 in stage 11.0 > (TID 219) in 137 ms on 10.178.149.22 (1/35) > 16/11/03 11:25:45 INFO SparkUI: Stopped Spark web UI at > http://10.178.149.133:4040 > 16/11/03 11:25:45 INFO StandaloneSchedulerBackend: Shutting down all > executors > 16/11/03 11:25:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: > Asking each executor to shut down > 16/11/03 11:25:45 ERROR TransportRequestHandler: Error while invoking > RpcHandler#receive() for one-way message. > org.apache.spark.SparkException: Could not find > CoarseGrainedScheduler. > at org.apache.spark.rpc.netty.Dispatcher.postMessage( > Dispatcher.scala:152) > at org.apache.spark.rpc.netty.Dispatcher.postOneWayMessage( > Dispatcher.scala:132) > at org.apache.spark.rpc.netty.NettyRpcHandler.receive( > NettyRpcEnv.scala:571) > at org.apache.spark.network.server.TransportRequestHandler. > processOneWayMessage(TransportRequestHandler.java:179) > at org.apache.spark.network.server.TransportRequestHandler.handle( > TransportRequestHandler.java:108) > at org.apache.spark.network.server.TransportChannelHandler. > channelRead0(TransportChannelHandler.java:119) > at org.apache.spark.network.server.TransportChannelHandler. > channelRead0(TransportChannelHandler.java:51) > at io.netty.channel.SimpleChannelInboundHandler.channelRead( > SimpleChannelInboundHandler.java:105) > at io.netty.channel.AbstractChannelHandlerContext. > invokeChannelRead(AbstractChannelHandlerContext.java:308) > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( > AbstractChannelHandlerContext.java:294) > at io.netty.handler.timeout.IdleStateHandler.channelRead( > IdleStateHandler.java:266) > at io.netty.channel.AbstractChannelHandlerContext. > invokeChannelRead(AbstractChannelHandlerContext.java:308) > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( > AbstractChannelHandlerContext.java:294) > at io.netty.handler.codec.MessageToMessageDecoder.channelRead( > MessageToMessageDecoder.java:103) > at io.netty.channel.AbstractChannelHandlerContext. > invokeChannelRead(AbstractChannelHandlerContext.java:308) > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( > AbstractChannelHandlerContext.java:294) > at org.apache.spark.network.util.TransportFrameDecoder. > channelRead(TransportFrameDecoder.java:85) > at io.netty.channel.AbstractChannelHandlerContext. > invokeChannelRead(AbstractChannelHandlerContext.java:308) > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( > AbstractChannelHandlerContext.java:294) > at io.netty.channel.DefaultChannelPipeline.fireChannelRead( > DefaultChannelPipeline.java:846) > at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read( > AbstractNioByteChannel.java:131) > at io.netty.channel.nio.NioEventLoop.processSelectedKey( > NioEventLoop.java:511) > at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized( > NioEventLoop.java:468) > at io.netty.channel.nio.NioEventLoop.processSelectedKeys( > NioEventLoop.java:382) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > at io.netty.util.concurrent.SingleThreadEventExecutor$2. > run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > 16/11/03 11:25:45 ERROR TransportRequestHandler: Error while invoking > RpcHandler#receive() for one-way message. > org.apache.spark.SparkException: Could not find > CoarseGrainedScheduler. > at org.apache.spark.rpc.netty.Dispatcher.postMessage( > Dispatcher.scala:152) > at org.apache.spark.rpc.netty.Dispatcher.postOneWayMessage( > Dispatcher.scala:132) > at org.apache.spark.rpc.netty.NettyRpcHandler.receive( > NettyRpcEnv.scala:571) > at org.apache.spark.network.server.TransportRequestHandler. > processOneWayMessage(TransportRequestHandler.java:179) > at org.apache.spark.network.server.TransportRequestHandler.handle( > TransportRequestHandler.java:108) > at org.apache.spark.network.server.TransportChannelHandler. > channelRead0(TransportChannelHandler.java:119) > at org.apache.spark.network.server.TransportChannelHandler. > channelRead0(TransportChannelHandler.java:51) > at io.netty.channel.SimpleChannelInboundHandler.channelRead( > SimpleChannelInboundHandler.java:105) > at io.netty.channel.AbstractChannelHandlerContext. > invokeChannelRead(AbstractChannelHandlerContext.java:308) > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( > AbstractChannelHandlerContext.java:294) > at io.netty.handler.timeout.IdleStateHandler.channelRead( > IdleStateHandler.java:266) > at io.netty.channel.AbstractChannelHandlerContext. > invokeChannelRead(AbstractChannelHandlerContext.java:308) > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( > AbstractChannelHandlerContext.java:294) > at io.netty.handler.codec.MessageToMessageDecoder.channelRead( > MessageToMessageDecoder.java:103) > at io.netty.channel.AbstractChannelHandlerContext. > invokeChannelRead(AbstractChannelHandlerContext.java:308) > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( > AbstractChannelHandlerContext.java:294) > at org.apache.spark.network.util.TransportFrameDecoder. > channelRead(TransportFrameDecoder.java:85) > at io.netty.channel.AbstractChannelHandlerContext. > invokeChannelRead(AbstractChannelHandlerContext.java:308) > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( > AbstractChannelHandlerContext.java:294) > at io.netty.channel.DefaultChannelPipeline.fireChannelRead( > DefaultChannelPipeline.java:846) > at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read( > AbstractNioByteChannel.java:131) > at io.netty.channel.nio.NioEventLoop.processSelectedKey( > NioEventLoop.java:511) > at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized( > NioEventLoop.java:468) > at io.netty.channel.nio.NioEventLoop.processSelectedKeys( > NioEventLoop.java:382) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > at io.netty.util.concurrent.SingleThreadEventExecutor$2. > run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > 16/11/03 11:25:45 INFO MapOutputTrackerMasterEndpoint: > MapOutputTrackerMasterEndpoint stopped! > 16/11/03 11:25:45 INFO MemoryStore: MemoryStore cleared > 16/11/03 11:25:45 INFO BlockManager: BlockManager stopped > 16/11/03 11:25:45 INFO BlockManagerMaster: BlockManagerMaster stopped > 16/11/03 11:25:45 INFO OutputCommitCoordinator$ > OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! > 16/11/03 11:25:45 ERROR TransportRequestHandler: Error while invoking > RpcHandler#receive() for one-way message. > org.apache.spark.rpc.RpcEnvStoppedException: RpcEnv already stopped. > at org.apache.spark.rpc.netty.Dispatcher.postMessage( > Dispatcher.scala:150) > at org.apache.spark.rpc.netty.Dispatcher.postOneWayMessage( > Dispatcher.scala:132) > at org.apache.spark.rpc.netty.NettyRpcHandler.receive( > NettyRpcEnv.scala:571) > at org.apache.spark.network.server.TransportRequestHandler. > processOneWayMessage(TransportRequestHandler.java:179) > at org.apache.spark.network.server.TransportRequestHandler.handle( > TransportRequestHandler.java:108) > at org.apache.spark.network.server.TransportChannelHandler. > channelRead0(TransportChannelHandler.java:119) > at org.apache.spark.network.server.TransportChannelHandler. > channelRead0(TransportChannelHandler.java:51) > at io.netty.channel.SimpleChannelInboundHandler.channelRead( > SimpleChannelInboundHandler.java:105) > at io.netty.channel.AbstractChannelHandlerContext. > invokeChannelRead(AbstractChannelHandlerContext.java:308) > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( > AbstractChannelHandlerContext.java:294) > at io.netty.handler.timeout.IdleStateHandler.channelRead( > IdleStateHandler.java:266) > at io.netty.channel.AbstractChannelHandlerContext. > invokeChannelRead(AbstractChannelHandlerContext.java:308) > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( > AbstractChannelHandlerContext.java:294) > at io.netty.handler.codec.MessageToMessageDecoder.channelRead( > MessageToMessageDecoder.java:103) > at io.netty.channel.AbstractChannelHandlerContext. > invokeChannelRead(AbstractChannelHandlerContext.java:308) > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( > AbstractChannelHandlerContext.java:294) > at org.apache.spark.network.util.TransportFrameDecoder. > channelRead(TransportFrameDecoder.java:85) > at io.netty.channel.AbstractChannelHandlerContext. > invokeChannelRead(AbstractChannelHandlerContext.java:308) > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( > AbstractChannelHandlerContext.java:294) > at io.netty.channel.DefaultChannelPipeline.fireChannelRead( > DefaultChannelPipeline.java:846) > at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read( > AbstractNioByteChannel.java:131) > at io.netty.channel.nio.NioEventLoop.processSelectedKey( > NioEventLoop.java:511) > at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized( > NioEventLoop.java:468) > at io.netty.channel.nio.NioEventLoop.processSelectedKeys( > NioEventLoop.java:382) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > at io.netty.util.concurrent.SingleThreadEventExecutor$2. > run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > 16/11/03 11:25:45 ERROR TransportRequestHandler: Error while invoking > RpcHandler#receive() for one-way message. > org.apache.spark.rpc.RpcEnvStoppedException: RpcEnv already stopped. > at org.apache.spark.rpc.netty.Dispatcher.postMessage( > Dispatcher.scala:150) > at org.apache.spark.rpc.netty.Dispatcher.postOneWayMessage( > Dispatcher.scala:132) > at org.apache.spark.rpc.netty.NettyRpcHandler.receive( > NettyRpcEnv.scala:571) > at org.apache.spark.network.server.TransportRequestHandler. > processOneWayMessage(TransportRequestHandler.java:179) > at org.apache.spark.network.server.TransportRequestHandler.handle( > TransportRequestHandler.java:108) > at org.apache.spark.network.server.TransportChannelHandler. > channelRead0(TransportChannelHandler.java:119) > at org.apache.spark.network.server.TransportChannelHandler. > channelRead0(TransportChannelHandler.java:51) > at io.netty.channel.SimpleChannelInboundHandler.channelRead( > SimpleChannelInboundHandler.java:105) > at io.netty.channel.AbstractChannelHandlerContext. > invokeChannelRead(AbstractChannelHandlerContext.java:308) > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( > AbstractChannelHandlerContext.java:294) > at io.netty.handler.timeout.IdleStateHandler.channelRead( > IdleStateHandler.java:266) > at io.netty.channel.AbstractChannelHandlerContext. > invokeChannelRead(AbstractChannelHandlerContext.java:308) > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( > AbstractChannelHandlerContext.java:294) > at io.netty.handler.codec.MessageToMessageDecoder.channelRead( > MessageToMessageDecoder.java:103) > at io.netty.channel.AbstractChannelHandlerContext. > invokeChannelRead(AbstractChannelHandlerContext.java:308) > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( > AbstractChannelHandlerContext.java:294) > at org.apache.spark.network.util.TransportFrameDecoder. > channelRead(TransportFrameDecoder.java:85) > at io.netty.channel.AbstractChannelHandlerContext. > invokeChannelRead(AbstractChannelHandlerContext.java:308) > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead( > AbstractChannelHandlerContext.java:294) > at io.netty.channel.DefaultChannelPipeline.fireChannelRead( > DefaultChannelPipeline.java:846) > at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read( > AbstractNioByteChannel.java:131) > at io.netty.channel.nio.NioEventLoop.processSelectedKey( > NioEventLoop.java:511) > at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized( > NioEventLoop.java:468) > at io.netty.channel.nio.NioEventLoop.processSelectedKeys( > NioEventLoop.java:382) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) > at io.netty.util.concurrent.SingleThreadEventExecutor$2. > run(SingleThreadEventExecutor.java:111) > at java.lang.Thread.run(Thread.java:745) > 16/11/03 11:25:45 INFO SparkContext: Successfully stopped SparkContext > 16/11/03 11:25:45 INFO ShutdownHookManager: Shutdown hook called > 16/11/03 11:25:45 INFO ShutdownHookManager: Deleting directory > /tmp/spark-c52a6da9-5702-4128-9950-805d5f9dd75e > > Sometimes the error trace changes and it gives only this: > 16/11/07 07:04:35 WARN TaskSetManager: Lost task 0.0 in stage 7.0 (TID > 199, 10.178.149.243): java.util.NoSuchElementException: None.get > at scala.None$.get(Option.scala:347) > at scala.None$.get(Option.scala:345) > at org.apache.spark.storage.BlockInfoManager.releaseAllLocksForTask( > BlockInfoManager.scala:343) > at org.apache.spark.storage.BlockManager.releaseAllLocksForTask( > BlockManager.scala:644) > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:281) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > 16/11/07 07:04:35 INFO TaskSetManager: Starting task 0.1 in stage 7.0 (TID > 200, 10.178.149.243, partition 0, NODE_LOCAL, 5286 bytes) > 16/11/07 07:04:35 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: > Launching task 200 on executor id: 5 hostname: 10.178.149.243. > 16/11/07 07:04:35 INFO TaskSetManager: Lost task 0.1 in stage 7.0 (TID > 200) on executor 10.178.149.243: java.util.NoSuchElementException > (None.get) [duplicate 1] > 16/11/07 07:04:35 INFO TaskSetManager: Starting task 0.2 in stage 7.0 (TID > 201, 10.178.149.243, partition 0, NODE_LOCAL, 5286 bytes) > 16/11/07 07:04:35 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: > Launching task 201 on executor id: 7 hostname: 10.178.149.243. > 16/11/07 07:04:35 INFO TaskSetManager: Lost task 0.2 in stage 7.0 (TID > 201) on executor 10.178.149.243: java.util.NoSuchElementException > (None.get) [duplicate 2] > 16/11/07 07:04:35 INFO TaskSetManager: Starting task 0.3 in stage 7.0 (TID > 202, 10.178.149.243, partition 0, NODE_LOCAL, 5286 bytes) > 16/11/07 07:04:35 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: > Launching task 202 on executor id: 5 hostname: 10.178.149.243. > 16/11/07 07:04:35 INFO TaskSetManager: Lost task 0.3 in stage 7.0 (TID > 202) on executor 10.178.149.243: java.util.NoSuchElementException > (None.get) [duplicate 3] > 16/11/07 07:04:35 ERROR TaskSetManager: Task 0 in stage 7.0 failed 4 > times; aborting job > 16/11/07 07:04:35 INFO TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks > have all completed, from pool > 16/11/07 07:04:35 INFO TaskSchedulerImpl: Cancelling stage 7 > 16/11/07 07:04:35 INFO DAGScheduler: ResultStage 7 (show at > RNFBackTagger.scala:90) failed in 0.105 s > 16/11/07 07:04:35 INFO DAGScheduler: Job 2 failed: show at > RNFBackTagger.scala:90, took 40.037558 s > Exception in thread "main" org.apache.spark.SparkException: Job aborted > due to stage failure: Task 0 in stage 7.0 failed 4 times, most recent > failure: Lost task 0.3 in stage 7.0 (TID 202, 10.178.149.243): > java.util.NoSuchElementException: > None.get > at scala.None$.get(Option.scala:347) > at scala.None$.get(Option.scala:345) > at org.apache.spark.storage.BlockInfoManager.releaseAllLocksForTask( > BlockInfoManager.scala:343) > at org.apache.spark.storage.BlockManager.releaseAllLocksForTask( > BlockManager.scala:644) > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:281) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > Driver stacktrace: > at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$ > scheduler$DAGScheduler$$failJobAndIndependentStages( > DAGScheduler.scala:1450) > at org.apache.spark.scheduler.DAGScheduler$$anonfun$ > abortStage$1.apply(DAGScheduler.scala:1438) > at org.apache.spark.scheduler.DAGScheduler$$anonfun$ > abortStage$1.apply(DAGScheduler.scala:1437) > at scala.collection.mutable.ResizableArray$class.foreach( > ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) > at org.apache.spark.scheduler.DAGScheduler.abortStage( > DAGScheduler.scala:1437) > at org.apache.spark.scheduler.DAGScheduler$$anonfun$ > handleTaskSetFailed$1.apply(DAGScheduler.scala:811) > at org.apache.spark.scheduler.DAGScheduler$$anonfun$ > handleTaskSetFailed$1.apply(DAGScheduler.scala:811) > at scala.Option.foreach(Option.scala:257) > at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed( > DAGScheduler.scala:811) > at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop. > doOnReceive(DAGScheduler.scala:1659) > at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop. > onReceive(DAGScheduler.scala:1618) > at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop. > onReceive(DAGScheduler.scala:1607) > at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) > at org.apache.spark.scheduler.DAGScheduler.runJob( > DAGScheduler.scala:632) > at org.apache.spark.SparkContext.runJob(SparkContext.scala:1871) > at org.apache.spark.SparkContext.runJob(SparkContext.scala:1884) > at org.apache.spark.SparkContext.runJob(SparkContext.scala:1897) > at org.apache.spark.sql.execution.SparkPlan. > executeTake(SparkPlan.scala:347) > at org.apache.spark.sql.execution.CollectLimitExec. > executeCollect(limit.scala:39) > at org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$ > Dataset$$execute$1$1.apply(Dataset.scala:2183) > at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId( > SQLExecution.scala:57) > at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2532) > at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$ > execute$1(Dataset.scala:2182) > at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$ > collect(Dataset.scala:2189) > at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset. > scala:1925) > at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset. > scala:1924) > at org.apache.spark.sql.Dataset.withTypedCallback(Dataset.scala:2562) > at org.apache.spark.sql.Dataset.head(Dataset.scala:1924) > at org.apache.spark.sql.Dataset.take(Dataset.scala:2139) > at org.apache.spark.sql.Dataset.showString(Dataset.scala:239) > at org.apache.spark.sql.Dataset.show(Dataset.scala:526) > at org.apache.spark.sql.Dataset.show(Dataset.scala:486) > at org.apache.spark.sql.Dataset.show(Dataset.scala:495) > at com.knoldus.xml.RNFBackTagger$.main(RNFBackTagger.scala:90) > at com.knoldus.xml.RNFBackTagger.main(RNFBackTagger.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$ > deploy$SparkSubmit$$runMain(SparkSubmit.scala:729) > at org.apache.spark.deploy.SparkSubmit$.doRunMain$1( > SparkSubmit.scala:185) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.util.NoSuchElementException: None.get > at scala.None$.get(Option.scala:347) > at scala.None$.get(Option.scala:345) > at org.apache.spark.storage.BlockInfoManager.releaseAllLocksForTask( > BlockInfoManager.scala:343) > at org.apache.spark.storage.BlockManager.releaseAllLocksForTask( > BlockManager.scala:644) > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:281) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > > > Earlier I was not able to pin point the problem ! > Then I tried the removing unncessary Code approach ! > > Then I found out the problem lies in this : > > val groupedDF = selectedDF.groupBy("id").agg(collect_list("value")) > groupedDF.show > > > Because if I try to show selectedDF it displays the correct result! > > > The spark version that I am using is 2.0.0 ! Please help me out and let me > know what is the problem. > > Link to Code is : > https://gist.github.com/shiv4nsh/0c3f62e3afd95634a6061b405c774582 > > Show on line 19 prints and the show on 28 throws this exception. > > > Server Configuration: I have spark 2.0 running on 8 core worker with 10 gb > memory and its running on centOS > > Script for launching application: > > > ./bin/spark-submit --class com.knoldus.Application > /root/code/newCode/project1/target/deployable.jar > > Any help is appreciated ! > > **Note:** The code works fine in local mode. This error is thrown when i > try to run it on cluster. > > > This question I have already asked on Stackoverflow here: > http://stackoverflow.com/questions/40400424/spark-exits-with-exception? > noredirect=1#comment68095238_40400424 > > -- > *Best Regards | Shivansh* > *Software Consultant* > *Knoldus Software LLP* > > *India - US - Canada* > * Twitter <http://www.twitter.com/shiv4nsh> | FB > <http://www.facebook.com/xeruo> | LinkedIn > <https://in.linkedin.com/pub/shivansh-srivastava/3a/204/251>* > -- *Best Regards | Shivansh* *Software Consultant* *Knoldus Software LLP* *India - US - Canada* * Twitter <http://www.twitter.com/shiv4nsh> | FB <http://www.facebook.com/xeruo> | LinkedIn <https://in.linkedin.com/pub/shivansh-srivastava/3a/204/251>*
