Hi All,When I run my application, it runs for a while and give me part of the
o/p correctly. I then get the following error and it then spark shell exits.
14/07/07 13:54:53 INFO SendingConnection: Initiating connection to
[localhost.localdomain/127.0.0.1:57423]14/07/07 13:54:53 INFO
ConnectionManager: Accepted connection from
[localhost.localdomain/127.0.0.1]14/07/07 13:54:53 INFO SendingConnection:
Connected to [localhost.localdomain/127.0.0.1:57423], 2 messages
pending14/07/07 13:54:53 INFO BlockManager: Removing block
taskresult_1414/07/07 13:54:53 INFO BlockManager: Removing block
taskresult_1314/07/07 13:54:53 INFO MemoryStore: Block taskresult_14 of size
12174859 dropped from memory (free 296532358)14/07/07 13:54:53 INFO
MemoryStore: Block taskresult_13 of size 12115603 dropped from memory (free
308647961)14/07/07 13:54:53 INFO BlockManagerInfo: Removed taskresult_14 on
pzxnvm2018.dcld.pldc.kp.org:50924 in memory (size: 11.6 MB, free: 282.9
MB)14/07/07 13:54:53 INFO BlockManagerMaster: Updated info of block
taskresult_1414/07/07 13:54:53 INFO BlockManagerInfo: Removed taskresult_13 on
pzxnvm2018.a.b.org:50924 in memory (size: 11.6 MB, free: 294.4 MB)14/07/07
13:54:53 INFO BlockManagerMaster: Updated info of block taskresult_1314/07/07
13:54:54 INFO TaskSetManager: Finished TID 13 in 3043 ms on localhost
(progress: 1/2)14/07/07 13:54:54 INFO DAGScheduler: Completed ResultTask(7,
0)14/07/07 13:54:54 INFO TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks
have all completed, from pool14/07/07 13:54:54 INFO DAGScheduler: Failed to run
collect at JaccardScore.scala:8414/07/07 13:54:54 INFO TaskSchedulerImpl:
Cancelling stage 7org.apache.spark.SparkException: Job aborted due to stage
failure: Exception while deserializing and fetching task:
com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID:
13994 at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1033)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1017)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1015)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1015) at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:633)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:633)
at scala.Option.foreach(Option.scala:236) at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:633)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1207)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) at
akka.actor.ActorCell.invoke(ActorCell.scala:456) at
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) at
akka.dispatch.Mailbox.run(Mailbox.scala:219) at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
scala> 14/07/07 13:54:57 INFO AppClient$ClientActor: Connecting to master
spark://pzxnvm2018:7077...14/07/07 13:55:17 INFO AppClient$ClientActor:
Connecting to master spark://pzxnvm2018:7077...14/07/07 13:55:37 ERROR
SparkDeploySchedulerBackend: Application has been killed. Reason: All masters
are unresponsive! Giving up.14/07/07 13:55:37 ERROR TaskSchedulerImpl: Exiting
due to error from cluster scheduler: All masters are unresponsive! Giving up.
Logs for the Master node:14/07/07 13:51:55 INFO Master:
akka.tcp://spark@localhost:45063 got disassociated, removing it.14/07/07
13:54:38 ERROR EndpointWriter: dropping message [class
akka.actor.SelectChildName] for non-local recipient
[Actor[akka.tcp://sparkMaster@pzxnvm2018:7077/]] arriving at
[akka.tcp://sparkMaster@pzxnvm2018:7077] inbound addresses are
[akka.tcp://[email protected]:7077]14/07/07 13:54:57 ERROR
EndpointWriter: dropping message [class akka.actor.SelectChildName] for
non-local recipient [Actor[akka.tcp://sparkMaster@pzxnvm2018:7077/]] arriving
at [akka.tcp://sparkMaster@pzxnvm2018:7077] inbound addresses are
[akka.tcp://[email protected]:7077]14/07/07 13:55:17 ERROR
EndpointWriter: dropping message [class akka.actor.SelectChildName] for
non-local recipient [Actor[akka.tcp://sparkMaster@pzxnvm2018:7077/]] arriving
at [akka.tcp://sparkMaster@pzxnvm2018:7077] inbound addresses are
[akka.tcp://[email protected]:7077]14/07/07 13:55:38 INFO Master:
akka.tcp://spark@localhost:38739 got disassociated, removing it.14/07/07
13:55:38 INFO Master: akka.tcp://spark@localhost:38739 got disassociated,
removing it.14/07/07 13:55:38 INFO LocalActorRef: Message
[akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from
Actor[akka://sparkMaster/deadLetters] to
Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%40172.16.48.41%3A49030-9#-767579404]
was not delivered. [2] dead letters encountered. This logging can be turned
off or adjusted with configuration settings 'akka.log-dead-letters' and
'akka.log-dead-letters-during-shutdown'.14/07/07 13:55:38 ERROR EndpointWriter:
AssociationError [akka.tcp://[email protected]:7077] ->
[akka.tcp://spark@localhost:38739]: Error [Association failed with
[akka.tcp://spark@localhost:38739]] [akka.remote.EndpointAssociationException:
Association failed with [akka.tcp://spark@localhost:38739]Caused by:
akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2:
Connection refused: localhost/127.0.0.1:38739]14/07/07 13:55:38 INFO Master:
akka.tcp://spark@localhost:38739 got disassociated, removing it.14/07/07
13:55:38 INFO Master: akka.tcp://spark@localhost:38739 got disassociated,
removing it.14/07/07 13:55:38 ERROR EndpointWriter: AssociationError
[akka.tcp://[email protected]:7077] ->
[akka.tcp://spark@localhost:38739]: Error [Association failed with
[akka.tcp://spark@localhost:38739]] [akka.remote.EndpointAssociationException:
Association failed with [akka.tcp://spark@localhost:38739]Caused by:
akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2:
Connection refused: localhost/127.0.0.1:38739]14/07/07 13:55:38 ERROR
EndpointWriter: AssociationError
[akka.tcp://[email protected]:7077] ->
[akka.tcp://spark@localhost:38739]: Error [Association failed with
[akka.tcp://spark@localhost:38739]] [akka.remote.EndpointAssociationException:
Association failed with [akka.tcp://spark@localhost:38739]Caused by:
akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2:
Connection refused: localhost/127.0.0.1:38739]14/07/07 13:55:38 INFO Master:
akka.tcp://spark@localhost:38739 got disassociated, removing it.
Master deployment logs:
14/07/07 13:44:41 INFO LocalActorRef: Message
[akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from
Actor[akka://sparkMaster/deadLetters] to
Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%40172.16.48.41%3A48763-33#-1703494807]
was not delivered. [8] dead letters encountered. This logging can be turned
off or adjusted with configuration settings 'akka.log-dead-letters' and
'akka.log-dead-letters-during-shutdown'.14/07/07 13:44:41 INFO Master:
akka.tcp://spark@localhost:47058 got disassociated, removing it.14/07/07
13:44:41 ERROR EndpointWriter: AssociationError
[akka.tcp://[email protected]:7077] ->
[akka.tcp://spark@localhost:47058]: Error [Association failed with
[akka.tcp://spark@localhost:47058]] [akka.remote.EndpointAssociationException:
Association failed with [akka.tcp://spark@localhost:47058]Caused by:
akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2:
Connection refused: localhost/127.0.0.1:47058]14/07/07 13:44:41 INFO Master:
akka.tcp://spark@localhost:47058 got disassociated, removing it.14/07/07
13:44:41 ERROR EndpointWriter: AssociationError
[akka.tcp://[email protected]:7077] ->
[akka.tcp://spark@localhost:47058]: Error [Association failed with
[akka.tcp://spark@localhost:47058]] [akka.remote.EndpointAssociationException:
Association failed with [akka.tcp://spark@localhost:47058]Caused by:
akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2:
Connection refused: localhost/127.0.0.1:47058]14/07/07 13:44:41 ERROR
EndpointWriter: AssociationError
[akka.tcp://[email protected]:7077] ->
[akka.tcp://spark@localhost:47058]: Error [Association failed with
[akka.tcp://spark@localhost:47058]] [akka.remote.EndpointAssociationException:
Association failed with [akka.tcp://spark@localhost:47058]Caused by:
akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2:
Connection refused: localhost/127.0.0.1:47058]14/07/07 13:44:41 INFO Master:
akka.tcp://spark@localhost:47058 got disassociated, removing it.
