Re: Actor not found
I have the same problem, i.e. exception with the same call stack when I start either pyspark or spark-shell. I use spark-1.3.0-bin-hadoop2.4 on ubuntu 14.10. bin/pyspark bunch of INFO messages, then ActorInitializationException exception. Shell starts, I can do this: >>> rd = sc.parallelize([1,2]) >>> rd.first() This call does not return. Also if I start master, and then I tried to connect shell to the master it fails to connect complaining about master URL. The same tar works fine on windows. Maybe some linux versions are not supported? Thank you Dima -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Actor-not-found-tp22265p22300.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Actor not found
Could you paste the whole stack trace here? Best Regards, Shixiong Zhu 2015-03-31 2:26 GMT+08:00 sparkdi : > I have the same problem, i.e. exception with the same call stack when I > start > either pyspark or spark-shell. I use spark-1.3.0-bin-hadoop2.4 on ubuntu > 14.10. > bin/pyspark > > bunch of INFO messages, then ActorInitializationException exception. > Shell starts, I can do this: > >>> rd = sc.parallelize([1,2]) > >>> rd.first() > This call does not return. > Also if I start master, and then I tried to connect shell to the master it > fails to connect complaining about master URL. > > The same tar works fine on windows. > > Maybe some linux versions are not supported? > Thank you > Dima > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Actor-not-found-tp22265p22300.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Re: Actor not found
This is the whole output from the shell:
~/spark-1.3.0-bin-hadoop2.4$ sudo bin/spark-shell
Spark assembly has been built with Hive, including Datanucleus jars on
classpath
log4j:WARN No appenders could be found for logger
(org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
more info.
Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
15/03/30 19:00:40 INFO SecurityManager: Changing view acls to: root
15/03/30 19:00:40 INFO SecurityManager: Changing modify acls to: root
15/03/30 19:00:40 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view pe
rmissions: Set(root); users with modify permissions: Set(root)
15/03/30 19:00:40 INFO HttpServer: Starting HTTP Server
15/03/30 19:00:40 INFO Server: jetty-8.y.z-SNAPSHOT
15/03/30 19:00:40 INFO AbstractConnector: Started
SocketConnector@0.0.0.0:47797
15/03/30 19:00:40 INFO Utils: Successfully started service 'HTTP class
server' on port 47797.
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.3.0
/_/
Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_75)
Type in expressions to have them evaluated.
Type :help for more information.
15/03/30 19:00:42 INFO SparkContext: Running Spark version 1.3.0
15/03/30 19:00:42 INFO SecurityManager: Changing view acls to: root
15/03/30 19:00:42 INFO SecurityManager: Changing modify acls to: root
15/03/30 19:00:42 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view pe
rmissions: Set(root); users with modify permissions: Set(root)
15/03/30 19:00:42 INFO Slf4jLogger: Slf4jLogger started
15/03/30 19:00:42 INFO Remoting: Starting remoting
15/03/30 19:00:43 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkDriver@vm:52574]
15/03/30 19:00:43 INFO Utils: Successfully started service 'sparkDriver' on
port 52574.
15/03/30 19:00:43 INFO SparkEnv: Registering MapOutputTracker
15/03/30 19:00:43 INFO SparkEnv: Registering BlockManagerMaster
15/03/30 19:00:43 INFO DiskBlockManager: Created local directory at
/tmp/spark-f71a8d86-6e49-4dfe-bb98-8e8581015acc/bl
ockmgr-57532f5a-38db-4ba3-86d8-edef84f592e5
15/03/30 19:00:43 INFO MemoryStore: MemoryStore started with capacity 265.4
MB
15/03/30 19:00:43 INFO HttpFileServer: HTTP File server directory is
/tmp/spark-95e0a143-0de3-4c96-861c-968c9fae2746/h
ttpd-cb029cd6-4943-479d-9b56-e7397489d9ea
15/03/30 19:00:43 INFO HttpServer: Starting HTTP Server
15/03/30 19:00:43 INFO Server: jetty-8.y.z-SNAPSHOT
15/03/30 19:00:43 INFO AbstractConnector: Started
SocketConnector@0.0.0.0:48500
15/03/30 19:00:43 INFO Utils: Successfully started service 'HTTP file
server' on port 48500.
15/03/30 19:00:43 INFO SparkEnv: Registering OutputCommitCoordinator
15/03/30 19:00:43 INFO Server: jetty-8.y.z-SNAPSHOT
15/03/30 19:00:43 INFO AbstractConnector: Started
SelectChannelConnector@0.0.0.0:4040
15/03/30 19:00:43 INFO Utils: Successfully started service 'SparkUI' on port
4040.
15/03/30 19:00:43 INFO SparkUI: Started SparkUI at http://vm:4040
15/03/30 19:00:43 INFO Executor: Starting executor ID on host
localhost
15/03/30 19:00:43 INFO Executor: Using REPL class URI:
http://10.11.204.80:47797
15/03/30 19:00:43 INFO AkkaUtils: Connecting to HeartbeatReceiver:
akka.tcp://sparkDriver@vm:5
2574/user/HeartbeatReceiver
15/03/30 19:00:43 ERROR OneForOneStrategy: Actor not found for:
ActorSelection[Anchor(akka://sparkDriver/deadLetters),
Path(/)]
akka.actor.ActorInitializationException: exception during creation
at akka.actor.ActorInitializationException$.apply(Actor.scala:164)
at akka.actor.ActorCell.create(ActorCell.scala:596)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoin
Re: Actor not found
Thanks for the log. It's really helpful. I created a JIRA to explain why it will happen: https://issues.apache.org/jira/browse/SPARK-6640 However, will this error always happens in your environment? Best Regards, Shixiong Zhu 2015-03-31 22:36 GMT+08:00 sparkdi : > This is the whole output from the shell: > > ~/spark-1.3.0-bin-hadoop2.4$ sudo bin/spark-shell > Spark assembly has been built with Hive, including Datanucleus jars on > classpath > log4j:WARN No appenders could be found for logger > (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for > more info. > Using Spark's default log4j profile: > org/apache/spark/log4j-defaults.properties > 15/03/30 19:00:40 INFO SecurityManager: Changing view acls to: root > 15/03/30 19:00:40 INFO SecurityManager: Changing modify acls to: root > 15/03/30 19:00:40 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view pe > rmissions: Set(root); users with modify permissions: Set(root) > 15/03/30 19:00:40 INFO HttpServer: Starting HTTP Server > 15/03/30 19:00:40 INFO Server: jetty-8.y.z-SNAPSHOT > 15/03/30 19:00:40 INFO AbstractConnector: Started > SocketConnector@0.0.0.0:47797 > 15/03/30 19:00:40 INFO Utils: Successfully started service 'HTTP class > server' on port 47797. > Welcome to > __ > / __/__ ___ _/ /__ > _\ \/ _ \/ _ `/ __/ '_/ >/___/ .__/\_,_/_/ /_/\_\ version 1.3.0 > /_/ > > Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_75) > Type in expressions to have them evaluated. > Type :help for more information. > 15/03/30 19:00:42 INFO SparkContext: Running Spark version 1.3.0 > 15/03/30 19:00:42 INFO SecurityManager: Changing view acls to: root > 15/03/30 19:00:42 INFO SecurityManager: Changing modify acls to: root > 15/03/30 19:00:42 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view pe > rmissions: Set(root); users with modify permissions: Set(root) > 15/03/30 19:00:42 INFO Slf4jLogger: Slf4jLogger started > 15/03/30 19:00:42 INFO Remoting: Starting remoting > 15/03/30 19:00:43 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://sparkDriver@vm:52574] > 15/03/30 19:00:43 INFO Utils: Successfully started service 'sparkDriver' on > port 52574. > 15/03/30 19:00:43 INFO SparkEnv: Registering MapOutputTracker > 15/03/30 19:00:43 INFO SparkEnv: Registering BlockManagerMaster > 15/03/30 19:00:43 INFO DiskBlockManager: Created local directory at > /tmp/spark-f71a8d86-6e49-4dfe-bb98-8e8581015acc/bl > ockmgr-57532f5a-38db-4ba3-86d8-edef84f592e5 > 15/03/30 19:00:43 INFO MemoryStore: MemoryStore started with capacity 265.4 > MB > 15/03/30 19:00:43 INFO HttpFileServer: HTTP File server directory is > /tmp/spark-95e0a143-0de3-4c96-861c-968c9fae2746/h > ttpd-cb029cd6-4943-479d-9b56-e7397489d9ea > 15/03/30 19:00:43 INFO HttpServer: Starting HTTP Server > 15/03/30 19:00:43 INFO Server: jetty-8.y.z-SNAPSHOT > 15/03/30 19:00:43 INFO AbstractConnector: Started > SocketConnector@0.0.0.0:48500 > 15/03/30 19:00:43 INFO Utils: Successfully started service 'HTTP file > server' on port 48500. > 15/03/30 19:00:43 INFO SparkEnv: Registering OutputCommitCoordinator > 15/03/30 19:00:43 INFO Server: jetty-8.y.z-SNAPSHOT > 15/03/30 19:00:43 INFO AbstractConnector: Started > SelectChannelConnector@0.0.0.0:4040 > 15/03/30 19:00:43 INFO Utils: Successfully started service 'SparkUI' on > port > 4040. > 15/03/30 19:00:43 INFO SparkUI: Started SparkUI at http://vm:4040 > 15/03/30 19:00:43 INFO Executor: Starting executor ID on host > localhost > 15/03/30 19:00:43 INFO Executor: Using REPL class URI: > http://10.11.204.80:47797 > 15/03/30 19:00:43 INFO AkkaUtils: Connecting to HeartbeatReceiver: > akka.tcp://sparkDriver@vm:5 > 2574/user/HeartbeatReceiver > 15/03/30 19:00:43 ERROR OneForOneStrategy: Actor not found for: > ActorSelection[Anchor(akka://sparkDriver/deadLetters), > Path(/)] > akka.actor.ActorInitializationException: exception during creation > at akka.actor.ActorInitializationException$.apply(Actor.scala:164) > at akka.actor.ActorCell.create(ActorCell.scala:596) > at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456) > at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) > at > akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263) > at akka.dispatch.Mailbox.run(Mailbox.scala:219) > at > > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJ
Re: Actor not found
13119 Exception in thread "main" akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka.tcp://sparkdri...@dmslave13.et2.tbsi te.net:5908/), Path(/user/OutputCommitCoordinator)] 13120 at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65) 13121 at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63) 13122 at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) 13123 at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67) 13124 at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82) 13125 at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) 13126 at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) 13127 at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) 13128 at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) 13129 at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74) 13130 at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110) 13131 at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73) 13132 at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) 13133 at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) 13134 at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267) 13135 at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508) 13136 at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541) 13137 at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531) 13138 at akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87) 13139 at akka.remote.EndpointManager$$anonfun$1.applyOrElse(Remoting.scala:575) 13140 at akka.actor.Actor$class.aroundReceive(Actor.scala:465) 13141 at akka.remote.EndpointManager.aroundReceive(Remoting.scala:395) 13142 at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) 13143 at akka.actor.ActorCell.invoke(ActorCell.scala:487) 13144 at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) 13145 at akka.dispatch.Mailbox.run(Mailbox.scala:220) 13146 at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) 13147 at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) 13148 at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) 13149 at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) 13150 at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) I met the same problem when I run spark on yarn. Is this a bug or what ? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Actor-not-found-tp22265p22508.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Actor not found
Forgot this one: I cannot find any issue about creating OutputCommitCoordinator. The order of creating OutputCommitCoordinato looks right. Best Regards, Shixiong(Ryan) Zhu 2015-04-17 16:57 GMT+08:00 Shixiong Zhu : > I just checked the codes about creating OutputCommitCoordinator. Could you > reproduce this issue? If so, could you provide details about how to > reproduce it? > > Best Regards, > Shixiong(Ryan) Zhu > > 2015-04-16 13:27 GMT+08:00 Canoe : > >> 13119 Exception in thread "main" akka.actor.ActorNotFound: Actor not found >> for: ActorSelection[Anchor(akka.tcp://sparkdri...@dmslave13.et2.tbsi >> te.net:5908/), Path(/user/OutputCommitCoordinator)] >> 13120 at >> >> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65) >> 13121 at >> >> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63) >> 13122 at >> scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) >> 13123 at >> >> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67) >> 13124 at >> >> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82) >> 13125 at >> >> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) >> 13126 at >> >> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) >> 13127 at >> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) >> 13128 at >> akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) >> 13129 at >> >> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74) >> 13130 at >> akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110) >> 13131 at >> >> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73) >> 13132 at >> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) >> 13133 at >> >> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) >> 13134 at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267) >> 13135 at >> akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508) >> 13136 at >> akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541) >> 13137 at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531) >> 13138 at >> >> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87) >> 13139 at >> akka.remote.EndpointManager$$anonfun$1.applyOrElse(Remoting.scala:575) >> 13140 at akka.actor.Actor$class.aroundReceive(Actor.scala:465) >> 13141 at >> akka.remote.EndpointManager.aroundReceive(Remoting.scala:395) >> 13142 at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) >> 13143 at akka.actor.ActorCell.invoke(ActorCell.scala:487) >> 13144 at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) >> 13145 at akka.dispatch.Mailbox.run(Mailbox.scala:220) >> 13146 at >> >> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) >> 13147 at >> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >> 13148 at >> >> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >> 13149 at >> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >> 13150 at >> >> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >> >> >> I met the same problem when I run spark on yarn. Is this a bug or what ? >> >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Actor-not-found-tp22265p22508.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> - >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
Re: Actor not found
I just checked the codes about creating OutputCommitCoordinator. Could you reproduce this issue? If so, could you provide details about how to reproduce it? Best Regards, Shixiong(Ryan) Zhu 2015-04-16 13:27 GMT+08:00 Canoe : > 13119 Exception in thread "main" akka.actor.ActorNotFound: Actor not found > for: ActorSelection[Anchor(akka.tcp://sparkdri...@dmslave13.et2.tbsi > te.net:5908/), Path(/user/OutputCommitCoordinator)] > 13120 at > > akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65) > 13121 at > > akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63) > 13122 at > scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) > 13123 at > > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67) > 13124 at > > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82) > 13125 at > > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) > 13126 at > > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) > 13127 at > scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) > 13128 at > akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) > 13129 at > > akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74) > 13130 at > akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110) > 13131 at > > akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73) > 13132 at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) > 13133 at > scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) > 13134 at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267) > 13135 at > akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508) > 13136 at > akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541) > 13137 at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531) > 13138 at > > akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87) > 13139 at > akka.remote.EndpointManager$$anonfun$1.applyOrElse(Remoting.scala:575) > 13140 at akka.actor.Actor$class.aroundReceive(Actor.scala:465) > 13141 at > akka.remote.EndpointManager.aroundReceive(Remoting.scala:395) > 13142 at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) > 13143 at akka.actor.ActorCell.invoke(ActorCell.scala:487) > 13144 at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) > 13145 at akka.dispatch.Mailbox.run(Mailbox.scala:220) > 13146 at > > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) > 13147 at > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > 13148 at > > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > 13149 at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > 13150 at > > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > > > I met the same problem when I run spark on yarn. Is this a bug or what ? > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Actor-not-found-tp22265p22508.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Re: Actor not found
Hi, Shixiong: Actually, I know nothing about this exception. I submitted a job that would read about 2.5T data and it threw this exception. And also I try to submit some jobs that can run successfully before this submission, it also failed with the same exception. Hope this will help you to do the troubleshooting. PS: spark 1.3.0 hadoop version: 2.3.0-cdh5.1.0. All my best, Zhihang Fan 2015-04-17 16:59 GMT+08:00 Shixiong Zhu : > Forgot this one: I cannot find any issue about creating > OutputCommitCoordinator. The order of creating OutputCommitCoordinato looks > right. > > Best Regards, > Shixiong(Ryan) Zhu > > 2015-04-17 16:57 GMT+08:00 Shixiong Zhu : > >> I just checked the codes about creating OutputCommitCoordinator. Could >> you reproduce this issue? If so, could you provide details about how to >> reproduce it? >> >> Best Regards, >> Shixiong(Ryan) Zhu >> >> 2015-04-16 13:27 GMT+08:00 Canoe : >> >>> 13119 Exception in thread "main" akka.actor.ActorNotFound: Actor not >>> found >>> for: ActorSelection[Anchor(akka.tcp://sparkdri...@dmslave13.et2.tbsi >>> te.net:5908/), Path(/user/OutputCommitCoordinator)] >>> 13120 at >>> >>> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65) >>> 13121 at >>> >>> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63) >>> 13122 at >>> scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) >>> 13123 at >>> >>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67) >>> 13124 at >>> >>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82) >>> 13125 at >>> >>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) >>> 13126 at >>> >>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) >>> 13127 at >>> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) >>> 13128 at >>> akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) >>> 13129 at >>> >>> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74) >>> 13130 at >>> akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110) >>> 13131 at >>> >>> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73) >>> 13132 at >>> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) >>> 13133 at >>> >>> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) >>> 13134 at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267) >>> 13135 at >>> akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508) >>> 13136 at >>> akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541) >>> 13137 at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531) >>> 13138 at >>> >>> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87) >>> 13139 at >>> akka.remote.EndpointManager$$anonfun$1.applyOrElse(Remoting.scala:575) >>> 13140 at akka.actor.Actor$class.aroundReceive(Actor.scala:465) >>> 13141 at >>> akka.remote.EndpointManager.aroundReceive(Remoting.scala:395) >>> 13142 at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) >>> 13143 at akka.actor.ActorCell.invoke(ActorCell.scala:487) >>> 13144 at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) >>> 13145 at akka.dispatch.Mailbox.run(Mailbox.scala:220) >>> 13146 at >>> >>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) >>> 13147 at >>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >>> 13148 at >>> >>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >>> 13149 at >>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >>> 13150 at >>> >>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >>> >>> >>> I met the same problem when I run spark on yarn. Is this a bug or what ? >>> >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/Actor-not-found-tp22265p22508.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> - >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >> > -- 谁谓河广，一苇航之
Actor not found for: ActorSelection
I just cloned the master repository of Spark from Github. I am running it on OSX 10.9, Spark 1.4.1 and Scala 2.10.4 I just tried to run the SparkPi example program using IntelliJ Idea but get the error : akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka.tcp://sparkMaster@myhost:7077/) I did checkout a similar post <http://apache-spark-user-list.1001560.n3.nabble.com/Actor-not-found-td22265.html> but found no solution. Find the complete stack trace below. Any help would be really appreciated. 2015-07-28 22:16:45,888 INFO [main] spark.SparkContext (Logging.scala:logInfo(59)) - Running Spark version 1.5.0-SNAPSHOT 2015-07-28 22:16:47,125 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2015-07-28 22:16:47,753 INFO [main] spark.SecurityManager (Logging.scala:logInfo(59)) - Changing view acls to: mac 2015-07-28 22:16:47,755 INFO [main] spark.SecurityManager (Logging.scala:logInfo(59)) - Changing modify acls to: mac 2015-07-28 22:16:47,756 INFO [main] spark.SecurityManager (Logging.scala:logInfo(59)) - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(mac); users with modify permissions: Set(mac) 2015-07-28 22:16:49,454 INFO [sparkDriver-akka.actor.default-dispatcher-2] slf4j.Slf4jLogger (Slf4jLogger.scala:applyOrElse(80)) - Slf4jLogger started 2015-07-28 22:16:49,695 INFO [sparkDriver-akka.actor.default-dispatcher-2] Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Starting remoting 2015-07-28 22:16:50,167 INFO [sparkDriver-akka.actor.default-dispatcher-2] Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.2.105:49981] 2015-07-28 22:16:50,215 INFO [main] util.Utils (Logging.scala:logInfo(59)) - Successfully started service 'sparkDriver' on port 49981. 2015-07-28 22:16:50,372 INFO [main] spark.SparkEnv (Logging.scala:logInfo(59)) - Registering MapOutputTracker 2015-07-28 22:16:50,596 INFO [main] spark.SparkEnv (Logging.scala:logInfo(59)) - Registering BlockManagerMaster 2015-07-28 22:16:50,948 INFO [main] storage.DiskBlockManager (Logging.scala:logInfo(59)) - Created local directory at /private/var/folders/8k/jfw576r50m97rlk5qpj1n4l8gn/T/blockmgr-309db4d1-d129-43e5-a90e-12cf51ad491f 2015-07-28 22:16:51,198 INFO [main] storage.MemoryStore (Logging.scala:logInfo(59)) - MemoryStore started with capacity 491.7 MB 2015-07-28 22:16:51,707 INFO [main] spark.HttpFileServer (Logging.scala:logInfo(59)) - HTTP File server directory is /private/var/folders/8k/jfw576r50m97rlk5qpj1n4l8gn/T/spark-f28e24e7-b798-4365-8209-409d8b27ad2f/httpd-ce32c41d-b618-49e9-bec1-f409454f3679 2015-07-28 22:16:51,777 INFO [main] spark.HttpServer (Logging.scala:logInfo(59)) - Starting HTTP Server 2015-07-28 22:16:52,091 INFO [main] server.Server (Server.java:doStart(272)) - jetty-8.1.14.v20131031 2015-07-28 22:16:52,116 INFO [main] server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started SocketConnector@0.0.0.0:49982 2015-07-28 22:16:52,116 INFO [main] util.Utils (Logging.scala:logInfo(59)) - Successfully started service 'HTTP file server' on port 49982. 2015-07-28 22:16:52,249 INFO [main] spark.SparkEnv (Logging.scala:logInfo(59)) - Registering OutputCommitCoordinator 2015-07-28 22:16:54,253 INFO [main] server.Server (Server.java:doStart(272)) - jetty-8.1.14.v20131031 2015-07-28 22:16:54,315 INFO [main] server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started SelectChannelConnector@0.0.0.0:4040 2015-07-28 22:16:54,317 INFO [main] util.Utils (Logging.scala:logInfo(59)) - Successfully started service 'SparkUI' on port 4040. 2015-07-28 22:16:54,386 INFO [main] ui.SparkUI (Logging.scala:logInfo(59)) - Started SparkUI at http://192.168.2.105:4040 2015-07-28 22:16:54,924 WARN [main] metrics.MetricsSystem (Logging.scala:logWarning(71)) - Using default name DAGScheduler for source because spark.app.id is not set. 2015-07-28 22:16:55,132 INFO [appclient-register-master-threadpool-0] client.AppClient$ClientEndpoint (Logging.scala:logInfo(59)) - Connecting to master spark://myhost:7077... 2015-07-28 22:16:55,392 WARN [sparkDriver-akka.actor.default-dispatcher-14] client.AppClient$ClientEndpoint (Logging.scala:logWarning(71)) - Could not connect to myhost:7077: akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkMaster@myhost:7077] 2015-07-28 22:16:55,412 WARN [sparkDriver-akka.actor.default-dispatcher-14] remote.ReliableDeliverySupervisor (Slf4jLogger.scala:apply$mcV$sp(71)) - Association with remote system [akka.tcp://sparkMaster@myhost:7077] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://sparkMaster@myhost:7077]] Caused by: [myhost: unknown error] 2015-07-28 22:16:55,447 WARN [appclient-register-master-threadpoo
Spark 1.4.0-rc3: Actor not found
Just compiled Spark 1.4.0-rc3 for Yarn 2.2 and tried running a job that worked fine for Spark 1.3. The job starts on the cluster (yarn-cluster mode), initial stage starts, but the job fails before any task succeeds with the following error. Any hints? [ERROR] [06/02/2015 09:05:36.962] [Executor task launch worker-0] [akka.tcp://sparkDriver@10.254.6.15:33986/user/CoarseGrainedScheduler] swallowing exception during message send (akka.remote.RemoteTransportExceptionNoStackTrace) Exception in thread "main" akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka.tcp://sparkDriver@10.254.6.15:33986/), Path(/user/OutputCommitCoordinator)] at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65) at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74) at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110) at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73) at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267) at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508) at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541) at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531) at akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87) at akka.remote.EndpointManager$$anonfun$1.applyOrElse(Remoting.scala:575) at akka.actor.Actor$class.aroundReceive(Actor.scala:465) at akka.remote.EndpointManager.aroundReceive(Remoting.scala:395) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) at akka.actor.ActorCell.invoke(ActorCell.scala:487) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) at akka.dispatch.Mailbox.run(Mailbox.scala:220) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
Re: Actor not found for: ActorSelection
The problem was that I was trying to start the example app in standalone cluster mode by passing in *-Dspark.master=spark://myhost:7077* as an argument to the JVM. I launched the example app locally using -*Dspark.master=local* and it worked. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Actor-not-found-for-ActorSelection-tp24035p24037.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Spark 1.4.0-rc3: Actor not found
How about other jobs? Is it an executor log, or a driver log? Could you post other logs near this error, please? Thank you. Best Regards, Shixiong Zhu 2015-06-02 17:11 GMT+08:00 Anders Arpteg : > Just compiled Spark 1.4.0-rc3 for Yarn 2.2 and tried running a job that > worked fine for Spark 1.3. The job starts on the cluster (yarn-cluster > mode), initial stage starts, but the job fails before any task succeeds > with the following error. Any hints? > > [ERROR] [06/02/2015 09:05:36.962] [Executor task launch worker-0] > [akka.tcp://sparkDriver@10.254.6.15:33986/user/CoarseGrainedScheduler] > swallowing exception during message send > (akka.remote.RemoteTransportExceptionNoStackTrace) > Exception in thread "main" akka.actor.ActorNotFound: Actor not found for: > ActorSelection[Anchor(akka.tcp://sparkDriver@10.254.6.15:33986/), > Path(/user/OutputCommitCoordinator)] > at > akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65) > at > akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63) > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) > at > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67) > at > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82) > at > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) > at > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) > at > scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) > at > akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) > at > akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74) > at > akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110) > at > akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73) > at > scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) > at > scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) > at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267) > at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508) > at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541) > at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531) > at > akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87) > at > akka.remote.EndpointManager$$anonfun$1.applyOrElse(Remoting.scala:575) > at akka.actor.Actor$class.aroundReceive(Actor.scala:465) > at akka.remote.EndpointManager.aroundReceive(Remoting.scala:395) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) > at akka.actor.ActorCell.invoke(ActorCell.scala:487) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) > at akka.dispatch.Mailbox.run(Mailbox.scala:220) > at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) > at > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > >
Re: Spark 1.4.0-rc3: Actor not found
The log is from the log aggregation tool (hortonworks, "yarn logs ..."), so both executors and driver. I'll send a private mail to you with the full logs. Also, tried another job as you suggested, and it actually worked fine. The first job was reading from a parquet source, and the second from an avro source. Could there be some issues with the parquet reader? Thanks, Anders On Tue, Jun 2, 2015 at 11:53 AM, Shixiong Zhu wrote: > How about other jobs? Is it an executor log, or a driver log? Could you > post other logs near this error, please? Thank you. > > Best Regards, > Shixiong Zhu > > 2015-06-02 17:11 GMT+08:00 Anders Arpteg : > >> Just compiled Spark 1.4.0-rc3 for Yarn 2.2 and tried running a job that >> worked fine for Spark 1.3. The job starts on the cluster (yarn-cluster >> mode), initial stage starts, but the job fails before any task succeeds >> with the following error. Any hints? >> >> [ERROR] [06/02/2015 09:05:36.962] [Executor task launch worker-0] >> [akka.tcp://sparkDriver@10.254.6.15:33986/user/CoarseGrainedScheduler] >> swallowing exception during message send >> (akka.remote.RemoteTransportExceptionNoStackTrace) >> Exception in thread "main" akka.actor.ActorNotFound: Actor not found for: >> ActorSelection[Anchor(akka.tcp://sparkDriver@10.254.6.15:33986/), >> Path(/user/OutputCommitCoordinator)] >> at >> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65) >> at >> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63) >> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) >> at >> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67) >> at >> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82) >> at >> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) >> at >> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) >> at >> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) >> at >> akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) >> at >> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74) >> at >> akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110) >> at >> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73) >> at >> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) >> at >> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) >> at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267) >> at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508) >> at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541) >> at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531) >> at >> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87) >> at >> akka.remote.EndpointManager$$anonfun$1.applyOrElse(Remoting.scala:575) >> at akka.actor.Actor$class.aroundReceive(Actor.scala:465) >> at akka.remote.EndpointManager.aroundReceive(Remoting.scala:395) >> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) >> at akka.actor.ActorCell.invoke(ActorCell.scala:487) >> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) >> at akka.dispatch.Mailbox.run(Mailbox.scala:220) >> at >> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) >> at >> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >> at >> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >> >> >
Re: Spark 1.4.0-rc3: Actor not found
Does it happen every time you read a parquet source? On Tue, Jun 2, 2015 at 3:42 AM, Anders Arpteg wrote: > The log is from the log aggregation tool (hortonworks, "yarn logs ..."), > so both executors and driver. I'll send a private mail to you with the full > logs. Also, tried another job as you suggested, and it actually worked > fine. The first job was reading from a parquet source, and the second from > an avro source. Could there be some issues with the parquet reader? > > Thanks, > Anders > > On Tue, Jun 2, 2015 at 11:53 AM, Shixiong Zhu wrote: > >> How about other jobs? Is it an executor log, or a driver log? Could you >> post other logs near this error, please? Thank you. >> >> Best Regards, >> Shixiong Zhu >> >> 2015-06-02 17:11 GMT+08:00 Anders Arpteg : >> >>> Just compiled Spark 1.4.0-rc3 for Yarn 2.2 and tried running a job that >>> worked fine for Spark 1.3. The job starts on the cluster (yarn-cluster >>> mode), initial stage starts, but the job fails before any task succeeds >>> with the following error. Any hints? >>> >>> [ERROR] [06/02/2015 09:05:36.962] [Executor task launch worker-0] >>> [akka.tcp://sparkDriver@10.254.6.15:33986/user/CoarseGrainedScheduler] >>> swallowing exception during message send >>> (akka.remote.RemoteTransportExceptionNoStackTrace) >>> Exception in thread "main" akka.actor.ActorNotFound: Actor not found >>> for: ActorSelection[Anchor(akka.tcp://sparkDriver@10.254.6.15:33986/), >>> Path(/user/OutputCommitCoordinator)] >>> at >>> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65) >>> at >>> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63) >>> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) >>> at >>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67) >>> at >>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82) >>> at >>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) >>> at >>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) >>> at >>> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) >>> at >>> akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) >>> at >>> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74) >>> at >>> akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110) >>> at >>> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73) >>> at >>> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) >>> at >>> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) >>> at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267) >>> at >>> akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508) >>> at >>> akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541) >>> at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531) >>> at >>> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87) >>> at >>> akka.remote.EndpointManager$$anonfun$1.applyOrElse(Remoting.scala:575) >>> at akka.actor.Actor$class.aroundReceive(Actor.scala:465) >>> at akka.remote.EndpointManager.aroundReceive(Remoting.scala:395) >>> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) >>> at akka.actor.ActorCell.invoke(ActorCell.scala:487) >>> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) >>> at akka.dispatch.Mailbox.run(Mailbox.scala:220) >>> at >>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) >>> at >>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >>> at >>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >>> >>> >> >
Re: Spark 1.4.0-rc3: Actor not found
Tried on some other data sources as well, and it actually works for some parquet sources. Potentially some specific problems with that first parquet source that I tried with, and not a Spark 1.4 problem. I'll get back with more info if I find any new information. Thanks, Anders On Tue, Jun 2, 2015 at 8:45 PM, Yin Huai wrote: > Does it happen every time you read a parquet source? > > On Tue, Jun 2, 2015 at 3:42 AM, Anders Arpteg wrote: > >> The log is from the log aggregation tool (hortonworks, "yarn logs ..."), >> so both executors and driver. I'll send a private mail to you with the full >> logs. Also, tried another job as you suggested, and it actually worked >> fine. The first job was reading from a parquet source, and the second from >> an avro source. Could there be some issues with the parquet reader? >> >> Thanks, >> Anders >> >> On Tue, Jun 2, 2015 at 11:53 AM, Shixiong Zhu wrote: >> >>> How about other jobs? Is it an executor log, or a driver log? Could you >>> post other logs near this error, please? Thank you. >>> >>> Best Regards, >>> Shixiong Zhu >>> >>> 2015-06-02 17:11 GMT+08:00 Anders Arpteg : >>> >>>> Just compiled Spark 1.4.0-rc3 for Yarn 2.2 and tried running a job that >>>> worked fine for Spark 1.3. The job starts on the cluster (yarn-cluster >>>> mode), initial stage starts, but the job fails before any task succeeds >>>> with the following error. Any hints? >>>> >>>> [ERROR] [06/02/2015 09:05:36.962] [Executor task launch worker-0] >>>> [akka.tcp://sparkDriver@10.254.6.15:33986/user/CoarseGrainedScheduler] >>>> swallowing exception during message send >>>> (akka.remote.RemoteTransportExceptionNoStackTrace) >>>> Exception in thread "main" akka.actor.ActorNotFound: Actor not found >>>> for: ActorSelection[Anchor(akka.tcp://sparkDriver@10.254.6.15:33986/), >>>> Path(/user/OutputCommitCoordinator)] >>>> at >>>> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65) >>>> at >>>> akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63) >>>> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) >>>> at >>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67) >>>> at >>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82) >>>> at >>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) >>>> at >>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) >>>> at >>>> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) >>>> at >>>> akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) >>>> at >>>> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74) >>>> at >>>> akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110) >>>> at >>>> akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73) >>>> at >>>> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) >>>> at >>>> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) >>>> at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267) >>>> at >>>> akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508) >>>> at >>>> akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541) >>>> at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531) >>>> at >>>> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87) >>>> at >>>> akka.remote.EndpointManager$$anonfun$1.applyOrElse(Remoting.scala:575) >>>> at akka.actor.Actor$class.aroundReceive(Actor.scala:465) >>>> at akka.remote.EndpointManager.aroundReceive(Remoting.scala:395) >>>> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) >>>> at akka.actor.ActorCell.invoke(ActorCell.scala:487) >>>> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) >>>> at akka.dispatch.Mailbox.run(Mailbox.scala:220) >>>> at >>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) >>>> at >>>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >>>> at >>>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >>>> >>>> >>> >> >
