Hi Nan,
In worker's log, I see the following exception thrown when try to launch on
executor. (The SPARK_HOME is wrongly specified on purpose, so there is no
such file "/usr/local/spark1/bin/compute-classpath.sh").
After the exception was thrown several times, the worker was requested to
kill the executor. Following the killing, the worker try to register again
with master, but master reject the registration with WARN message" Got
heartbeat from unregistered worker
worker-20140504140005-host-spark-online001"
Looks like the issue wasn't fixed in 0.9.1. Do you know any pull request
addressing this issue? Thanks.
java.io.IOException: Cannot run program
"/usr/local/spark1/bin/compute-classpath.sh" (in directory "."): error=2,
No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
at org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:600)
at
org.apache.spark.deploy.worker.CommandUtils$.buildJavaOpts(CommandUtils.scala:58)
at
org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:37)
at
org.apache.spark.deploy.worker.ExecutorRunner.getCommandSeq(ExecutorRunner.scala:104)
at
org.apache.spark.deploy.worker.ExecutorRunner.fetchAndRunExecutor(ExecutorRunner.scala:119)
at
org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:59)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:135)
at java.lang.ProcessImpl.start(ProcessImpl.java:130)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1021)
... 6 more
......
14/05/04 21:35:45 INFO Worker: Asked to kill executor
app-20140504213545-0034/18
14/05/04 21:35:45 INFO Worker: Executor app-20140504213545-0034/18 finished
with state FAILED message class java.io.IOException: Cannot run program
"/usr/local/spark1/bin/compute-classpath.sh" (in directory "."): error=2,
No such file or directory
14/05/04 21:35:45 ERROR OneForOneStrategy: key not found:
app-20140504213545-0034/18
java.util.NoSuchElementException: key not found: app-20140504213545-0034/18
at scala.collection.MapLike$class.default(MapLike.scala:228)
at scala.collection.AbstractMap.default(Map.scala:58)
at scala.collection.mutable.HashMap.apply(HashMap.scala:64)
at
org.apache.spark.deploy.worker.Worker$$anonfun$receive$1.applyOrElse(Worker.scala:232)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
14/05/04 21:35:45 INFO Worker: Starting Spark worker
host-spark-online001:7078 with 10 cores, 28.0 GB RAM
14/05/04 21:35:45 INFO Worker: Spark home: /usr/local/spark-0.9.1-cdh4.2.0
14/05/04 21:35:45 INFO WorkerWebUI: Started Worker web UI at
http://host-spark-online001:8081
14/05/04 21:35:45 INFO Worker: Connecting to master
spark://host-spark-online001:7077...
14/05/04 21:35:45 INFO Worker: Successfully registered with master
spark://host-spark-online001:7077
