Re: spark application was submitted twice unexpectedly
looking into the work folder of problematic application, seems that the application is continuing creating executors, and error log of worker is as below: Exception in thread "main" java.lang.reflect.UndeclaredThrowableException: Unknown exception in doAs at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1134) at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:52) at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:113) at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:156) at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala) Caused by: java.security.PrivilegedActionException: java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) ... 4 more Caused by: java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) at scala.concurrent.Await$.result(package.scala:107) at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:125) at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:53) at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:52) ... 7 more In master's log, I also see errors are generated continuously: ... 15/04/19 17:24:20 ERROR EndpointWriter: dropping message [class org.apache.spark.deploy.DeployMessages$ExecutorUpdated] for non-local recipient [Actor[akka.tcp://sparkDriver@spark101:46215/user/$b#-642262056]] arriving at [akka.tcp://sparkDriver@spark101:46215] inbound addresses are [akka.tcp://sparkMaster@spark101:7077] 15/04/19 17:24:20 ERROR EndpointWriter: dropping message [class org.apache.spark.deploy.DeployMessages$ExecutorAdded] for non-local recipient [Actor[akka.tcp://sparkDriver@spark101:46215/user/$b#-642262056]] arriving at [akka.tcp://sparkDriver@spark101:46215] inbound addresses are [akka.tcp://sparkMaster@spark101:7077] 15/04/19 17:24:20 ERROR EndpointWriter: dropping message [class org.apache.spark.deploy.DeployMessages$ExecutorUpdated] for non-local recipient [Actor[akka.tcp://sparkDriver@spark101:46215/user/$b#-642262056]] arriving at [akka.tcp://sparkDriver@spark101:46215] inbound addresses are [akka.tcp://sparkMaster@spark101:7077] 15/04/19 17:24:21 ERROR EndpointWriter: dropping message [class org.apache.spark.deploy.DeployMessages$ExecutorUpdated] for non-local recipient [Actor[akka.tcp://sparkDriver@spark101:46215/user/$b#-642262056]] arriving at [akka.tcp://sparkDriver@spark101:46215] inbound addresses are [akka.tcp://sparkMaster@spark101:7077] 15/04/19 17:24:21 ERROR UserGroupInformation: PriviledgedActionException as:root cause:java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] 15/04/19 17:24:22 ERROR EndpointWriter: dropping message [class org.apache.spark.deploy.DeployMessages$ExecutorUpdated] for non-local recipient [Actor[akka.tcp://sparkDriver@spark101:53140/user/$b#-510580371]] arriving at [akka.tcp://sparkDriver@spark101:53140] inbound addresses are [akka.tcp://sparkMaster@spark101:7077] 15/04/19 17:24:22 ERROR EndpointWriter: dropping message [class org.apache.spark.deploy.DeployMessages$ExecutorAdded] for non-local recipient [Actor[akka.tcp://sparkDriver@spark101:53140/user/$b#-510580371]] arriving at [akka.tcp://sparkDriver@spark101:53140] inbound addresses are [akka.tcp://sparkMaster@spark101:7077] ... -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-application-was-submitted-twice-unexpectedly-tp22551p22560.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
spark application was submitted twice unexpectedly
Hi guys, I was trying to submit several spark applications to a standalone cluster at the same time from a shell script. One issue I met is sometimes one application may be submitted to cluster twice unexpectedly(from web UI, I can see the two applications of same name were generated exactly at the same time). One of the two applications will run to end, but the other will always stay in running state and never exit and release resources. Does anyone meet the same issue? The spark version I am using is spark1.1.1. Best Regards, Pengcheng -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-application-was-submitted-twice-unexpectedly-tp22551.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
