You're specifying the queue in the spark-submit command line: --queue thequeue
Are you sure that queue exists? On Thu, Jan 15, 2015 at 11:23 AM, Manoj Samel <manojsamelt...@gmail.com> wrote: > Hi, > > Setup is as follows > > Hadoop Cluster 2.3.0 (CDH5.0) > - Namenode HA > - Resource manager HA > - Secured with Kerberos > > Spark 1.2 > > Run SparkPi as follows > - conf/spark-defaults.conf has following entries > spark.yarn.queue myqueue > spark.yarn.access.namenodes hdfs://namespace (remember this is namenode HA) > - Do kinit with some user keytab > - submit SparkPi as follows > spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client > --num-executors 3 --driver-memory 4g --executor-memory 2g --executor-cores 1 > --queue thequeue $MY_SPARK_DIR/lib/spark-examples*.jar 10 > > Gives following trace (not sure why it shows unknown queue when queue name > is specified in the spark-defaults.conf above. > > 15/01/15 19:18:27 INFO impl.YarnClientImpl: Submitted application > application_1415648563285_31469 > 15/01/15 19:18:28 INFO yarn.Client: Application report for > application_1415648563285_31469 (state: FAILED) > 15/01/15 19:18:28 INFO yarn.Client: > client token: N/A > diagnostics: Application application_1415648563285_31469 submitted by user > XYZ to unknown queue: thequeue <<--- WHY UNKNOWN QUEUE ??? > ApplicationMaster host: N/A > ApplicationMaster RPC port: -1 > queue: thequeue <<--- WHY UNKNOWN QUEUE ??? > start time: 1421349507652 > final status: FAILED > tracking URL: N/A > user: XYZ > Exception in thread "main" org.apache.spark.SparkException: Yarn application > has already ended! It might have been killed or unable to launch application > master. > at > org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:102) > at > org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:58) -- Marcelo --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
