Hi Sam,
Did you specify the MASTER in your spark-env.sh? I ask because I didn't see
a --master in your launch command. Also, your app seems to take in a master
("yarn-standalone"). This is not exactly correct because by the time the
SparkContext is launched locally, which is the default, it is too late to
use "yarn-cluster" mode by definition, since the driver should launched
within one of the containers on the worker machines.
I would suggest the following:
- change your application to not take in the Spark master as a command line
argument
- use "yarn-cluster" instead of "yarn-standalone" (which is deprecated)
- add "--master yarn-cluster" in your spark-submit command
Another worrying thing is the warning from your logs:
14/07/21 22:38:42 WARN spark.SparkConf: null jar passed to SparkContext
constructor
How are you creating your SparkContext?
Andrew
2014-07-21 7:47 GMT-07:00 Sam Liu <liuqiyun_sp...@sina.com>:
> Hi Experts,
>
>
> I setup Yarn and Spark env: all services runs on a single node. And then
> submited a WordCount job using spark-submit script with command:
>
> ./bin/spark-submit tests/wordcount-spark-scala.jar --class
> scala.spark.WordCount --num-executors 1 --driver-memory 300M
> --executor-memory 300M --executor-cores 1 "yarn-standalone"
> "hdfs://hostname/tmp/input" "hdfs://hostname/tmp/output"
>
>
> However, the command hangs and no job is submited to Yarn. Any comments?
>
>
> output:
>
> Spark assembly has been built with Hive, including Datanucleus jars on
> classpath
> 14/07/21 22:38:42 WARN spark.SparkConf: null jar passed to SparkContext
> constructor
> 14/07/21 22:38:43 INFO spark.SecurityManager: Changing view acls to:
> biadmin
> 14/07/21 22:38:43 INFO spark.SecurityManager: SecurityManager:
> authentication disabled; ui acls disabled; users with view permissions:
> Set(biadmin)
> 14/07/21 22:38:43 INFO slf4j.Slf4jLogger: Slf4jLogger started
> 14/07/21 22:38:43 INFO Remoting: Starting remoting
> 14/07/21 22:38:43 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://spark@hostname:56903]
> 14/07/21 22:38:43 INFO Remoting: Remoting now listens on addresses:
> [akka.tcp://spark@hostname:56903]
> 14/07/21 22:38:43 INFO spark.SparkEnv: Registering MapOutputTracker
> 14/07/21 22:38:43 INFO spark.SparkEnv: Registering BlockManagerMaster
> 14/07/21 22:38:43 INFO storage.DiskBlockManager: Created local directory
> at /tmp/spark-local-20140721223843-75cd
> 14/07/21 22:38:43 INFO storage.MemoryStore: MemoryStore started with
> capacity 180.0 MB.
> 14/07/21 22:38:43 INFO network.ConnectionManager: Bound socket to port
> 57453 with id = ConnectionManagerId(hostname,57453)
> 14/07/21 22:38:43 INFO storage.BlockManagerMaster: Trying to register
> BlockManager
> 14/07/21 22:38:43 INFO storage.BlockManagerInfo: Registering block manager
> hostname:57453 with 180.0 MB RAM
> 14/07/21 22:38:43 INFO storage.BlockManagerMaster: Registered BlockManager
> 14/07/21 22:38:43 INFO spark.HttpServer: Starting HTTP Server
> 14/07/21 22:38:43 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 14/07/21 22:38:43 INFO server.AbstractConnector: Started
> SocketConnector@0.0.0.0:19323
> 14/07/21 22:38:43 INFO broadcast.HttpBroadcast: Broadcast server started
> at http://9.123.99.10:19323
> 14/07/21 22:38:43 INFO spark.HttpFileServer: HTTP File server directory is
> /tmp/spark-e224a31b-4517-43d8-9778-4b6af07dcad2
> 14/07/21 22:38:43 INFO spark.HttpServer: Starting HTTP Server
> 14/07/21 22:38:43 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 14/07/21 22:38:43 INFO server.AbstractConnector: Started
> SocketConnector@0.0.0.0:35420
> 14/07/21 22:38:43 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 14/07/21 22:38:43 INFO server.AbstractConnector: Started
> SelectChannelConnector@0.0.0.0:4040
> 14/07/21 22:38:43 INFO ui.SparkUI: Started SparkUI at http://hostname:4040
> 14/07/21 22:38:44 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/07/21 22:38:44 WARN spark.SparkContext: "yarn-standalone" is deprecated
> as of Spark 1.0. Use "yarn-cluster" instead.
> 14/07/21 22:38:44 INFO cluster.YarnClusterScheduler: Created
> YarnClusterScheduler
> 14/07/21 22:38:44 INFO yarn.ApplicationMaster$$anon$1: Adding shutdown
> hook for context org.apache.spark.SparkContext@610c610c
>
>
>
>
> Thanks!
> --------------------------------
> Sam Liu
>
>
