Hi Sam,
Did you specify the MASTER in your spark-env.sh? I ask because I didn't see
a --master in your launch command. Also, your app seems to take in a master
(yarn-standalone). This is not exactly correct because by the time the
SparkContext is launched locally, which is the default, it is too late to
use yarn-cluster mode by definition, since the driver should launched
within one of the containers on the worker machines.
I would suggest the following:
- change your application to not take in the Spark master as a command line
argument
- use yarn-cluster instead of yarn-standalone (which is deprecated)
- add --master yarn-cluster in your spark-submit command
Another worrying thing is the warning from your logs:
14/07/21 22:38:42 WARN spark.SparkConf: null jar passed to SparkContext
constructor
How are you creating your SparkContext?
Andrew
2014-07-21 7:47 GMT-07:00 Sam Liu liuqiyun_sp...@sina.com:
Hi Experts,
I setup Yarn and Spark env: all services runs on a single node. And then
submited a WordCount job using spark-submit script with command:
./bin/spark-submit tests/wordcount-spark-scala.jar --class
scala.spark.WordCount --num-executors 1 --driver-memory 300M
--executor-memory 300M --executor-cores 1 yarn-standalone
hdfs://hostname/tmp/input hdfs://hostname/tmp/output
However, the command hangs and no job is submited to Yarn. Any comments?
output:
Spark assembly has been built with Hive, including Datanucleus jars on
classpath
14/07/21 22:38:42 WARN spark.SparkConf: null jar passed to SparkContext
constructor
14/07/21 22:38:43 INFO spark.SecurityManager: Changing view acls to:
biadmin
14/07/21 22:38:43 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(biadmin)
14/07/21 22:38:43 INFO slf4j.Slf4jLogger: Slf4jLogger started
14/07/21 22:38:43 INFO Remoting: Starting remoting
14/07/21 22:38:43 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://spark@hostname:56903]
14/07/21 22:38:43 INFO Remoting: Remoting now listens on addresses:
[akka.tcp://spark@hostname:56903]
14/07/21 22:38:43 INFO spark.SparkEnv: Registering MapOutputTracker
14/07/21 22:38:43 INFO spark.SparkEnv: Registering BlockManagerMaster
14/07/21 22:38:43 INFO storage.DiskBlockManager: Created local directory
at /tmp/spark-local-20140721223843-75cd
14/07/21 22:38:43 INFO storage.MemoryStore: MemoryStore started with
capacity 180.0 MB.
14/07/21 22:38:43 INFO network.ConnectionManager: Bound socket to port
57453 with id = ConnectionManagerId(hostname,57453)
14/07/21 22:38:43 INFO storage.BlockManagerMaster: Trying to register
BlockManager
14/07/21 22:38:43 INFO storage.BlockManagerInfo: Registering block manager
hostname:57453 with 180.0 MB RAM
14/07/21 22:38:43 INFO storage.BlockManagerMaster: Registered BlockManager
14/07/21 22:38:43 INFO spark.HttpServer: Starting HTTP Server
14/07/21 22:38:43 INFO server.Server: jetty-8.y.z-SNAPSHOT
14/07/21 22:38:43 INFO server.AbstractConnector: Started
SocketConnector@0.0.0.0:19323
14/07/21 22:38:43 INFO broadcast.HttpBroadcast: Broadcast server started
at http://9.123.99.10:19323
14/07/21 22:38:43 INFO spark.HttpFileServer: HTTP File server directory is
/tmp/spark-e224a31b-4517-43d8-9778-4b6af07dcad2
14/07/21 22:38:43 INFO spark.HttpServer: Starting HTTP Server
14/07/21 22:38:43 INFO server.Server: jetty-8.y.z-SNAPSHOT
14/07/21 22:38:43 INFO server.AbstractConnector: Started
SocketConnector@0.0.0.0:35420
14/07/21 22:38:43 INFO server.Server: jetty-8.y.z-SNAPSHOT
14/07/21 22:38:43 INFO server.AbstractConnector: Started
SelectChannelConnector@0.0.0.0:4040
14/07/21 22:38:43 INFO ui.SparkUI: Started SparkUI at http://hostname:4040
14/07/21 22:38:44 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/07/21 22:38:44 WARN spark.SparkContext: yarn-standalone is deprecated
as of Spark 1.0. Use yarn-cluster instead.
14/07/21 22:38:44 INFO cluster.YarnClusterScheduler: Created
YarnClusterScheduler
14/07/21 22:38:44 INFO yarn.ApplicationMaster$$anon$1: Adding shutdown
hook for context org.apache.spark.SparkContext@610c610c
Thanks!
Sam Liu
