Hi Gaurav and Arun, Your settings seem reasonable; as long as YARN_CONF_DIR or HADOOP_CONF_DIR is properly set, the application should be able to find the correct RM port. Have you tried running the examples in yarn-client mode, and your custom application in yarn-standalone (now yarn-cluster) mode?
2014-05-20 5:17 GMT-07:00 gaurav.dasgupta <gaurav.d...@gmail.com>: > Few more details I would like to provide (Sorry as I should have provided > with the previous post): > > *- Spark Version = 0.9.1 (using pre-built spark-0.9.1-bin-hadoop2) > - Hadoop Version = 2.4.0 (Hortonworks) > - I am trying to execute a Spark Streaming program* > > Because I am using Hortornworks Hadoop (HDP), YARN is configured with > different port numbers than the default Apache's default configurations. > For > example, *resourcemanager.address* is <IP>:8050 in HDP whereas it defaults > to <IP>:8032. > > When I run the Spark examples using bin/run-example, I can see in the > console logs, that it is connecting to the right port configured by HDP, > i.e., 8050. Please refer the below console log: > > */[root@host spark-0.9.1-bin-hadoop2]# SPARK_YARN_MODE=true > > SPARK_JAR=assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop2.2.0.jar > > SPARK_YARN_APP_JAR=examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.1.jar > bin/run-example org.apache.spark.examples.HdfsTest yarn-client > /user/root/test > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > > [jar:file:/usr/local/spark-0.9.1-bin-hadoop2/examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > > [jar:file:/usr/local/spark-0.9.1-bin-hadoop2/assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop2.2.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > 14/05/20 06:55:29 INFO slf4j.Slf4jLogger: Slf4jLogger started > 14/05/20 06:55:29 INFO Remoting: Starting remoting > 14/05/20 06:55:29 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://spark@<IP:60988] > 14/05/20 06:55:29 INFO Remoting: Remoting now listens on addresses: > [akka.tcp://spark@<IP>:60988] > 14/05/20 06:55:29 INFO spark.SparkEnv: Registering BlockManagerMaster > 14/05/20 06:55:29 INFO storage.DiskBlockManager: Created local directory at > /tmp/spark-local-20140520065529-924f > 14/05/20 06:55:29 INFO storage.MemoryStore: MemoryStore started with > capacity 4.2 GB. > 14/05/20 06:55:29 INFO network.ConnectionManager: Bound socket to port > 35359 > with id = ConnectionManagerId(<IP>,35359) > 14/05/20 06:55:29 INFO storage.BlockManagerMaster: Trying to register > BlockManager > 14/05/20 06:55:29 INFO storage.BlockManagerMasterActor$BlockManagerInfo: > Registering block manager <IP>:35359 with 4.2 GB RAM > 14/05/20 06:55:29 INFO storage.BlockManagerMaster: Registered BlockManager > 14/05/20 06:55:29 INFO spark.HttpServer: Starting HTTP Server > 14/05/20 06:55:29 INFO server.Server: jetty-7.x.y-SNAPSHOT > 14/05/20 06:55:29 INFO server.AbstractConnector: Started > SocketConnector@0.0.0.0:59418 > 14/05/20 06:55:29 INFO broadcast.HttpBroadcast: Broadcast server started at > http://<IP>:59418 > 14/05/20 06:55:29 INFO spark.SparkEnv: Registering MapOutputTracker > 14/05/20 06:55:29 INFO spark.HttpFileServer: HTTP File server directory is > /tmp/spark-fc34fdc8-d940-420b-b184-fc7a8a65501a > 14/05/20 06:55:29 INFO spark.HttpServer: Starting HTTP Server > 14/05/20 06:55:29 INFO server.Server: jetty-7.x.y-SNAPSHOT > 14/05/20 06:55:29 INFO server.AbstractConnector: Started > SocketConnector@0.0.0.0:53425 > 14/05/20 06:55:29 INFO server.Server: jetty-7.x.y-SNAPSHOT > 14/05/20 06:55:29 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/storage/rdd,null} > 14/05/20 06:55:29 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/storage,null} > 14/05/20 06:55:29 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/stages/stage,null} > 14/05/20 06:55:29 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/stages/pool,null} > 14/05/20 06:55:29 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/stages,null} > 14/05/20 06:55:29 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/environment,null} > 14/05/20 06:55:29 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/executors,null} > 14/05/20 06:55:29 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/metrics/json,null} > 14/05/20 06:55:29 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/static,null} > 14/05/20 06:55:29 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/,null} > 14/05/20 06:55:29 INFO server.AbstractConnector: Started > SelectChannelConnector@0.0.0.0:4040 > 14/05/20 06:55:29 INFO ui.SparkUI: Started Spark Web UI at http:// > <IP>:4040 > 14/05/20 06:55:29 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 14/05/20 06:55:29 INFO spark.SparkContext: Added JAR > > /usr/local/spark-0.9.1-bin-hadoop2/examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.1.jar > at http://<IP>:53425/jars/spark-examples_2.10-assembly-0.9.1.jar with > timestamp 1400586929921 > 14/05/20 06:55:30 INFO client.RMProxy: Connecting to ResourceManager at > <IP>:8050 > 14/05/20 06:55:30 INFO yarn.Client: Got Cluster metric info from > ApplicationsManager (ASM), number of NodeManagers: 9 > 14/05/20 06:55:30 INFO yarn.Client: Queue info ... queueName: default, > queueCurrentCapacity: 0.0, queueMaxCapacity: 1.0,/* > > But, when I running my own custom spark streaming code, it is trying to > connect to port number 8032 instead and hence unable to connect. Refer the > below log: > > */[root@host spark-0.9.1-bin-hadoop2]# SPARK_YARN_MODE=true > > SPARK_JAR=assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop2.2.0.jar > SPARK_YARN_APP_JAR=/home/gaurav/SparkStreamExample.jar java -cp > > /home/gaurav/SparkStreamExample.jar:assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop2.2.0.jar > SparkStreamExample yarn-client <IP> 9999 > log4j:WARN No appenders could be found for logger > (akka.event.slf4j.Slf4jLogger). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for > more info. > 14/05/20 07:04:38 INFO SparkEnv: Using Spark's default log4j profile: > org/apache/spark/log4j-defaults.properties > 14/05/20 07:04:38 INFO SparkEnv: Registering BlockManagerMaster > 14/05/20 07:04:38 INFO DiskBlockManager: Created local directory at > /tmp/spark-local-20140520070438-5eae > 14/05/20 07:04:38 INFO MemoryStore: MemoryStore started with capacity 4.2 > GB. > 14/05/20 07:04:38 INFO ConnectionManager: Bound socket to port 49869 with > id > = ConnectionManagerId(<IP>,49869) > 14/05/20 07:04:38 INFO BlockManagerMaster: Trying to register BlockManager > 14/05/20 07:04:38 INFO BlockManagerMasterActor$BlockManagerInfo: > Registering > block manager <IP>:49869 with 4.2 GB RAM > 14/05/20 07:04:38 INFO BlockManagerMaster: Registered BlockManager > 14/05/20 07:04:38 INFO HttpServer: Starting HTTP Server > 14/05/20 07:04:38 INFO HttpBroadcast: Broadcast server started at > http://<IP>:36946 > 14/05/20 07:04:38 INFO SparkEnv: Registering MapOutputTracker > 14/05/20 07:04:38 INFO HttpFileServer: HTTP File server directory is > /tmp/spark-414ba274-adc0-4a0e-b1a4-9c1f048cbf37 > 14/05/20 07:04:38 INFO HttpServer: Starting HTTP Server > 14/05/20 07:04:38 INFO SparkUI: Started Spark Web UI at http://<IP>:4040 > 14/05/20 07:04:38 WARN NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 14/05/20 07:04:38 INFO SparkContext: Added JAR > /home/gaurav/SparkStreamExample.jar at > http://<IP>:40053/jars/SparkStreamExample.jar with timestamp 1400587478500 > 14/05/20 07:04:38 INFO RMProxy: Connecting to ResourceManager at > /0.0.0.0:8032 > 14/05/20 07:04:39 INFO Client: Retrying connect to server: > 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) > 14/05/20 07:04:40 INFO Client: Retrying connect to server: > 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) > 14/05/20 07:04:41 INFO Client: Retrying connect to server: > 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) > 14/05/20 07:04:42 INFO Client: Retrying connect to server: > 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)/* > > Do I need to specify the YARN ports configured by HDP to Spark somehow? How > the example jobs can detect the correct YARN ports? > > Thanks in advance. > > -- Gaurav > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Yarn-configuration-file-doesn-t-work-when-run-with-yarn-client-mode-tp1418p6097.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >
