Hi Andrew, Thanks for the quick reply. It works with the yarn-client mode.
One question about the yarn-cluster mode: actually I was checking the AM for the log, since the spark driver is running in the AM, the UI should also work, right? But that is not true in my case. Best, Fang, Yan yanfang...@gmail.com +1 (206) 849-4108 On Mon, Jul 7, 2014 at 11:42 AM, Andrew Or <and...@databricks.com> wrote: > I will assume that you are running in yarn-cluster mode. Because the > driver is launched in one of the containers, it doesn't make sense to > expose port 4040 for the node that contains the container. (Imagine if > multiple driver containers are launched on the same node. This will cause a > port collision). If you're launching Spark from a gateway node that is > physically near your worker nodes, then you can just launch your > application in yarn-client mode, in which case the SparkUI will always be > started on port 4040 on the node that you ran spark-submit on. The reason > why sometimes you see the red text is because it appears only on the driver > containers, not the executor containers. This is because SparkUI belongs to > the SparkContext, which only exists on the driver. > > Andrew > > > 2014-07-07 11:20 GMT-07:00 Yan Fang <yanfang...@gmail.com>: > > Hi guys, >> >> Not sure if you have similar issues. Did not find relevant tickets in >> JIRA. When I deploy the Spark Streaming to YARN, I have following two >> issues: >> >> 1. The UI port is random. It is not default 4040. I have to look at the >> container's log to check the UI port. Is this suppose to be this way? >> >> 2. Most of the time, the UI does not work. The difference between logs >> are (I ran the same program): >> >> >> >> >> >> >> *14/07/03 11:38:50 INFO spark.HttpServer: Starting HTTP Server14/07/03 >> 11:38:50 INFO server.Server: jetty-8.y.z-SNAPSHOT 14/07/03 11:38:50 INFO >> server.AbstractConnector: Started SocketConnector@0.0.0.0:12026 >> <http://SocketConnector@0.0.0.0:12026>14/07/03 11:38:51 INFO >> executor.CoarseGrainedExecutorBackend: Got assigned task 0 14/07/03 >> 11:38:51 INFO executor.Executor: Running task ID 0...* >> >> 14/07/02 16:55:32 INFO spark.HttpServer: Starting HTTP Server >> 14/07/02 16:55:32 INFO server.Server: jetty-8.y.z-SNAPSHOT >> 14/07/02 16:55:32 INFO server.AbstractConnector: Started >> SocketConnector@0.0.0.0:14211 >> >> >> >> >> *14/07/02 16:55:32 INFO ui.JettyUtils: Adding filter: >> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter14/07/02 16:55:32 >> INFO server.Server: jetty-8.y.z-SNAPSHOT14/07/02 16:55:32 INFO >> server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:21867 >> <http://SelectChannelConnector@0.0.0.0:21867> 14/07/02 16:55:32 INFO >> ui.SparkUI: Started SparkUI at http://myNodeName:21867 >> <http://myNodeName:21867>14/07/02 16:55:32 INFO >> cluster.YarnClusterScheduler: Created YarnClusterScheduler* >> >> When the red part comes, the UI works sometime. Any ideas? Thank you. >> >> Best, >> >> Fang, Yan >> yanfang...@gmail.com >> +1 (206) 849-4108 >> > >
