As Andrew explained, the port is random rather than 4040, as the the spark driver is started in Application Master and the port is random selected.
But I have the similar UI issue. I am running Yarn Cluster mode against my local CDH5 cluster. The log states "14/07/07 11:59:29 INFO ui.SparkUI: Started SparkUI at http://10.0.0.63:58750 " but when you client the spark UI link (ApplicationMaster or http://10.0.0.63:58750), I will got a 404 with the redirect URI http://localhost/proxy/application_1404443455764_0010/ Looking at the Spark code, notice that the "proxy" is reallya variable to get the proxy at the yarn-site.xml http address. But when I specified the value at yarn-site.xml, it still doesn't work for me. Oddly enough, it works for my co-worker on Pivotal HD cluster, therefore I am still looking what's the difference in terms of cluster setup or something else. Chester On Mon, Jul 7, 2014 at 11:42 AM, Andrew Or <and...@databricks.com> wrote: > I will assume that you are running in yarn-cluster mode. Because the > driver is launched in one of the containers, it doesn't make sense to > expose port 4040 for the node that contains the container. (Imagine if > multiple driver containers are launched on the same node. This will cause a > port collision). If you're launching Spark from a gateway node that is > physically near your worker nodes, then you can just launch your > application in yarn-client mode, in which case the SparkUI will always be > started on port 4040 on the node that you ran spark-submit on. The reason > why sometimes you see the red text is because it appears only on the driver > containers, not the executor containers. This is because SparkUI belongs to > the SparkContext, which only exists on the driver. > > Andrew > > > 2014-07-07 11:20 GMT-07:00 Yan Fang <yanfang...@gmail.com>: > > Hi guys, >> >> Not sure if you have similar issues. Did not find relevant tickets in >> JIRA. When I deploy the Spark Streaming to YARN, I have following two >> issues: >> >> 1. The UI port is random. It is not default 4040. I have to look at the >> container's log to check the UI port. Is this suppose to be this way? >> >> 2. Most of the time, the UI does not work. The difference between logs >> are (I ran the same program): >> >> >> >> >> >> >> *14/07/03 11:38:50 INFO spark.HttpServer: Starting HTTP Server14/07/03 >> 11:38:50 INFO server.Server: jetty-8.y.z-SNAPSHOT 14/07/03 11:38:50 INFO >> server.AbstractConnector: Started SocketConnector@0.0.0.0:12026 >> <http://SocketConnector@0.0.0.0:12026>14/07/03 11:38:51 INFO >> executor.CoarseGrainedExecutorBackend: Got assigned task 0 14/07/03 >> 11:38:51 INFO executor.Executor: Running task ID 0...* >> >> 14/07/02 16:55:32 INFO spark.HttpServer: Starting HTTP Server >> 14/07/02 16:55:32 INFO server.Server: jetty-8.y.z-SNAPSHOT >> 14/07/02 16:55:32 INFO server.AbstractConnector: Started >> SocketConnector@0.0.0.0:14211 >> >> >> >> >> *14/07/02 16:55:32 INFO ui.JettyUtils: Adding filter: >> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter14/07/02 16:55:32 >> INFO server.Server: jetty-8.y.z-SNAPSHOT14/07/02 16:55:32 INFO >> server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:21867 >> <http://SelectChannelConnector@0.0.0.0:21867> 14/07/02 16:55:32 INFO >> ui.SparkUI: Started SparkUI at http://myNodeName:21867 >> <http://myNodeName:21867>14/07/02 16:55:32 INFO >> cluster.YarnClusterScheduler: Created YarnClusterScheduler* >> >> When the red part comes, the UI works sometime. Any ideas? Thank you. >> >> Best, >> >> Fang, Yan >> yanfang...@gmail.com >> +1 (206) 849-4108 >> > >
