Hi guys,

Not sure if you  have similar issues. Did not find relevant tickets in
JIRA. When I deploy the Spark Streaming to YARN, I have following two
issues:

1. The UI port is random. It is not default 4040. I have to look at the
container's log to check the UI port. Is this suppose to be this way?

2. Most of the time, the UI does not work. The difference between logs are
(I ran the same program):






*14/07/03 11:38:50 INFO spark.HttpServer: Starting HTTP Server14/07/03
11:38:50 INFO server.Server: jetty-8.y.z-SNAPSHOT 14/07/03 11:38:50 INFO
server.AbstractConnector: Started SocketConnector@0.0.0.0:12026
<http://SocketConnector@0.0.0.0:12026>14/07/03 11:38:51 INFO
executor.CoarseGrainedExecutorBackend: Got assigned task 0 14/07/03
11:38:51 INFO executor.Executor: Running task ID 0...*

14/07/02 16:55:32 INFO spark.HttpServer: Starting HTTP Server
14/07/02 16:55:32 INFO server.Server: jetty-8.y.z-SNAPSHOT
14/07/02 16:55:32 INFO server.AbstractConnector: Started
SocketConnector@0.0.0.0:14211




*14/07/02 16:55:32 INFO ui.JettyUtils: Adding filter:
org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter14/07/02 16:55:32
INFO server.Server: jetty-8.y.z-SNAPSHOT14/07/02 16:55:32 INFO
server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:21867
<http://SelectChannelConnector@0.0.0.0:21867> 14/07/02 16:55:32 INFO
ui.SparkUI: Started SparkUI at http://myNodeName:21867
<http://myNodeName:21867>14/07/02 16:55:32 INFO
cluster.YarnClusterScheduler: Created YarnClusterScheduler*

When the red part comes, the UI works sometime. Any ideas? Thank you.

Best,

Fang, Yan
yanfang...@gmail.com
+1 (206) 849-4108

Reply via email to