Yup in cluster mode you need to figure out what machine the driver is running on. That is the machine the UI will run on
https://issues.apache.org/jira/browse/SPARK-15829 You may also have fire wall issues Here are some notes I made about how to figure out what machine the driver is running on when using cluster mode Application UI Bug work around If our application is running in cluster mode the Master Console UI will incorrectly assume the Application UI¹s are running on the master machine. The UI is actually running on the same machine as the driver To find the URL to the apps running in cluster mode 1. find the worker id for the app you are interested in. * on the master console UI go to the Running App¹s section * you should see a column Worker¹ * a worker is something like worker-20160322041632-172.31.23.201-34909 2. got to the AWS ec2 console 3. from the private ip find the public DNS name * the private ip in our example is 172.31.23.201 From: Jacek Laskowski <ja...@japila.pl> Date: Tuesday, July 26, 2016 at 6:38 AM To: Jestin Ma <jestinwith.a...@gmail.com> Cc: Chanh Le <giaosu...@gmail.com>, "user @spark" <user@spark.apache.org> Subject: Re: Spark Web UI port 4040 not working > Hi, > > Do you perhaps deploy using cluster mode? Is this EC2? You'd need to > figure out where the driver runs and use the machine's IP. > > Pozdrawiam, > Jacek Laskowski > ---- > https://medium.com/@jaceklaskowski/ > Mastering Apache Spark http://bit.ly/mastering-apache-spark > Follow me at https://twitter.com/jaceklaskowski > > > On Tue, Jul 26, 2016 at 3:36 PM, Jestin Ma <jestinwith.a...@gmail.com> wrote: >> I tried doing that on my master node. >> I got nothing. >> However, I grep'd port 8080 and I got the standalone UI. >> >> On Tue, Jul 26, 2016 at 12:39 AM, Chanh Le <giaosu...@gmail.com> wrote: >>> >>> You¹re running in StandAlone Mode? >>> Usually inside active task it will show the address of current job. >>> or you can check in master node by using netstat -apn | grep 4040 >>> >>> >>> >>>> > On Jul 26, 2016, at 8:21 AM, Jestin Ma <jestinwith.a...@gmail.com> >>>> > wrote: >>>> > >>>> > Hello, when running spark jobs, I can access the master UI (port 8080 >>>> > one) no problem. However, I'm confused as to how to access the web UI to >>>> see >>>> > jobs/tasks/stages/etc. >>>> > >>>> > I can access the master UI at http://<master-node>:8080. But port 4040 >>>> > gives me a -connection cannot be reached-. >>>> > >>>> > Is the web UI http://<master-node> with a port of 4040? >>>> > >>>> > I'm running my Spark job on a cluster machine and submitting it to a >>>> > master node part of the cluster. I heard of ssh tunneling; is that >>>> relevant >>>> > here? >>>> > >>>> > Thank you! >>> >> > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
