I just recently tried again, the port 4040 is not used. And even if it were, I think the log would reflect that trying to use the following port (4041) as you mentioned.
This is what the driver log says: 16/08/01 13:55:56 INFO Utils: Successfully started service 'SparkUI' on port 4040. 16/08/01 13:55:56 INFO SparkUI: Started SparkUI at http://hostname:4040 If I go to {hostname}: ~$ jps 6548 CoarseGrainedExecutorBackend 19286 Worker 6843 Jps 19182 Master ~$ netstat -nltp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp6 0 0 192.168.22.245:43037 :::* LISTEN 6548/java tcp6 0 0 192.168.22.245:56929 :::* LISTEN 19286/java tcp6 0 0 192.168.22.245:7077 :::* LISTEN 19182/java tcp6 0 0 :::33296 :::* LISTEN 6548/java tcp6 0 0 :::8080 :::* LISTEN 19182/java tcp6 0 0 :::8081 :::* LISTEN 19286/java tcp6 0 0 192.168.22.245:6066 :::* LISTEN 19182/java ~$ netstat -nltap | grep 4040 I'm really lost here and don't know much about spark yet, but shouldn't there be a DriverWrapper process which holds the bind on port 4040? 2016-08-01 13:49 GMT-03:00 Mich Talebzadeh <mich.talebza...@gmail.com>: > Can you check if port 4040 is actually used? If it used the next available > one would 4041. For example below Zeppelin uses it > > > *netstat -plten|grep 4040*tcp 0 0 :::4040 > :::* LISTEN 1005 73372882 *10699*/java > *ps aux|grep 10699* > hduser 10699 0.1 3.8 3172308 952932 pts/3 SNl Jul30 5:57 > /usr/java/latest/bin/java -cp /data6/hduser/zeppelin-0.6.0/ ... > > HTH > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > On 1 August 2016 at 17:44, Maximiliano Patricio Méndez < > mmen...@despegar.com> wrote: > >> Hi, >> >> Thanks for the answers. >> >> @Jacek: To verify if the ui is up, I enter to all the worker nodes of my >> cluster and run netstat -nltp | grep 4040 with no result. The log of the >> driver tells me in which server and on which port should the spark ui be >> up, but it isn't. >> >> >> @Mich: I've tried to specify spark.ui.port=nnn but I only manage to >> change the log, reporting that the driver should be in another port. >> >> The ui has no problem to start in that port (4040) when I run my >> application in client mode. >> >> Could there be a network issue making the ui to fail silently? I've read >> some of the code regarding those parts of the driver log, but couldn't find >> anything weird. >> >> 2016-07-29 19:45 GMT-03:00 Mich Talebzadeh <mich.talebza...@gmail.com>: >> >>> why chance it. Best to explicitly specify in spark-submit (or whatever) >>> which port to listen to >>> >>> --conf "spark.ui.port=nnn" >>> >>> and see if it works >>> >>> HTH >>> >>> >>> Dr Mich Talebzadeh >>> >>> >>> >>> LinkedIn * >>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>> >>> >>> >>> http://talebzadehmich.wordpress.com >>> >>> >>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>> any loss, damage or destruction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damages >>> arising from such loss, damage or destruction. >>> >>> >>> >>> On 29 July 2016 at 23:37, Jacek Laskowski <ja...@japila.pl> wrote: >>> >>>> Hi, >>>> >>>> I'm curious about "For some reason, sometimes the SparkUI does not >>>> appear to be bound on port 4040 (or any other) but the application >>>> runs perfectly and finishes giving the expected answer." How do you >>>> check that web UI listens to the port 4040? >>>> >>>> Pozdrawiam, >>>> Jacek Laskowski >>>> ---- >>>> https://medium.com/@jaceklaskowski/ >>>> Mastering Apache Spark http://bit.ly/mastering-apache-spark >>>> Follow me at https://twitter.com/jaceklaskowski >>>> >>>> >>>> On Thu, Jul 28, 2016 at 11:37 PM, Maximiliano Patricio Méndez >>>> <mmen...@despegar.com> wrote: >>>> > Hi, >>>> > >>>> > I'm having some trouble trying to submit an application to my spark >>>> cluster. >>>> > For some reason, sometimes the SparkUI does not appear to be bound on >>>> port >>>> > 4040 (or any other) but the application runs perfectly and finishes >>>> giving >>>> > the expected answer. >>>> > >>>> > And don't know why, but if I restart all the workers at once >>>> sometimes it >>>> > begins to work and sometimes it doesn't. >>>> > >>>> > In the driver logs, when it fails to start the SparkUI I see some >>>> these >>>> > lines: >>>> > 16/07/28 16:13:37 INFO Utils: Successfully started service 'SparkUI' >>>> on port >>>> > 4040. >>>> > 16/07/28 16:13:37 INFO SparkUI: Started SparkUI at >>>> http://hostname-00:4040 >>>> > >>>> > but nothing running in those ports. >>>> > >>>> > I'm attaching the full driver log in which I've activated jetty logs >>>> on >>>> > DEBUG but couldn't find anything. >>>> > >>>> > The only properties that I'm not leaving at default at the >>>> configuration is >>>> > the SPARK_PUBLIC_DNS=$(hostname), SPARK_WORKER_CORES and >>>> SPARK_WORKER_MEMORY >>>> > >>>> > Have anyone faced something similar? >>>> > >>>> > Thanks >>>> > >>>> > >>>> > --------------------------------------------------------------------- >>>> > To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>>> >>>> >>> >> >
