Re: Problems initializing SparkUI

sounds strange. What happens if you submit the job from the other node? Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw *

Re: Problems initializing SparkUI

Hi, the output of the host where the driver is running: ~$ jps 9895 DriverWrapper 24057 Jps 3531 Worker In that host, I gave too much memory for the driver and no executor could be place for that worker. 2016-08-01 18:06 GMT-03:00 Mich Talebzadeh : > OK > > Can you

Re: Problems initializing SparkUI

OK Can you on the hostname that driver program is running do jps and send the output please? HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw *

Re: Problems initializing SparkUI

Hi, thanks again for the answer. Looking a little bit closer into that, I found out that the DriverWrapper process was not running in the hostname the log reported. It is runnning, but in another host. Mistery. If I manually go to the host that has the DriverWrapper running in it, on port 4040,

Re: Problems initializing SparkUI

Fine. In that case which process is your driver program (from jps output)? Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw *

Re: Problems initializing SparkUI

Hi, MIch, thanks for replying. I'm deploying from the same instance from where I showed the logs and commands using --deploy-mode cluster. The SparkSubmit process only appears while the bin/spark-submit binary is active. When the application starts and the driver takes control, the SparkSubmit

Re: Problems initializing SparkUI

OK I can see the Worker (19286 Worker and the executor(6548 CoarseGrainedExecutorBackend) running on it Where is spark-submit? Did you submit your job from another node or used another method to run it? HTH Dr Mich Talebzadeh LinkedIn *

Re: Problems initializing SparkUI

I just recently tried again, the port 4040 is not used. And even if it were, I think the log would reflect that trying to use the following port (4041) as you mentioned. This is what the driver log says: 16/08/01 13:55:56 INFO Utils: Successfully started service 'SparkUI' on port 4040. 16/08/01

Re: Problems initializing SparkUI

Can you check if port 4040 is actually used? If it used the next available one would 4041. For example below Zeppelin uses it *netstat -plten|grep 4040*tcp0 0 :::4040 :::*LISTEN 1005 73372882 *10699*/java *ps aux|grep 10699* hduser 10699 0.1

Re: Problems initializing SparkUI

Hi, Thanks for the answers. @Jacek: To verify if the ui is up, I enter to all the worker nodes of my cluster and run netstat -nltp | grep 4040 with no result. The log of the driver tells me in which server and on which port should the spark ui be up, but it isn't. @Mich: I've tried to specify

Re: Problems initializing SparkUI

why chance it. Best to explicitly specify in spark-submit (or whatever) which port to listen to --conf "spark.ui.port=nnn" and see if it works HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: Problems initializing SparkUI

Hi, I'm curious about "For some reason, sometimes the SparkUI does not appear to be bound on port 4040 (or any other) but the application runs perfectly and finishes giving the expected answer." How do you check that web UI listens to the port 4040? Pozdrawiam, Jacek Laskowski