Not sure if this can help, but a quick search on stackoverflow return this <https://stackoverflow.com/questions/33504798/how-to-find-the-master-url-for-an-existing-spark-cluster> and this one <https://stackoverflow.com/questions/37420537/how-to-check-status-of-spark-applications-from-the-command-line> ...
Alonso Isidoro Roman [image: https://]about.me/alonso.isidoro.roman <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links> 2017-07-17 14:21 GMT+02:00 <marina.bru...@orange.com>: > Hello, > > > > > > In our project, we have a Spark cluster with 2 master and 4 workers and > Zookeeper decides which master is alive. > > We have a problem with our reverse proxy to display the Spark Web UI. The > RP redirect on a master with IP address configured in initial configuration > but if Zookeeper decides to change the master, our spark Web UI is not > accessible because the IP address of master changed. > > We want to find dynamically which master is elected every time. > > We search in Internet a solution to know with Zookeeper which master is > alive but we don’t find anything. It is possible with confd to search if > property changed but none property is saved in Zookeeper. In folder /spark > in Zookeeper, nothing is logged. > > Why Spark does not send property to Zookeeper to indicate which ip address > or hostname is elected ? In your class ZooKeeperLeaderElectionAgent.scala, > you logged which master is elected but perhaps it will be also a good > solution to send a property to Zookeeper to indicate host. > > > > We already asked to Zookeeper user mailing list and they said that: > > “This question may be better suited for the Spark mailing lists as > Zookeeper doesn't really "decide" which master is alive but rather provides > a mechanism for the application to make the correct decision.” > > > > So, we think that we are not alone with this type of problem but we can’t > find anything on Internet. > > > > Can you help us to solve this problem ? > > Regards, > > Marina > > _________________________________________________________________________________________________________________________ > > Ce message et ses pieces jointes peuvent contenir des informations > confidentielles ou privilegiees et ne doivent donc > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu > ce message par erreur, veuillez le signaler > a l'expediteur et le detruire ainsi que les pieces jointes. Les messages > electroniques etant susceptibles d'alteration, > Orange decline toute responsabilite si ce message a ete altere, deforme ou > falsifie. Merci. > > This message and its attachments may contain confidential or privileged > information that may be protected by law; > they should not be distributed, used or copied without authorisation. > If you have received this email in error, please notify the sender and delete > this message and its attachments. > As emails may be altered, Orange is not liable for messages that have been > modified, changed or falsified. > Thank you. > >
