Hi, I am currently working on deploying Spark on kuberentes (K8s) and it is working fine. I am running Spark with standalone mode and checkpointing the state to shared system. So if master fails K8s starts it and from checkpoint it recover the earlier state and things just works fine. I have an issue with the Spark master Web UI to access the worker and application UI links. In brief, kubernetes service model allows me to expose the master service to internet, but accessing the application/workers UI is not possible as then I have to expose them too individually and given I can have multiple application it becomes hard to manage.
One solution can be that the master can act as reverse proxy to access information/state/logs from application/workers. As it has the information about their endpoint when application/worker register with master, so when a user initiate a request to access the information, master can proxy the request to corresponding endpoint. So I am wondering if someone has already done work in this direction then it would be great to know. If not then would the community will be interesting in such feature. If yes then how and where I should get started as it would be helpful for me to have some guidance to start working on this. Kind Regards, Gurvinder --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
