On 05/22/2016 08:30 AM, Sun Rui wrote: > I think “reverse proxy” is beneficial to monitoring a cluster in a > secure way. This feature is not only desired for Spark on standalone, > but also Spark on YARN, and also projects other than spark. I think to secure the Spark you can use any reverse proxy out there e.g Knox or light weight as nginx/node-http-proxy or pick your favorite language. There are even oauth2-proxy (https://github.com/bitly/oauth2_proxy) too which can secure for example Spark UI using github/google accounts.
But the issue here is that in the Spark master UI page has links to information about workers which points to their internal IP addresses, so you need to have either VPN or on the same network to get the worker information e.g logs, etc. Same goes for application UI as driver is inside the spark cluster network. So the idea is that the Spark master UI can act as a reverse proxy to get these information. for example Worker with ID worker1 running at IP address 10.2.3.4:8081 in current situation a user accessing the master UI and want to see information from worker1, user needs to either connect VPN or have 10.2.3.4 accessible from his/her machine. So the proposal is to have a functionality in spark master UI where to access the worker with ID worker1 the link will be like spark-master.com/worker1 when user access this link, master will proxy this to 10.2.3.4:8081 and back. So user does not need to be on the same network. This will really simplify the spark ui access in general case too where you will need to expose only one IP to the public. I have done preliminary study of the code and it seems Spark is using Jetty for it and Jetty has ProxyServlet which can serve this purpose. So would be good to know if community is interested in having such a feature and get together to add it then :) - Gurvinder > > Maybe Apache Knox can help you. Not sure how Knox can integrate with Spark. >> On May 22, 2016, at 00:30, Gurvinder Singh <gurvinder.si...@uninett.no >> <mailto:gurvinder.si...@uninett.no>> wrote: >> >> standalone mod > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
