This will also simplify Mesos users as well, DCOS has to work around this with our own proxying.
Tim On Sun, May 22, 2016 at 11:53 PM, Gurvinder Singh <gurvinder.si...@uninett.no> wrote: > Hi Reynold, > > So if that's OK with you, can I go ahead and create JIRA for this. As it > seems this feature is missing currently and can benefit not just for > kubernetes users but in general Spark standalone mode users too. > > - Gurvinder > On 05/22/2016 12:49 PM, Gurvinder Singh wrote: >> On 05/22/2016 10:23 AM, Sun Rui wrote: >>> If it is possible to rewrite URL in outbound responses in Knox or other >>> reverse proxy, would that solve your issue? >> Any process which can keep track of workers and application drivers IP >> addresses and route traffic to those will work. Considering Spark Master >> does exactly this due to all workers and application has to register to >> the master, therefore I propose master to be the place to add such a >> functionality. >> >> I am not aware with Knox capabilities but Nginx or any other normal >> reverse proxy will not be able to this on its own due to dynamic nature >> of application drivers and to some extent workers too. >> >> - Gurvinder >>>> On May 22, 2016, at 14:55, Gurvinder Singh <gurvinder.si...@uninett.no> >>>> wrote: >>>> >>>> On 05/22/2016 08:32 AM, Reynold Xin wrote: >>>>> Kubernetes itself already has facilities for http proxy, doesn't it? >>>>> >>>> Yeah kubernetes has ingress controller which can act the L7 load >>>> balancer and router traffic to Spark UI in this case. But I am referring >>>> to link present in UI to worker and application UI. Replied in the >>>> detail to Sun Rui's mail where I gave example of possible scenario. >>>> >>>> - Gurvinder >>>>> >>>>> On Sat, May 21, 2016 at 9:30 AM, Gurvinder Singh >>>>> <gurvinder.si...@uninett.no <mailto:gurvinder.si...@uninett.no>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> I am currently working on deploying Spark on kuberentes (K8s) and it is >>>>> working fine. I am running Spark with standalone mode and checkpointing >>>>> the state to shared system. So if master fails K8s starts it and from >>>>> checkpoint it recover the earlier state and things just works fine. I >>>>> have an issue with the Spark master Web UI to access the worker and >>>>> application UI links. In brief, kubernetes service model allows me to >>>>> expose the master service to internet, but accessing the >>>>> application/workers UI is not possible as then I have to expose them >>>>> too >>>>> individually and given I can have multiple application it becomes hard >>>>> to manage. >>>>> >>>>> One solution can be that the master can act as reverse proxy to access >>>>> information/state/logs from application/workers. As it has the >>>>> information about their endpoint when application/worker register with >>>>> master, so when a user initiate a request to access the information, >>>>> master can proxy the request to corresponding endpoint. >>>>> >>>>> So I am wondering if someone has already done work in this direction >>>>> then it would be great to know. If not then would the community will be >>>>> interesting in such feature. If yes then how and where I should get >>>>> started as it would be helpful for me to have some guidance to start >>>>> working on this. >>>>> >>>>> Kind Regards, >>>>> Gurvinder >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >>>>> <mailto:dev-unsubscr...@spark.apache.org> >>>>> For additional commands, e-mail: dev-h...@spark.apache.org >>>>> <mailto:dev-h...@spark.apache.org> >>>>> >>>>> >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >>>> For additional commands, e-mail: dev-h...@spark.apache.org >>>> >>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >>> For additional commands, e-mail: dev-h...@spark.apache.org >>> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> For additional commands, e-mail: dev-h...@spark.apache.org >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
