Re: [Spark in Kubernetes] Question about running in client mode
Hi Attila, Ah that makes sense. Thanks for the clarification! Best, Shiqi On Mon, Apr 26, 2021 at 8:09 PM Attila Zsolt Piros < piros.attila.zs...@gmail.com> wrote: > Hi Shiqi, > > In case of client mode the driver runs locally: in the same machine, even > in the same process, of the spark submit. > > So if the application was submitted in a running POD then the driver will > be running in a POD and when outside of K8s then it will be running > outside. > This is why there is no config mentioned for this. > > From the deploy mode in general you can read here: > https://spark.apache.org/docs/latest/submitting-applications.html > > Best Regards, > Attila > > On Tue, Apr 27, 2021 at 12:03 AM Shiqi Sun wrote: > >> Hi Spark User group, >> >> I have a couple of quick questions about running Spark in Kubernetes >> between different deploy modes. >> >> As specified in >> https://spark.apache.org/docs/latest/running-on-kubernetes.html#client-mode, >> since Spark 2.4, client mode support is available when running in >> Kubernetes, and it says "when your application runs in client mode, the >> driver can run inside a pod or on a physical host". Then here come the >> questions. >> >> 1. If I understand correctly, in cluster mode, the driver is also running >> inside a k8s pod. Then, what's the difference between running it in cluster >> mode, versus running it in client mode when I choose to run my driver in a >> pod? >> >> 2. What does it mean by "running driver on a physical host"? Does it mean >> that it runs outside of the k8s cluster? What config should I pass to spark >> submit so that it runs this way, instead of running my driver into a k8s >> pod? >> >> Thanks! >> >> Best, >> Shiqi >> >
Re: [Spark in Kubernetes] Question about running in client mode
Hi Shiqi, In case of client mode the driver runs locally: in the same machine, even in the same process, of the spark submit. So if the application was submitted in a running POD then the driver will be running in a POD and when outside of K8s then it will be running outside. This is why there is no config mentioned for this. >From the deploy mode in general you can read here: https://spark.apache.org/docs/latest/submitting-applications.html Best Regards, Attila On Tue, Apr 27, 2021 at 12:03 AM Shiqi Sun wrote: > Hi Spark User group, > > I have a couple of quick questions about running Spark in Kubernetes > between different deploy modes. > > As specified in > https://spark.apache.org/docs/latest/running-on-kubernetes.html#client-mode, > since Spark 2.4, client mode support is available when running in > Kubernetes, and it says "when your application runs in client mode, the > driver can run inside a pod or on a physical host". Then here come the > questions. > > 1. If I understand correctly, in cluster mode, the driver is also running > inside a k8s pod. Then, what's the difference between running it in cluster > mode, versus running it in client mode when I choose to run my driver in a > pod? > > 2. What does it mean by "running driver on a physical host"? Does it mean > that it runs outside of the k8s cluster? What config should I pass to spark > submit so that it runs this way, instead of running my driver into a k8s > pod? > > Thanks! > > Best, > Shiqi >
[Spark in Kubernetes] Question about running in client mode
Hi Spark User group, I have a couple of quick questions about running Spark in Kubernetes between different deploy modes. As specified in https://spark.apache.org/docs/latest/running-on-kubernetes.html#client-mode, since Spark 2.4, client mode support is available when running in Kubernetes, and it says "when your application runs in client mode, the driver can run inside a pod or on a physical host". Then here come the questions. 1. If I understand correctly, in cluster mode, the driver is also running inside a k8s pod. Then, what's the difference between running it in cluster mode, versus running it in client mode when I choose to run my driver in a pod? 2. What does it mean by "running driver on a physical host"? Does it mean that it runs outside of the k8s cluster? What config should I pass to spark submit so that it runs this way, instead of running my driver into a k8s pod? Thanks! Best, Shiqi
