Re: HPA - Kubernetes for Spark
Hi , Yes I know by setting shuffle tracking property enabled we can use DRA. But , it is marked as experimental. Is it advised to use ? Also , regarding HPA. We do not have HPA differently as such for Spark. Right? Kind Regards, Sachit Murarka On Mon, Jan 11, 2021 at 2:17 AM Sandish Kumar HN wrote: > Sachit, > > K8S based spark dynamic allocation is only available on Spark 3.0.X+ and > that too without External Shuffling Service. > > > https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#dynamic-allocation > http://spark.apache.org/docs/latest/running-on-kubernetes.html#future-work > > On Sun, 10 Jan 2021 at 13:23, Sachit Murarka > wrote: > >> Hi All, >> >> I have read about HPA Horizontal Pod Autoscaling(for pod scaling). >> >> I understand it can be achieved by setting the request and limit for >> resources in yaml: >> kubectl autoscale deploy/application-cpu --cpu-percent=95 --min=1 >> --max=10 // example command. >> >> But does Kubernetes actually work with Spark for HPA? Since 1 pod is used >> to launch 1 unique executor. Here in spark ideally pod can be scaled by >> dynamical allocation of executors(which in turn is a pod) instead of HPA. >> But Dynamic allocation is not supported as shuffle service is not there >> till Spark 3 release. >> >> Could any one suggest how can I proceed achieving pod scaling in Spark? >> >> Please note : I am using Kubernetes with Spark operator. >> >> >> Kind Regards, >> Sachit Murarka >> > > > -- > > Thanks, > Regards, > SandishKumar HN >
Re: HPA - Kubernetes for Spark
Sachit, K8S based spark dynamic allocation is only available on Spark 3.0.X+ and that too without External Shuffling Service. https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#dynamic-allocation http://spark.apache.org/docs/latest/running-on-kubernetes.html#future-work On Sun, 10 Jan 2021 at 13:23, Sachit Murarka wrote: > Hi All, > > I have read about HPA Horizontal Pod Autoscaling(for pod scaling). > > I understand it can be achieved by setting the request and limit for > resources in yaml: > kubectl autoscale deploy/application-cpu --cpu-percent=95 --min=1 > --max=10 // example command. > > But does Kubernetes actually work with Spark for HPA? Since 1 pod is used > to launch 1 unique executor. Here in spark ideally pod can be scaled by > dynamical allocation of executors(which in turn is a pod) instead of HPA. > But Dynamic allocation is not supported as shuffle service is not there > till Spark 3 release. > > Could any one suggest how can I proceed achieving pod scaling in Spark? > > Please note : I am using Kubernetes with Spark operator. > > > Kind Regards, > Sachit Murarka > -- Thanks, Regards, SandishKumar HN
HPA - Kubernetes for Spark
Hi All, I have read about HPA Horizontal Pod Autoscaling(for pod scaling). I understand it can be achieved by setting the request and limit for resources in yaml: kubectl autoscale deploy/application-cpu --cpu-percent=95 --min=1 --max=10 // example command. But does Kubernetes actually work with Spark for HPA? Since 1 pod is used to launch 1 unique executor. Here in spark ideally pod can be scaled by dynamical allocation of executors(which in turn is a pod) instead of HPA. But Dynamic allocation is not supported as shuffle service is not there till Spark 3 release. Could any one suggest how can I proceed achieving pod scaling in Spark? Please note : I am using Kubernetes with Spark operator. Kind Regards, Sachit Murarka