Re: [Spark K8s] Seeking Advice on Scaling Spark Cluster in Kubernetes

I do not know about AWS but in Google Kubernetes Cluster (GKE), you can autoscale as follows gcloud container clusters create example-cluster \ --num-nodes 2 \ --zone us-central1-a \ --node-locations us-central1-a,us-central1-b,us-central1-f \ --enable-autoscaling --min-nodes 1 --max-nodes

Re: [Spark K8s] Seeking Advice on Scaling Spark Cluster in Kubernetes

For Kubernetes yes -- we use AWS EKS. We manage the Spark cluster ourselves and it is deployed via Helm chart. On Wed, Feb 2, 2022 at 8:32 AM Mich Talebzadeh wrote: > Is this part of cloud service offerings like Google GKE? > > > >view my Linkedin profile >

Re: [Spark K8s] Seeking Advice on Scaling Spark Cluster in Kubernetes

Is this part of cloud service offerings like Google GKE? view my Linkedin profile *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise f

[Spark K8s] Seeking Advice on Scaling Spark Cluster in Kubernetes

Hello, Before I ask my questions, let me say what I am trying to do and briefly describe the setup I have so far. I am basically building an API service that serves a ML model which uses Spark ML. I have Spark deployed in Kubernetes in standalone mode (the default Spark manager) with 2 worker node