Re: Dynamic Scaling without Kubernetes

I am a bit late on this. K8s like GKE do not use YARN. We now have the option of auto scaling by say Google Dataproc . These benefit from autoscaling by enabling autoscaling policy.

Re: Dynamic Scaling without Kubernetes

Wouldn't you need to run Spark on Hadoop in order to use YARN?  I believe that YARN only manages Hadoop nodes, not Spark workers directly.  Besides, what I read was that you would need some extra plug-ins to be able to get nodes managed dynamically. Our use case would be like this: 1. A

Re: Dynamic Scaling without Kubernetes

So Spark can dynamically scale on YARN, but standalone mode becomes a bit complicated — where do you envision Spark gets the extra resources from? On Wed, Oct 26, 2022 at 12:18 PM Artemis User wrote: > Has anyone tried to make a Spark cluster dynamically scalable, i.e., > adding a new worker

Dynamic Scaling without Kubernetes

Has anyone tried to make a Spark cluster dynamically scalable, i.e., adding a new worker node automatically to the cluster when no more executors are available upon a new job submitted?  We need to make the whole cluster on-prem and really lightweight, so standalone mode is preferred and no