Re: Unsubscribe

Please send an e-mail: user-unsubscr...@spark.apache.org to unsubscribe yourself from the mailing list. On Sun, Feb 6, 2022 at 2:21 PM Rishi Raj Tandon wrote: > Unsubscribe >

Unsubscribe

Unsubscribe

Unsubscribe

Unsubscribe

Re: spark, autoscaling and handling node loss with autoscaling

I did some tests on a three node Dataproc cluster with autoscaling on. One master node and 2 work nodes. the master node was called ctpcluster-m and the worker nodes were ctpcluster-w-0 and ctpcluster-w-1 respectively I started a spark-submit job with the following autoscaling parameters added

spark, autoscaling and handling node loss with autoscaling

This question arises when Spark is offered as a managed service on a cluster of VMs in Cloud. For example, Google Dataproc or Amazon EMR among others >From what I can see in autoscaling setup, you will always need a minimum of two