Is there a way to dynamically modify value of 'maxOffsetsPerTrigger' while a Stateful Structured Streaming job is running?
We are thinking of auto-scaling our Spark cluster but if we don't modify the value of 'maxOffsetsPerTrigger' dynamically would adding more VMs to the cluster help? I don't think it would, would it? In other words, if I add 2 new VMs to the cluster but value of 'maxOffsetsPerTrigger' is still the same would performance improve? I would think not. We would have to explicitly stop the job, add VMs & then restart the job after changing the value of 'maxOffsetsPerTrigger' - which defeats the purpose of Auto-scaling. Please tell me if my understanding is not correct. Thanks.
