Hi Team, I have been working on Spark Structured Streaming and trying to autoscale our application through dynamic allocation. But I couldn't find any documentation or configurations that supports dynamic scaling in Spark Structured Streaming, due to which I had been using Spark Batch mode dynamic scaling which is not so efficient with streaming use case.
I also tried with Spark streaming dynamic allocation configurations which didn't work with structured streaming. Below are the configurations I tried for dynamic scaling of my Spark Structured Streaming Application: With Batch Spark configurations: spark.dynamicAllocation.enabled: true spark.dynamicAllocation.executorAllocationRatio: 0.5 spark.dynamicAllocation.minExecutors: 1 spark.dynamicAllocation.maxExecutors: 5 With Streaming Spark configurations: spark.dynamicAllocation.enabled: false spark.streaming.dynamicAllocation.enabled: true spark.streaming.dynamicAllocation.scaleUpRatio: 0.7 spark.streaming.dynamicAllocation.scaleDownRatio: 0.2 spark.streaming.dynamicAllocation.minExecutors: 1 spark.streaming.dynamicAllocation.maxExecutors: 5 Kindly let me know if there is any configuration for the dynamic allocation of Spark Structured Streaming which I'm missing due to which autoscaling of my application is not working properly. Awaiting your response. Thanks and Regards, Aishwarya
