Who can reply me about SPARK-36584

Hi All I recently raised an issue to the community about Spark's dynamic scaling. When there is a broadcast and dynamic scaling is enabled at the same time, the driver will send a BlockUpdate message when the broadcast is completed. ExecutorMonitor#OnBlockUpdate will receive this event and

Spark Structured Streaming Dyanamic Allocation

Hey folks： does Spark Structured Streaming have any plans for dynamic scaling? Currently Spark only has a dynamic scaling mechanism for batch jobs

Spark DStream Dynamic Allocation

1. First of all, I would like to ask whether the dynamic scaling of Spark DStream is available now? It is not mentioned in the Spark documentation 2. Spark DStream dynamic scaling will randomly kill a non-receiver executor when the average processing delay divided by the batch processing interval

Dynamic Allocation& ExecutorMonitor Shuffle Timeout & CacheTimeout

In private class Tracker of org.apache.spark.scheduler.dynalloc.ExecutorMonitor, the method ` updateTimeout ` will take the min of

ExecutorMoniitor timeout

In private class Tracker of org.apache.spark.scheduler.dynalloc.ExecutorMonitor, the method ` updateTimeout ` will take the min of