Hi, I have some doubts about dynamic resource allocation with spark streaming.
If spark had allocated 5 executors for me, then he would dispatch every batch tasks on all of them equally. So if batchSize < spark.dynamicAllocation.executorIdleTimeout then spark will never free any executor. Moreover to free executor the processing time of batch must be lower than (batchSize - spark.dynamicAllocation.executorIdleTimeout), e.g. for batchSize = 30s spark.dynamicAllocation.executorIdleTimeout = 25s my batch should be processed under 5 seconds to free executors. If everything I have written above is true, it is not so great mechanism, because I would like to free an executor, for example, when processing time was lower than batchSize/2. I can set spark.dynamicAllocation.executorIdleTimeout to batchSize/2 but then spark will probably free all my executors... Have anyone worked out some sensible solution to this problem?
