Hello everyone: I'm trying the dynamic allocation in Spark on YARN. I have followed configuration steps and started the shuffle service.
Now it can request executors when the workload is heavy but it cannot remove executors. I try to open the spark shell and don’t run any command, no executor is removed after spark.dynamicAllocation.executorIdleTimeout interval. Am I missing something? Thanks.
