bmarcott commented on issue #26696: [WIP][SPARK-18886][CORE] Make locality wait time be the time since a TSM's available slots were fully utilized URL: https://github.com/apache/spark/pull/26696#issuecomment-568617329 [Here is an approach](https://github.com/apache/spark/compare/master...bmarcott:nmarcott-fulfill-slots-2?expand=1) which avoids trying to simulate any type of scheduling logic. This change resets locality timers only when no resources have been rejected due to delay scheduling. The complication in this change is that resetting the locality timer is dependent on when `TaskSchedulerImpl.resourceOffers` is called with all resources, rather than a single resource. That is guaranteed to happen only as frequent as `spark.scheduler.revive.interval` which defaults to 1s. This means there would be problems if `locality.wait.time` is smaller than the revive interval. Ways around this would be to either make the revive interval at least as small as locality wait or to always offer all free resources on calls to `TaskSchedulerImpl.resourceOffers`. We could also restrict how low you could make locality wait timeout... Thoughts or know of any other issues with this approach?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
