Github user mridulm commented on the pull request: https://github.com/apache/spark/pull/1313#issuecomment-49200080 I just noticed that pendingTasksWithNotReadyPrefs is not being used now ? It is getting updated but never actually queried from ... Do we need to maintain it ? The way I initially thought about this problem was, 1) When a task has no preferred location by definition : schedule it on any node when there are no NODE_LOCAL tasks available for that executor. 2) When a task has preferred location defined, but none available right now, treat is as ANY task : so that other PROCESS/NODE/RACK local tasks have precedence over it. If/when a node/rack local host pops in, it becomes eligible for better schedule preference. @CodingCat, @kayousterhout @lirui-intel any thoughts ? I might be missing somethere here !
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---