Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18874
Why is 0 executors a 'deadlock'? if there is no work to do, 0 executors is
fine. If there is work to do, of course, at least 1 executor should not time
out. Is that what you're claiming happens? that would be a bug, but that's not
what this addresses though.
I don't understand the claim that a small number of executors is somehow
wrong. 0 is a fine number to have if you have no work.
What do delays have to do with timing out idle executors? they're idle
because no work is sent, not because they can't be reached.
Yes, you may have a situation where the scheduler prefers to wait to
schedule on executor A, even though B has slots (or no tasks at all) because of
locality. That is by design. You are claiming this is an error because it will
need it 'later' or something -- how would you even know that?
If B times out because nobody sends it work, that is fine. You're saying
let B wait a little longer: fine, increase the idle timeout. Or you're saying,
wait for A for a shorter time: fine, decrease the locality wait.
Or: if B is getting no work, how can it matter whether it's removed?
But this change just ignores the minimum, and I dont see how it's related
to any of these ideas.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
