Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/18874
I'm saying you have a stage running that has > 0 tasks to run. If dynamic
allocation has already got all the executors it originally thought it needed
and they all idle timeout then you have 0 executors, then you can't run the
tasks anywhere. Deadlock. We don't try to reacquire executors until the next
stage. This patch does address that bug by not allowing you to go to 0
executors. it keeps enough executors to run all the tasks in parallel.
the executors idle timeout is based on a time. If there are other delays
in the system that cause the scheduler to not schedule tasks fast enough the
executors will idle timeout. This could be delays in network, could be because
driver was Gcing, etc. if the scheduler doesn't put a task on that executor
fast enough we can idle timeout it and never get a replacement back.
The problem with locality is that you don't know unless you give the
scheduler time to do its logic. it starts by scheduling things node local and
eventually falls back to rack and then any. If you don't allow the executors
to stay long enough for it to fall back then it can't schedule them there and
by the nature things will go slower because it can't run the tasks in parallel.
NO this is not a problem the user should be solving by increasing the idle
timeout. You want things to timeout fairly quickly when there are no tasks
that could be run on that. Changing the timeout will affect that badly. and NO
I don't necessarily want to decrease the locality wait as it could again affect
job performance. This again is going to be very job dependent as well as what
executors I get in this particular instance of the application. A user should
not have to work around this issue by changing the configurations. The fact is
we give back executors and never get more when the spark job could be using
them.
It matters if its removed because it would be used shortly and we never get
another executor back and thus I can never run all my tasks in parallel.
This does not ignore the minimum, the minimum is used when there aren't
enough tasks to actually use all the executors.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
