Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/11205
@vanzin , in the current code `stageIdToTaskIndices` cannot be used to
track number of running tasks, because this structure doesn't remove task index
from itself when task is finished successfully.
Yes `isExecutorIdle` is used to take care of executor idle, but the way to
identify whether executor is idle is not robust enough. In this scenario, when
stage is aborted because of max task failures, some task end event will be
missing, so using number of tasks per executor will lead to residual data, and
makes executor always be busy.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
