agrawaldevesh commented on a change in pull request #29422:
URL: https://github.com/apache/spark/pull/29422#discussion_r471249082
##########
File path:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -136,7 +139,21 @@ private[spark] class TaskSchedulerImpl(
// IDs of the tasks running on each executor
private val executorIdToRunningTaskIds = new HashMap[String, HashSet[Long]]
- private val executorsPendingDecommission = new HashMap[String,
ExecutorDecommissionInfo]
+ // We add executors here when we first get decommission notification for
them. Executors can
+ // continue to run even after being asked to decommission, but they will
eventually exit.
+ val executorsPendingDecommission = new HashMap[String,
ExecutorDecommissionInfo]
+
+ // When they exit and we know of that via heartbeat failure, we will add
them to this cache.
+ // This cache is consulted to know if a fetch failure is because a source
executor was
+ // decommissioned.
+ lazy val decommissionedExecutorsRemoved = CacheBuilder.newBuilder()
+ .expireAfterWrite(
+ conf.getLong("spark.decommissioningRememberAfterRemoval.seconds", 60L),
TimeUnit.SECONDS)
Review comment:
Yeah the fetch failure is most likely going to arrive well after the
executor is removed because the reducer typically retries before giving up with
a fetch failure.
The idea here is to be best effort: to act on fetch failures stemming from
decommissioned mappers and purge out all of the other mappers on that host
asap. Ofcourse, it is possible that the fetch failure arrives too late and we
have forgotten that this was a decommissioned host.
We are not trying to promise 100% robustness with decommissioning, just
trying to improve it a bit.
So now one question please: what should be the value of this timeout. I am
thinking more like 5 minutes ? Obviously the smaller it is, the less the space
overhead. But I think 5 minutes should be fine. how much of a delay have you
seen in practice ? What is the delay in the RPC framework we are talking about ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
