Norio Akagi created SPARK-60914:
-----------------------------------
Summary: YarnAllocator never removes a recommissioned node from
decommissioningNodesCache, so a second decommissioning of the same node is
ignored
Key: SPARK-60914
URL: https://issues.apache.org/jira/browse/SPARK-60914
Project: Spark
Issue Type: Bug
Components: YARN
Affects Versions: 4.0.0, 4.1.0, 3.5.0, 3.4.0
Reporter: Norio Akagi
YarnAllocator keeps a \{{decommissioningNodesCache}} (added in SPARK-39018) so
that the driver is notified only once per node that enters the DECOMMISSIONING
state. Hosts are only ever added to the cache in
\{{handleNodesInDecommissioningState}}. Nothing removes them except LRU
eviction once the cache exceeds 200 entries.
When a node is recommissioned, the RM reports it to the AM in
\{{AllocateResponse.getUpdatedNodes}} with state RUNNING (YARN-10538, Hadoop
3.2.3 / 3.3.1 / 3.4.0+). YarnAllocator ignores every node state other than
DECOMMISSIONING, so the host stays in the cache after it is back to RUNNING.
If YARN later allocates new containers on that host and the host is
decommissioned again, the DECOMMISSIONING report is filtered out by the stale
cache entry. \{{DecommissionExecutorsOnHost}} is never sent to the driver, the
new executors keep running tasks, and when the node is finally shut down their
tasks and shuffle data are lost without the graceful decommission that
\{{spark.decommission.enabled}} is supposed to provide.
Sequence:
# host1 is reported as DECOMMISSIONING. YarnAllocator sends
DecommissionExecutorsOnHost(host1) and adds host1 to the cache.
# host1 is recommissioned. The RM reports host1 as RUNNING, YarnAllocator
ignores it. host1 remains in the cache.
# YARN allocates new containers on host1 and executors start there.
# host1 is reported as DECOMMISSIONING again. The report is dropped because
host1 is already in the cache. The executors from step 3 are never
decommissioned.
This is easy to hit on clusters where the same hostname goes through
decommission and recommission repeatedly, for example autoscaling setups that
cancel a scale-in.
Proposed fix: in \{{handleNodesInDecommissioningState}}, also look at RUNNING
node reports and remove the host from \{{decommissioningNodesCache}}. A unit
test that replays the sequence above against YarnAllocator fails on current
master at step 4 ("Wanted 2 times, but was 1 time") and passes with the fix.
References:
* Original implementation: SPARK-39018,
https://github.com/apache/spark/pull/35683
* RM side notification on recommission: YARN-10538
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
