Ryu Kobayashi created TEZ-4720:
----------------------------------
Summary: DagAwareYarnTaskScheduler.getAvailableResources() should
clamp negative resource values to 0
Key: TEZ-4720
URL: https://issues.apache.org/jira/browse/TEZ-4720
Project: Apache Tez
Issue Type: Bug
Reporter: Ryu Kobayashi
When YARN Resource Manager temporarily reports negative available resources,
DagAwareYarnTaskScheduler.getAvailableResources() returns the negative value
as-is. This negative value is then used to initialize totalResources in
getProgress() on the first heartbeat. Once set to a negative value,
totalResources.getMemory() != 0 so it is never updated again, causing all
subsequent resource calculations to be incorrect.
getAvailableResources() returns the raw value from
client.getAvailableResources() without any validation. There is no guard
against negative values, unlike Resource.castToIntSafely() in Hadoop which was
fixed in YARN-11964 to clamp negative values to 0.
The Hadoop-side root cause has been resolved in YARN-11964.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
