This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 10427c3b2ca7 [SPARK-57650][YARN][TEST] Allow AMs to use the whole
queue in BaseYarnClusterSuite to fix ACCEPTED-state timeouts
10427c3b2ca7 is described below
commit 10427c3b2ca704dc01b4a246cb8253986ff2e68f
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Thu Jun 25 05:09:33 2026 +0900
[SPARK-57650][YARN][TEST] Allow AMs to use the whole queue in
BaseYarnClusterSuite to fix ACCEPTED-state timeouts
### What changes were proposed in this pull request?
`BaseYarnClusterSuite` configures a mini `CapacityScheduler` but never sets
`yarn.scheduler.capacity.maximum-am-resource-percent`, so it defaults to `0.1`.
On memory-constrained CI runners the queue's total AM resource budget becomes
~1GB, which is smaller than the 1–2GB AM/driver memory these tests request.
Applications then wedge in the `ACCEPTED` state (never activated) and the suite
times out after 3 minutes with `handle.getState().isFinal() was false`.
This sets `maximum-am-resource-percent` to `1.0` (global + `root.default`)
so AMs can use the whole test queue and applications are always activated.
### Why are the changes needed?
`YarnClusterSuite` fails 6 tests with a 3-minute `eventually` timeout on
the scheduled Maven builds (`resource-managers#yarn` module):
- run Spark in yarn-client/cluster mode with different configurations,
ensuring redaction
- yarn-cluster should respect conf overrides in SparkHadoopUtil
(SPARK-16414, SPARK-23630)
- SPARK-35672: additional jar using URI scheme 'local' (client, cluster,
client + gateway-replacement)
The YARN diagnostics show `Queue's AM resource limit exceeded. AM Resource
Request = <memory:2048>; Queue Resource Limit for AM = <memory:1024>` repeated
>1000 times.
**Failing job (before):**
https://github.com/apache/spark/actions/runs/28045133937/job/83029837948 —
`Build / Maven (branch-4.2, Scala 2.13, JDK 21)`, `resource-managers#yarn` (6
failures).
**Passing job (with this fix):**
https://github.com/HyukjinKwon/spark/actions/runs/28066027338/job/83090387029 —
`resource-managers/yarn` tests: `YarnClusterSuite` 30/30 pass, the 6
formerly-failing tests now complete in ~11s each (was 180s timeout).
### Does this PR introduce any user-facing change?
No. Test-only.
### How was this patch tested?
Ran the `resource-managers/yarn` module tests on a fork (link above);
`YarnClusterSuite` passes 30/30.
This pull request and its description were written by Isaac.
Closes #56715 from HyukjinKwon/ci-fix/yarn-cluster-am-resource-percent.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala | 7 +++++++
1 file changed, 7 insertions(+)
diff --git
a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
index 1f755ba5efee..69c515d6a381 100644
---
a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
+++
b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
@@ -103,6 +103,13 @@ abstract class BaseYarnClusterSuite extends SparkFunSuite
with Matchers {
yarnConf.set("yarn.scheduler.capacity.root.default.acl_submit_applications",
"*")
yarnConf.set("yarn.scheduler.capacity.root.default.acl_administer_queue",
"*")
yarnConf.setInt("yarn.scheduler.capacity.node-locality-delay", -1)
+ // `maximum-am-resource-percent` defaults to 0.1, which caps the queue's
total AM resource
+ // usage to 10% of its capacity. On memory-constrained CI runners this
becomes ~1GB, smaller
+ // than the AM/driver memory these tests request (1-2GB), so applications
get stuck in the
+ // ACCEPTED state (never activated) and the suite times out waiting for a
final state. Let
+ // AMs use the whole queue in tests so applications are always activated.
+ yarnConf.setFloat("yarn.scheduler.capacity.maximum-am-resource-percent",
1.0f)
+
yarnConf.setFloat("yarn.scheduler.capacity.root.default.maximum-am-resource-percent",
1.0f)
// Support both IPv4 and IPv6
yarnConf.set("yarn.resourcemanager.hostname", Utils.localHostNameForURI())
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
