This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 41f3e67e3ebe [SPARK-57710][YARN][TEST] Reduce YarnClusterSuite
flakiness from CI runner contention
41f3e67e3ebe is described below
commit 41f3e67e3ebe9ca654af98c6464d607c92ae97f3
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Jun 27 11:09:09 2026 +0900
[SPARK-57710][YARN][TEST] Reduce YarnClusterSuite flakiness from CI runner
contention
### What changes were proposed in this pull request?
Follow-up to
[SPARK-57650](https://issues.apache.org/jira/browse/SPARK-57650), which fixed
the deterministic "AM stuck in ACCEPTED" hang in `BaseYarnClusterSuite`. Two
further **test-only** changes to reduce the remaining flakiness of
`YarnClusterSuite`:
1. Give the single mini `NodeManager` 8GB
(`yarn.nodemanager.resource.memory-mb` +
`yarn.scheduler.maximum-allocation-mb`) so executor allocation is never starved
once the ~1.4GB AM is running.
2. Raise the executor→driver connection retry budget for the launched apps
(`spark.rpc.io.maxRetries=10`, `spark.rpc.io.retryWait=2s`) so a transient
RPC-accept stall does not permanently fail an executor. These are defaults that
individual tests can still override via `extraConf`.
### Why are the changes needed?
Even after SPARK-57650, the scheduled `Build / Java21` and `Build / Java25`
master lanes fail in the `yarn` module roughly **50% of runs** (e.g. fork run
`28151220075` PASS vs `28151247521` FAIL — same commit, 40s apart). All
failures are the same six `YarnClusterSuite` tests timing out after 3 minutes
(`The code passed to eventually never returned normally ...
handle.getState().isFinal() was false`).
From the `yarn-app-log` / `unit-tests-log` artifacts, the AM/driver comes
up, but the executor (and sometimes the AM) intermittently fail to connect back
to the driver's RPC server on `localhost` (`java.io.IOException: Failed to
connect to localhost/127.0.0.1:<port>`, connection refused). The in-JVM mini
RM+NM, the driver subprocess and the AM/executor JVMs all contend for CPU on a
single CI runner, so the driver's accept loop occasionally stalls; an executor
that loses this race exit [...]
### Does this PR introduce any user-facing change?
No. Test-only.
### How was this patch tested?
`YarnClusterSuite` was previously failing ~50% of the time. With this
change the `yarn` module job was run **6 times** on the fork; all 6 passed,
with `YarnClusterSuite` reporting `tests=30, failures=0, skipped=0` (the 6
formerly-failing tests now pass):
- Before (master, failing): apache/spark run `28148781009` (`Build /
Java21`) — 6 `YarnClusterSuite` timeouts.
- After (this branch): HyukjinKwon/spark runs `28162182834`, `28162247111`,
`28162249819`, `28175759262`, `28175762257`, `28175765871` — `yarn` job green
in all six.
### Was this patch authored or co-authored using generative AI tooling?
Yes, Generated-by: Claude Code.
Closes #56785 from HyukjinKwon/ci-fix/agent8-yarn-cluster-flaky.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../spark/deploy/yarn/BaseYarnClusterSuite.scala | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
diff --git
a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
index 69c515d6a381..5112e62d838c 100644
---
a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
+++
b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
@@ -111,6 +111,14 @@ abstract class BaseYarnClusterSuite extends SparkFunSuite
with Matchers {
yarnConf.setFloat("yarn.scheduler.capacity.maximum-am-resource-percent",
1.0f)
yarnConf.setFloat("yarn.scheduler.capacity.root.default.maximum-am-resource-percent",
1.0f)
+ // Give the single mini NodeManager generous memory. By default the mini
cluster advertises
+ // only a small amount of memory, so once the AM (~1.4GB) is running there
is barely enough
+ // headroom left for the executors these tests request. On a busy CI
runner that makes
+ // executor allocation slow/racy and the YarnClusterSuite apps time out
waiting to finish.
+ // The CI hosts have plenty of RAM, so let the NM offer enough for the AM
plus a few executors.
+ yarnConf.setInt("yarn.nodemanager.resource.memory-mb", 8192)
+ yarnConf.setInt("yarn.scheduler.maximum-allocation-mb", 8192)
+
// Support both IPv4 and IPv6
yarnConf.set("yarn.resourcemanager.hostname", Utils.localHostNameForURI())
@@ -292,6 +300,18 @@ abstract class BaseYarnClusterSuite extends SparkFunSuite
with Matchers {
props.setProperty(k, v)
}
}
+
+ // On a busy CI runner the in-JVM mini cluster, the driver and the
container JVMs all compete
+ // for CPU, and the driver's RPC server occasionally cannot accept a
connection in time. With
+ // the default of 3 retries an executor that loses this race gives up and
exits, which leaves
+ // the application unable to finish and the suite times out. Give the
executor->driver
+ // connection a larger retry budget so a transient stall does not
permanently fail the app.
+ // Set after the spark.* JVM properties are copied above so these values
are not silently
+ // overridden by an inherited -Dspark.rpc.io.* flag; individual tests can
still override them
+ // via extraConf below.
+ props.setProperty("spark.rpc.io.maxRetries", "10")
+ props.setProperty("spark.rpc.io.retryWait", "2s")
+
extraConf.foreach { case (k, v) => props.setProperty(k, v) }
val propsFile = File.createTempFile("spark", ".properties", tempDir)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
