This is an automated email from the ASF dual-hosted git repository. HyukjinKwon pushed a commit to branch branch-4.1 in repository https://gitbox.apache.org/repos/asf/spark.git
commit 012053f3c94669a1ae6a8b18bd25a0d46b386e03 Author: Hyukjin Kwon <[email protected]> AuthorDate: Sat Jun 27 11:09:09 2026 +0900 [SPARK-57710][YARN][TEST] Reduce YarnClusterSuite flakiness from CI runner contention ### What changes were proposed in this pull request? Follow-up to [SPARK-57650](https://issues.apache.org/jira/browse/SPARK-57650), which fixed the deterministic "AM stuck in ACCEPTED" hang in `BaseYarnClusterSuite`. Two further **test-only** changes to reduce the remaining flakiness of `YarnClusterSuite`: 1. Give the single mini `NodeManager` 8GB (`yarn.nodemanager.resource.memory-mb` + `yarn.scheduler.maximum-allocation-mb`) so executor allocation is never starved once the ~1.4GB AM is running. 2. Raise the executor→driver connection retry budget for the launched apps (`spark.rpc.io.maxRetries=10`, `spark.rpc.io.retryWait=2s`) so a transient RPC-accept stall does not permanently fail an executor. These are defaults that individual tests can still override via `extraConf`. ### Why are the changes needed? Even after SPARK-57650, the scheduled `Build / Java21` and `Build / Java25` master lanes fail in the `yarn` module roughly **50% of runs** (e.g. fork run `28151220075` PASS vs `28151247521` FAIL — same commit, 40s apart). All failures are the same six `YarnClusterSuite` tests timing out after 3 minutes (`The code passed to eventually never returned normally ... handle.getState().isFinal() was false`). From the `yarn-app-log` / `unit-tests-log` artifacts, the AM/driver comes up, but the executor (and sometimes the AM) intermittently fail to connect back to the driver's RPC server on `localhost` (`java.io.IOException: Failed to connect to localhost/127.0.0.1:<port>`, connection refused). The in-JVM mini RM+NM, the driver subprocess and the AM/executor JVMs all contend for CPU on a single CI runner, so the driver's accept loop occasionally stalls; an executor that loses this race exit [...] ### Does this PR introduce any user-facing change? No. Test-only. ### How was this patch tested? `YarnClusterSuite` was previously failing ~50% of the time. With this change the `yarn` module job was run **6 times** on the fork; all 6 passed, with `YarnClusterSuite` reporting `tests=30, failures=0, skipped=0` (the 6 formerly-failing tests now pass): - Before (master, failing): apache/spark run `28148781009` (`Build / Java21`) — 6 `YarnClusterSuite` timeouts. - After (this branch): HyukjinKwon/spark runs `28162182834`, `28162247111`, `28162249819`, `28175759262`, `28175762257`, `28175765871` — `yarn` job green in all six. ### Was this patch authored or co-authored using generative AI tooling? Yes, Generated-by: Claude Code. Closes #56785 from HyukjinKwon/ci-fix/agent8-yarn-cluster-flaky. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]> (cherry picked from commit 41f3e67e3ebe9ca654af98c6464d607c92ae97f3) --- .../spark/deploy/yarn/BaseYarnClusterSuite.scala | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala index 69c515d6a381..5112e62d838c 100644 --- a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala +++ b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala @@ -111,6 +111,14 @@ abstract class BaseYarnClusterSuite extends SparkFunSuite with Matchers { yarnConf.setFloat("yarn.scheduler.capacity.maximum-am-resource-percent", 1.0f) yarnConf.setFloat("yarn.scheduler.capacity.root.default.maximum-am-resource-percent", 1.0f) + // Give the single mini NodeManager generous memory. By default the mini cluster advertises + // only a small amount of memory, so once the AM (~1.4GB) is running there is barely enough + // headroom left for the executors these tests request. On a busy CI runner that makes + // executor allocation slow/racy and the YarnClusterSuite apps time out waiting to finish. + // The CI hosts have plenty of RAM, so let the NM offer enough for the AM plus a few executors. + yarnConf.setInt("yarn.nodemanager.resource.memory-mb", 8192) + yarnConf.setInt("yarn.scheduler.maximum-allocation-mb", 8192) + // Support both IPv4 and IPv6 yarnConf.set("yarn.resourcemanager.hostname", Utils.localHostNameForURI()) @@ -292,6 +300,18 @@ abstract class BaseYarnClusterSuite extends SparkFunSuite with Matchers { props.setProperty(k, v) } } + + // On a busy CI runner the in-JVM mini cluster, the driver and the container JVMs all compete + // for CPU, and the driver's RPC server occasionally cannot accept a connection in time. With + // the default of 3 retries an executor that loses this race gives up and exits, which leaves + // the application unable to finish and the suite times out. Give the executor->driver + // connection a larger retry budget so a transient stall does not permanently fail the app. + // Set after the spark.* JVM properties are copied above so these values are not silently + // overridden by an inherited -Dspark.rpc.io.* flag; individual tests can still override them + // via extraConf below. + props.setProperty("spark.rpc.io.maxRetries", "10") + props.setProperty("spark.rpc.io.retryWait", "2s") + extraConf.foreach { case (k, v) => props.setProperty(k, v) } val propsFile = File.createTempFile("spark", ".properties", tempDir) --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
