HyukjinKwon opened a new pull request, #56800: URL: https://github.com/apache/spark/pull/56800
### What changes were proposed in this pull request? Two test-only mitigations in `BaseYarnClusterSuite` to reduce `YarnClusterSuite` flakiness on busy CI runners: - Give the mini `NodeManager` 8GB (`yarn.nodemanager.resource.memory-mb`) and a matching `yarn.scheduler.maximum-allocation-mb`, so executor allocation is never starved once the ~1.4GB AM is running. - Raise the executor→driver connection retry budget (`spark.rpc.io.maxRetries=10`, `spark.rpc.io.retryWait=2s`) so a transient accept stall does not permanently fail the executor. These are defaults; individual tests can still override them. ### Why are the changes needed? This is a follow-up to SPARK-57650, which fixed the deterministic ACCEPTED-state hang (`maximum-am-resource-percent`). After that fix, the master `Build / Maven` `yarn` lanes still go red intermittently: `YarnClusterSuite` tests time out (`handle.getState().isFinal() was false`) because the AM/executor containers fail to connect to the driver's RPC server on localhost (Connection refused). The in-JVM mini RM+NM, the driver subprocess, and the AM/executor JVMs all contend for CPU on a single CI runner, so the driver's accept loop occasionally stalls; an executor that loses the race exits after the default 3 connection retries, and the application can then never finish. This is one of the two remaining unmerged fixes keeping the apache/spark master matrix builds red. ### Does this PR introduce _any_ user-facing change? No. Test-only. ### How was this patch tested? `resource-managers/yarn` module tests on a fork, repeated multiple times to confirm the flakiness is gone (the formerly-failing `YarnClusterSuite` tests pass consistently). - **Before (still flaky on apache/spark master, after the SPARK-57650 base fix merged):** `Build / Maven (Scala 2.13, JDK 17)` — `resource-managers#yarn`, `YarnClusterSuite` times out (`handle.getState().isFinal() was false`, `BaseYarnClusterSuite.scala:220`): https://github.com/apache/spark/actions/runs/28174751831 - **After (passing on this branch):** `resource-managers/yarn` module green on the fork — https://github.com/HyukjinKwon/spark/actions/runs/28204995015/job/83556094951 ### Was this patch authored or co-authored using generative AI tooling? Yes. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
