This is an automated email from the ASF dual-hosted git repository.
SteNicholas pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new 83be00326 [CELEBORN-2431] Fix flaky RemoteShuffleMasterSuiteJ
shuffleCount assertions racing with app heartbeat counter reset
83be00326 is described below
commit 83be003260455cbb1b905b4884b23b1d92796e74
Author: yew1eb <[email protected]>
AuthorDate: Thu Aug 20 23:05:15 2026 +0800
[CELEBORN-2431] Fix flaky RemoteShuffleMasterSuiteJ shuffleCount assertions
racing with app heartbeat counter reset
### What changes were proposed in this pull request?
Increase `celeborn.client.application.heartbeatInterval` from `30s` to `1h`
in `RemoteShuffleMasterSuiteJ.setUp()`, which effectively pauses the
LifecycleManager app heartbeater for the duration of this test suite. All
version-specific subclasses (`RemoteShuffleMasterSuiteJV118` ~ `V23`) inherit
from this base suite, so a single change covers all of them.
### Why are the changes needed?
`RemoteShuffleMasterSuiteJ.testRegisterPartitionWithProducer` (and the
other `shuffleCount` assertions in the same suite) is flaky:
```
java.lang.AssertionError: expected:<1> but was:<0>
at
RemoteShuffleMasterSuiteJ.testRegisterPartitionWithProducer(RemoteShuffleMasterSuiteJ.java:124)
```
Root cause is a race on `LifecycleManager#shuffleCount`:
- `RemoteShuffleMaster.registerPartitionWithProducer` increments the
counter (`RemoteShuffleMaster.java:161`)
- the LifecycleManager app-heartbeat thread periodically ships the counter
to the master via `shuffleCount.sumThenReset()` (`LifecycleManager.scala:234`)
The suite configures `celeborn.client.application.heartbeatInterval=30s`.
On slow CI runners a test can run past the 30s mark, so a heartbeat fires
between the increment and the assertion and resets the counter to 0.
`shuffleFallbackCounts` assertions are affected by the same race
(`resetFallbackCounts`). No test in this suite depends on an app heartbeat
actually being sent, so enlarging the interval has no side effects.
Example failure (flink1 (11, 1.18)):
https://github.com/apache/celeborn/actions/runs/32273603672/job/96135975529
### Does this PR resolve a correctness bug?
- [ ] Yes
### Does this PR introduce _any_ user-facing change?
- [ ] Yes
### How was this patch tested?
`./build/mvn -Pflink-1.18 -pl client-flink/flink-1.18 test
-Dtest=RemoteShuffleMasterSuiteJV118` — all 7 tests passed (previously
`shuffleCount` assertions could intermittently fail on slow runners).
Closes #3811 from yew1eb/CELEBORN-2431.
Authored-by: yew1eb <[email protected]>
Signed-off-by: Nicholas Jiang <[email protected]>
---
.../org/apache/celeborn/plugin/flink/RemoteShuffleMasterSuiteJ.java | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git
a/client-flink/common/src/test/java/org/apache/celeborn/plugin/flink/RemoteShuffleMasterSuiteJ.java
b/client-flink/common/src/test/java/org/apache/celeborn/plugin/flink/RemoteShuffleMasterSuiteJ.java
index 018bd4b4d..c0e73422b 100644
---
a/client-flink/common/src/test/java/org/apache/celeborn/plugin/flink/RemoteShuffleMasterSuiteJ.java
+++
b/client-flink/common/src/test/java/org/apache/celeborn/plugin/flink/RemoteShuffleMasterSuiteJ.java
@@ -72,7 +72,10 @@ public class RemoteShuffleMasterSuiteJ {
int startPort = Utils$.MODULE$.selectRandomInt(1024, 65535);
configuration.setString("celeborn.master.port", String.valueOf(startPort));
configuration.setString("celeborn.master.endpoints", "localhost:" +
startPort);
- configuration.setString("celeborn.client.application.heartbeatInterval",
"30s");
+ // Set a large app heartbeat interval to avoid the app heartbeater
resetting
+ // LifecycleManager#shuffleCount/shuffleFallbackCounts via sumThenReset in
the
+ // middle of a test, which makes the count assertions below flaky.
+ configuration.setString("celeborn.client.application.heartbeatInterval",
"1h");
remoteShuffleMaster = createShuffleMaster(configuration);
}
