This is an automated email from the ASF dual-hosted git repository.
yangjie01 pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new 22a83caa489 [SPARK-45283][CORE][TESTS][3.5] Make StatusTrackerSuite
less fragile
22a83caa489 is described below
commit 22a83caa4896a8d03ec7e76b3e7a3bd08930adcb
Author: Bo Xiong <xion...@amazon.com>
AuthorDate: Tue Oct 17 18:05:23 2023 +0800
[SPARK-45283][CORE][TESTS][3.5] Make StatusTrackerSuite less fragile
### Why are the changes needed?
It's discovered from [Github
Actions](https://github.com/xiongbo-sjtu/spark/actions/runs/6270601155/job/17028788767)
that StatusTrackerSuite can run into random failures, as shown by the
following error message. The proposed fix is to update the unit test to remove
the nondeterministic behavior.
The fix has been made to the master branch in
https://github.com/apache/spark/pull/43194. This PR is meant to patch
branch-3.5 only.
```
[info] StatusTrackerSuite:
[info] - basic status API usage (99 milliseconds)
[info] - getJobIdsForGroup() (56 milliseconds)
[info] - getJobIdsForGroup() with takeAsync() (48 milliseconds)
[info] - getJobIdsForGroup() with takeAsync() across multiple
partitions (58 milliseconds)
[info] - getJobIdsForTag() *** FAILED *** (10 seconds, 77 milliseconds)
[info] The code passed to eventually never returned normally.
Attempted 651 times over 10.005059944000001 seconds.
Last failure message: Set(3, 2, 1) was not equal to Set(1, 2).
(StatusTrackerSuite.scala:148)
```
Full trace can be found
[here](https://issues.apache.org/jira/browse/SPARK-45283).
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
```
build/mvn package -DskipTests -pl core
build/mvn -Dtest=none -DwildcardSuites=org.apache.spark.StatusTrackerSuite
test
```
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #43388 from xiongbo-sjtu/branch-3.5.
Authored-by: Bo Xiong <xion...@amazon.com>
Signed-off-by: yangjie01 <yangji...@baidu.com>
---
core/src/test/scala/org/apache/spark/StatusTrackerSuite.scala | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/core/src/test/scala/org/apache/spark/StatusTrackerSuite.scala
b/core/src/test/scala/org/apache/spark/StatusTrackerSuite.scala
index 0817abbc6a3..9019ea484b3 100644
--- a/core/src/test/scala/org/apache/spark/StatusTrackerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/StatusTrackerSuite.scala
@@ -140,16 +140,19 @@ class StatusTrackerSuite extends SparkFunSuite with
Matchers with LocalSparkCont
}
sc.removeJobTag("tag1")
+
// takeAsync() across multiple partitions
val thirdJobFuture = sc.parallelize(1 to 1000, 2).takeAsync(999)
- val thirdJobId = eventually(timeout(10.seconds)) {
- thirdJobFuture.jobIds.head
+ val thirdJobIds = eventually(timeout(10.seconds)) {
+ // Wait for the two jobs triggered by takeAsync
+ thirdJobFuture.jobIds.size should be(2)
+ thirdJobFuture.jobIds
}
eventually(timeout(10.seconds)) {
sc.statusTracker.getJobIdsForTag("tag1").toSet should be (
Set(firstJobId, secondJobId))
sc.statusTracker.getJobIdsForTag("tag2").toSet should be (
- Set(secondJobId, thirdJobId))
+ Set(secondJobId) ++ thirdJobIds)
}
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
