This is an automated email from the ASF dual-hosted git repository. yangjie01 pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push: new 22a83caa489 [SPARK-45283][CORE][TESTS][3.5] Make StatusTrackerSuite less fragile 22a83caa489 is described below commit 22a83caa4896a8d03ec7e76b3e7a3bd08930adcb Author: Bo Xiong <xion...@amazon.com> AuthorDate: Tue Oct 17 18:05:23 2023 +0800 [SPARK-45283][CORE][TESTS][3.5] Make StatusTrackerSuite less fragile ### Why are the changes needed? It's discovered from [Github Actions](https://github.com/xiongbo-sjtu/spark/actions/runs/6270601155/job/17028788767) that StatusTrackerSuite can run into random failures, as shown by the following error message. The proposed fix is to update the unit test to remove the nondeterministic behavior. The fix has been made to the master branch in https://github.com/apache/spark/pull/43194. This PR is meant to patch branch-3.5 only. ``` [info] StatusTrackerSuite: [info] - basic status API usage (99 milliseconds) [info] - getJobIdsForGroup() (56 milliseconds) [info] - getJobIdsForGroup() with takeAsync() (48 milliseconds) [info] - getJobIdsForGroup() with takeAsync() across multiple partitions (58 milliseconds) [info] - getJobIdsForTag() *** FAILED *** (10 seconds, 77 milliseconds) [info] The code passed to eventually never returned normally. Attempted 651 times over 10.005059944000001 seconds. Last failure message: Set(3, 2, 1) was not equal to Set(1, 2). (StatusTrackerSuite.scala:148) ``` Full trace can be found [here](https://issues.apache.org/jira/browse/SPARK-45283). ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? ``` build/mvn package -DskipTests -pl core build/mvn -Dtest=none -DwildcardSuites=org.apache.spark.StatusTrackerSuite test ``` ### Was this patch authored or co-authored using generative AI tooling? No Closes #43388 from xiongbo-sjtu/branch-3.5. Authored-by: Bo Xiong <xion...@amazon.com> Signed-off-by: yangjie01 <yangji...@baidu.com> --- core/src/test/scala/org/apache/spark/StatusTrackerSuite.scala | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/core/src/test/scala/org/apache/spark/StatusTrackerSuite.scala b/core/src/test/scala/org/apache/spark/StatusTrackerSuite.scala index 0817abbc6a3..9019ea484b3 100644 --- a/core/src/test/scala/org/apache/spark/StatusTrackerSuite.scala +++ b/core/src/test/scala/org/apache/spark/StatusTrackerSuite.scala @@ -140,16 +140,19 @@ class StatusTrackerSuite extends SparkFunSuite with Matchers with LocalSparkCont } sc.removeJobTag("tag1") + // takeAsync() across multiple partitions val thirdJobFuture = sc.parallelize(1 to 1000, 2).takeAsync(999) - val thirdJobId = eventually(timeout(10.seconds)) { - thirdJobFuture.jobIds.head + val thirdJobIds = eventually(timeout(10.seconds)) { + // Wait for the two jobs triggered by takeAsync + thirdJobFuture.jobIds.size should be(2) + thirdJobFuture.jobIds } eventually(timeout(10.seconds)) { sc.statusTracker.getJobIdsForTag("tag1").toSet should be ( Set(firstJobId, secondJobId)) sc.statusTracker.getJobIdsForTag("tag2").toSet should be ( - Set(secondJobId, thirdJobId)) + Set(secondJobId) ++ thirdJobIds) } } } --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org