This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new a78579ea915 [SPARK-41972][TESTS] Fix a flaky test in StreamingQueryStatusListenerSuite a78579ea915 is described below commit a78579ea91562eb62fdd5f567a8201c78a219669 Author: Gengliang Wang <gengli...@apache.org> AuthorDate: Tue Jan 10 23:13:08 2023 -0800 [SPARK-41972][TESTS] Fix a flaky test in StreamingQueryStatusListenerSuite ### What changes were proposed in this pull request? Fix flaky test in StreamingQueryStatusListenerSuite. Example failed test https://github.com/gengliangwang/spark/actions/runs/3880260857/jobs/6629409770: ``` StreamingQueryStatusListenerSuite.test small retained queries org.scalatest.exceptions.TestFailedDueToTimeoutException: The code passed to eventually never returned normally. Attempted 638 times over 10.006178314 seconds. Last failure message: Set(301dad06-8fd7-4766-8d61-a58724d2c966, fab95baf-25ad-47dc-89fd-ff49bf1a4c4e) did not equal Set(60d3b5b0-7397-4212-8a18-ff67a17f362a, fab95baf-25ad-47dc-89fd-ff49bf1a4c4e). ``` The retention queries after eviction(by the ending timestamp) are different from what is expected. I suspect that some of the input queries has the same ending timestamp. Another proof: with RocksDB as backend, the test case always pass since writing to RocksDB takes more time. So, I suggest adding `Thread.sleep` between the termination of these queries to fix the flakiness. ### Why are the changes needed? Fix flaky test ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? GA test Closes #39494 from gengliangwang/fixFlaky. Authored-by: Gengliang Wang <gengli...@apache.org> Signed-off-by: Dongjoon Hyun <dongj...@apache.org> --- .../spark/sql/streaming/ui/StreamingQueryStatusListenerSuite.scala | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/sql/core/src/test/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatusListenerSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatusListenerSuite.scala index ad4c1ff28d1..58e04eb285a 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatusListenerSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatusListenerSuite.scala @@ -181,9 +181,15 @@ class StreamingQueryStatusListenerSuite extends StreamTest { val terminateEvent1 = new StreamingQueryListener.QueryTerminatedEvent(id1, runId1, None) listener.onQueryTerminated(terminateEvent1) checkInactiveQueryStatus(1, Seq(id1)) + // SPARK-41972: having a short sleep here to make sure the end time of query 2 is larger than + // query 1. + Thread.sleep(20) val terminateEvent2 = new StreamingQueryListener.QueryTerminatedEvent(id2, runId2, None) listener.onQueryTerminated(terminateEvent2) checkInactiveQueryStatus(2, Seq(id1, id2)) + // SPARK-41972: having a short sleep here to make sure the end time of query 3 is larger than + // query 2. + Thread.sleep(20) val terminateEvent3 = new StreamingQueryListener.QueryTerminatedEvent(id3, runId3, None) listener.onQueryTerminated(terminateEvent3) checkInactiveQueryStatus(2, Seq(id2, id3)) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org