This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 2a7e8900ab5f [SPARK-47076][CORE][TESTS] Fix HistoryServerSuite.`incomplete apps get refreshed` test to start with empty storeDir 2a7e8900ab5f is described below commit 2a7e8900ab5f3845c3ebe6f43ce9b5ccd95cec99 Author: Dongjoon Hyun <dh...@apple.com> AuthorDate: Fri Feb 16 14:27:16 2024 -0800 [SPARK-47076][CORE][TESTS] Fix HistoryServerSuite.`incomplete apps get refreshed` test to start with empty storeDir ### What changes were proposed in this pull request? This PR aims to fix `HistoryServerSuite`'s `incomplete apps get refreshed` test case to start with empty `storeDir`. ### Why are the changes needed? To fix a flakiness in `LevelDBBackendHistoryServerSuite` and `RocksDBBackendHistoryServerSuite`. ``` LevelDBBackendHistoryServerSuite ... info] - incomplete apps get refreshed *** FAILED *** (15 seconds, 397 milliseconds) [info] The code passed to eventually never returned normally. Attempted 43 times over 10.213112356000002 seconds. Last failure message: 4 did not equal 0. (HistoryServerSuite.scala:564) ``` ``` RocksDBBackendHistoryServerSuite ... [info] - incomplete apps get refreshed *** FAILED *** (15 seconds, 339 milliseconds) [info] The code passed to eventually never returned normally. Attempted 42 times over 10.135717083000001 seconds. Last failure message: 0 did not equal 4. (HistoryServerSuite.scala:564) ``` For the other test cases, `init` method recreates `storeDir`. https://github.com/apache/spark/blob/5faa512bf8c7c268613220d93ae4733cdf734cd0/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala#L85-L87 Also, `init` is invoked if `stop` is used manually. Only `incomplete apps get refreshed` test case doesn't clean up. https://github.com/apache/spark/blob/5faa512bf8c7c268613220d93ae4733cdf734cd0/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala#L611-L612 ### Does this PR introduce _any_ user-facing change? No. This is a single test-case change. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No Closes #45143 from dongjoon-hyun/SPARK-47076. Authored-by: Dongjoon Hyun <dh...@apple.com> Signed-off-by: Dongjoon Hyun <dh...@apple.com> --- .../scala/org/apache/spark/deploy/history/HistoryServerSuite.scala | 3 +++ 1 file changed, 3 insertions(+) diff --git a/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala b/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala index c8ffbf68a0c5..b1738b4e742b 100644 --- a/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala +++ b/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala @@ -390,6 +390,9 @@ abstract class HistoryServerSuite extends SparkFunSuite with BeforeAndAfter with // a new conf is used with the background thread set and running at its fastest // allowed refresh rate (1Hz) stop() + // Like 'init()', we need to clear the store directory of previously stopped server. + Utils.deleteRecursively(storeDir) + assert(storeDir.mkdir()) val myConf = new SparkConf() .set(HISTORY_LOG_DIR, logDir.getAbsolutePath) .set(EVENT_LOG_DIR, logDir.getAbsolutePath) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org