This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 630dfdf [SPARK-28101][DSTREAM][TEST] Fix Flaky Test: `InputStreamsSuite.Modified files are correctly detected` in JDK9+ 630dfdf is described below commit 630dfdf550690b5519c69a96d7aca041587c5359 Author: Dongjoon Hyun <dh...@apple.com> AuthorDate: Wed Jun 19 07:55:00 2019 -0700 [SPARK-28101][DSTREAM][TEST] Fix Flaky Test: `InputStreamsSuite.Modified files are correctly detected` in JDK9+ ## What changes were proposed in this pull request? It seems that https://bugs.openjdk.java.net/browse/JDK-8068730 makes `InputStreamsSuite` very flaky. <img width="903" alt="error" src="https://user-images.githubusercontent.com/9700541/59727067-017eb780-91e9-11e9-8bb0-ac5f4c1bc44d.png"> As we can see the Jenkins result, this can be reproduced frequently with JDK9+. ``` $ build/sbt "streaming/testOnly *.InputStreamsSuite" [info] - Modified files are correctly detected. *** FAILED *** (134 milliseconds) [info] Set("renamed") did not equal Set() (InputStreamsSuite.scala:312) [info] org.scalatest.exceptions.TestFailedException: ``` The reason is the `renamed.txt`'s modification time becomes greater than the clock in JDK9+ and Spark ignored it with **not selected** message. In JDK8, the modification time generated by this test case doesn't have `milliseconds` part. ``` Getting new files for time 1560896662000, ignoring files older than 1560896659679 file:/.../streaming/subdir/renamed.txt not selected as mod time 1560896662679 > current time 1560896662000 file:/.../streaming/subdir/existing ignored as mod time 1560896657679 <= ignore time 1560896659679 Finding new files took 0 ms New files at time 1560896662000 ms: ``` ## How was this patch tested? Pass the Jenkins and manually repeat the following with JDK11 10 times. ``` $ build/sbt "streaming/testOnly *.InputStreamsSuite" ``` Closes #24904 from dongjoon-hyun/SPARK-28101. Authored-by: Dongjoon Hyun <dh...@apple.com> Signed-off-by: Dongjoon Hyun <dh...@apple.com> --- .../src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala b/streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala index e81cfb5..035ed4a 100644 --- a/streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala +++ b/streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala @@ -293,8 +293,7 @@ class InputStreamsSuite extends TestSuiteBase with BeforeAndAfter { val textPath = new Path(generatedSubDir, "renamed.txt") write(textPath, "renamed\n") val now = clock.getTimeMillis() - val modTime = now + durationMs / 2 - fs.setTimes(textPath, modTime, modTime) + fs.setTimes(textPath, now, now) val textFilestatus = fs.getFileStatus(existingFile) assert(textFilestatus.getModificationTime < now + durationMs) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org