[spark] branch master updated: [SPARK-28101][DSTREAM][TEST] Fix Flaky Test: `InputStreamsSuite.Modified files are correctly detected` in JDK9+

[dongjoon]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 630dfdf  [SPARK-28101][DSTREAM][TEST] Fix Flaky Test: 
`InputStreamsSuite.Modified files are correctly detected` in JDK9+
630dfdf is described below

commit 630dfdf550690b5519c69a96d7aca041587c5359
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Wed Jun 19 07:55:00 2019 -0700

    [SPARK-28101][DSTREAM][TEST] Fix Flaky Test: `InputStreamsSuite.Modified 
files are correctly detected` in JDK9+
    
    ## What changes were proposed in this pull request?
    
    It seems that https://bugs.openjdk.java.net/browse/JDK-8068730 makes 
`InputStreamsSuite` very flaky.
    
    <img width="903" alt="error" 
src="https://user-images.githubusercontent.com/9700541/59727067-017eb780-91e9-11e9-8bb0-ac5f4c1bc44d.png";>
    
    As we can see the Jenkins result, this can be reproduced frequently with 
JDK9+.
    ```
    $ build/sbt "streaming/testOnly *.InputStreamsSuite"
    [info] - Modified files are correctly detected. *** FAILED *** (134 
milliseconds)
    [info]   Set("renamed") did not equal Set() (InputStreamsSuite.scala:312)
    [info]   org.scalatest.exceptions.TestFailedException:
    ```
    
    The reason is the `renamed.txt`'s modification time becomes greater than 
the clock in JDK9+ and Spark ignored it with **not selected** message. In JDK8, 
 the modification time generated by this test case doesn't have `milliseconds` 
part.
    ```
    Getting new files for time 1560896662000, ignoring files older than 
1560896659679
    file:/.../streaming/subdir/renamed.txt not selected as mod time 
1560896662679 > current time 1560896662000
    file:/.../streaming/subdir/existing ignored as mod time 1560896657679 <= 
ignore time 1560896659679
    Finding new files took 0 ms
    New files at time 1560896662000 ms:
    ```
    
    ## How was this patch tested?
    
    Pass the Jenkins and manually repeat the following with JDK11 10 times.
    ```
    $ build/sbt "streaming/testOnly *.InputStreamsSuite"
    ```
    
    Closes #24904 from dongjoon-hyun/SPARK-28101.
    
    Authored-by: Dongjoon Hyun <dh...@apple.com>
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
 .../src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala  | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git 
a/streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala 
b/streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala
index e81cfb5..035ed4a 100644
--- 
a/streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala
+++ 
b/streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala
@@ -293,8 +293,7 @@ class InputStreamsSuite extends TestSuiteBase with 
BeforeAndAfter {
         val textPath = new Path(generatedSubDir, "renamed.txt")
         write(textPath, "renamed\n")
         val now = clock.getTimeMillis()
-        val modTime = now + durationMs / 2
-        fs.setTimes(textPath, modTime, modTime)
+        fs.setTimes(textPath, now, now)
         val textFilestatus = fs.getFileStatus(existingFile)
         assert(textFilestatus.getModificationTime < now + durationMs)
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org