[jira] [Commented] (FLINK-16198) FileUtilsTest fails on Mac OS
[
https://issues.apache.org/jira/browse/FLINK-16198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17126385#comment-17126385
]
Zhe Yu commented on FLINK-16198:
PR: [https://github.com/apache/flink/pull/12413]
> FileUtilsTest fails on Mac OS
> -
>
> Key: FLINK-16198
> URL: https://issues.apache.org/jira/browse/FLINK-16198
> Project: Flink
> Issue Type: Bug
> Components: FileSystems, Tests
>Reporter: Andrey Zagrebin
>Assignee: Zhe Yu
>Priority: Major
> Labels: pull-request-available, starter
>
> The following tests fail if run on Mac OS (IDE/maven).
>
> FileUtilsTest.testCompressionOnRelativePath:
> {code:java}
> java.nio.file.NoSuchFileException:
> ../../../../../var/folders/67/v4yp_42d21j6_n8k1h556h0cgn/T/junit6496651678375117676/compressDir/rootDirjava.nio.file.NoSuchFileException:
>
> ../../../../../var/folders/67/v4yp_42d21j6_n8k1h556h0cgn/T/junit6496651678375117676/compressDir/rootDir
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
> at java.nio.file.Files.createDirectory(Files.java:674) at
> org.apache.flink.util.FileUtilsTest.verifyDirectoryCompression(FileUtilsTest.java:440)
> at
> org.apache.flink.util.FileUtilsTest.testCompressionOnRelativePath(FileUtilsTest.java:261)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at
> org.junit.rules.RunRules.evaluate(RunRules.java:20) at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363) at
> org.junit.runner.JUnitCore.run(JUnitCore.java:137) at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> {code}
>
> FileUtilsTest.testDeleteDirectoryConcurrently:
> {code:java}
> java.nio.file.FileSystemException:
> /var/folders/67/v4yp_42d21j6_n8k1h556h0cgn/T/junit7558825557740784886/junit3566161583262218465/ab1fa0bde8b22cad58b717508c7a7300/121fdf5f7b057183843ed2e1298f9b66/6598025f390d3084d69c98b36e542fe2/8db7cd9c063396a19a86f5b63ce53f66:
> Invalid argument at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> at
> sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
> at java.nio.file.Files.deleteIfExists(Files.java:1165)
> at
> org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:324)
> at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
> at
> org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:258)
> at
> org.apache.flink.util.FileUtils.cleanDirectoryInternal(FileUtils.java:376)
> at
> org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:335)
> at
>
[jira] [Commented] (FLINK-16198) FileUtilsTest fails on Mac OS
[
https://issues.apache.org/jira/browse/FLINK-16198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119295#comment-17119295
]
Zhe Yu commented on FLINK-16198:
Hi, [~aljoscha] I'm more than happy to take this.
Since this is about two test cases, I'm planning to create 2 sub issues for
them. But I'm also OK to deal with them together in this ticket. Let me know
which way you prefer. Feel free to assign this issue to me.
Besides, for the zipping test. I think it's also easy to fix, because the test
trying to prepare data using relative path, which break the test. We can just
use absolute path to prepare data, when test compressDirectory method feed
relative path.
Please let me know if this makes sense to you.
> FileUtilsTest fails on Mac OS
> -
>
> Key: FLINK-16198
> URL: https://issues.apache.org/jira/browse/FLINK-16198
> Project: Flink
> Issue Type: Bug
> Components: FileSystems, Tests
>Reporter: Andrey Zagrebin
>Priority: Major
> Labels: starter
>
> The following tests fail if run on Mac OS (IDE/maven).
>
> FileUtilsTest.testCompressionOnRelativePath:
> {code:java}
> java.nio.file.NoSuchFileException:
> ../../../../../var/folders/67/v4yp_42d21j6_n8k1h556h0cgn/T/junit6496651678375117676/compressDir/rootDirjava.nio.file.NoSuchFileException:
>
> ../../../../../var/folders/67/v4yp_42d21j6_n8k1h556h0cgn/T/junit6496651678375117676/compressDir/rootDir
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
> at java.nio.file.Files.createDirectory(Files.java:674) at
> org.apache.flink.util.FileUtilsTest.verifyDirectoryCompression(FileUtilsTest.java:440)
> at
> org.apache.flink.util.FileUtilsTest.testCompressionOnRelativePath(FileUtilsTest.java:261)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at
> org.junit.rules.RunRules.evaluate(RunRules.java:20) at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363) at
> org.junit.runner.JUnitCore.run(JUnitCore.java:137) at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> {code}
>
> FileUtilsTest.testDeleteDirectoryConcurrently:
> {code:java}
> java.nio.file.FileSystemException:
> /var/folders/67/v4yp_42d21j6_n8k1h556h0cgn/T/junit7558825557740784886/junit3566161583262218465/ab1fa0bde8b22cad58b717508c7a7300/121fdf5f7b057183843ed2e1298f9b66/6598025f390d3084d69c98b36e542fe2/8db7cd9c063396a19a86f5b63ce53f66:
> Invalid argument at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> at
> sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
> at java.nio.file.Files.deleteIfEx
[jira] [Comment Edited] (FLINK-16198) FileUtilsTest fails on Mac OS
[
https://issues.apache.org/jira/browse/FLINK-16198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098164#comment-17098164
]
Zhe Yu edited comment on FLINK-16198 at 5/3/20, 1:54 AM:
-
h3. Failed test case 1: FileUtilsTest.testCompressionOnRelativePath
*Root cause*: failed to create directory using relative path when preparing
file for testing.
*Solution*: Since goal of this test case is to verify correctness of
FileUtils.compressDirectory and FileUtils.expandDirectory
Thus refactor verifyDirectoryCompression to prepare test data using absolute
path while we will still use relative path to test FileUtils method will be
sufficient.
h3. Failed test case 2: FileUtilsTest.testDeleteDirectoryConcurrently
*Root cause:* FileUtils.deleteDirectory is not thread safe. This test case is
not always failing. In my local env(MacOS), it failed 7,8 times out of 10. The
test case is creating 3 level file hierarchy like below:
a
- b
- e
- c
- d
Then the test case started 3 threads. All of them are trying to iterate from
root directory: a. Thus, race condition will happen. There will be a chance 2
threads trying to delete /a/b/e/ at the same time. When this happens the test
will file. Maybe I'm missing something given [this comment|#L248]
*Solution:* make [deleteFileOrDirectoryInternal|#L317] synchronized will
protect the test from failing and
[guardIfWindows|[https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/util/FileUtils.java#L389]]
can be removed. But it also means concurrent deletion is disabled if 2 threads
trying to delete from the same root.
If above proposal sounds reasonable to you, feel free to assign this issue to
me. I'm also more than glad to talk about both of them.
was (Author: fishzhe):
h3. Failed test case 1: FileUtilsTest.testCompressionOnRelativePath
*Root cause*: failed to create directory using relative path when preparing
file for testing.
*Solution*: Since goal of this test case is to verify correctness of
FileUtils.compressDirectory and FileUtils.expandDirectory
Thus refactor verifyDirectoryCompression to prepare test data using absolute
path while we will still use relative path to test FileUtils method will be
sufficient.
h3. Failed test case 2: FileUtilsTest.testDeleteDirectoryConcurrently
*Root cause:* FileUtils.deleteDirectory is not thread safe. This test case is
not always failing. In my local env(MacOS), it failed 7,8 times out of 10. The
test case is creating 3 level file hierarchy like below:
a
- b
- e
- c
- d
Then the test case started 3 threads. All of them are trying to iterate from
root directory: a. Thus, race condition will happen. There will be a chance 2
threads trying to delete /a/b/e/ at the same time. When this happens the test
will file. Maybe I'm missing something given [this comment|#L248]]
*Solution:* make [deleteFileOrDirectoryInternal|#L317]] synchronized will
protect the test from failing. But it also means concurrent deletion is
disabled if 2 threads trying to delete from the same root.
If above proposal sounds reasonable to you, feel free to assign this issue to
me. I'm also more than glad to talk about both of them.
> FileUtilsTest fails on Mac OS
> -
>
> Key: FLINK-16198
> URL: https://issues.apache.org/jira/browse/FLINK-16198
> Project: Flink
> Issue Type: Bug
> Components: FileSystems, Tests
>Affects Versions: 1.11.0
>Reporter: Andrey Zagrebin
>Priority: Blocker
> Labels: starter
> Fix For: 1.11.0
>
>
> The following tests fail if run on Mac OS (IDE/maven).
>
> FileUtilsTest.testCompressionOnRelativePath:
> {code:java}
> java.nio.file.NoSuchFileException:
> ../../../../../var/folders/67/v4yp_42d21j6_n8k1h556h0cgn/T/junit6496651678375117676/compressDir/rootDirjava.nio.file.NoSuchFileException:
>
> ../../../../../var/folders/67/v4yp_42d21j6_n8k1h556h0cgn/T/junit6496651678375117676/compressDir/rootDir
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
> at java.nio.file.Files.createDirectory(Files.java:674) at
> org.apache.flink.util.FileUtilsTest.verifyDirectoryCompression(FileUtilsTest.java:440)
> at
> org.apache.flink.util.FileUtilsTest.testCompressionOnRelativePath(FileUtilsTest.java:261)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(Delegating
[jira] [Comment Edited] (FLINK-16198) FileUtilsTest fails on Mac OS
[
https://issues.apache.org/jira/browse/FLINK-16198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098164#comment-17098164
]
Zhe Yu edited comment on FLINK-16198 at 5/3/20, 1:49 AM:
-
h3. Failed test case 1: FileUtilsTest.testCompressionOnRelativePath
*Root cause*: failed to create directory using relative path when preparing
file for testing.
*Solution*: Since goal of this test case is to verify correctness of
FileUtils.compressDirectory and FileUtils.expandDirectory
Thus refactor verifyDirectoryCompression to prepare test data using absolute
path while we will still use relative path to test FileUtils method will be
sufficient.
h3. Failed test case 2: FileUtilsTest.testDeleteDirectoryConcurrently
*Root cause:* FileUtils.deleteDirectory is not thread safe. This test case is
not always failing. In my local env(MacOS), it failed 7,8 times out of 10. The
test case is creating 3 level file hierarchy like below:
a
- b
- e
- c
- d
Then the test case started 3 threads. All of them are trying to iterate from
root directory: a. Thus, race condition will happen. There will be a chance 2
threads trying to delete /a/b/e/ at the same time. When this happens the test
will file. Maybe I'm missing something given [this comment|#L248]]
*Solution:* make [deleteFileOrDirectoryInternal|#L317]] synchronized will
protect the test from failing. But it also means concurrent deletion is
disabled if 2 threads trying to delete from the same root.
If above proposal sounds reasonable to you, feel free to assign this issue to
me. I'm also more than glad to talk about both of them.
was (Author: fishzhe):
h3. Failed test case 1: FileUtilsTest.testCompressionOnRelativePath
*Root cause*: failed to create directory using relative path when preparing
file for testing.
*Solution*: Since goal of this test case is to verify correctness of
FileUtils.compressDirectory and FileUtils.expandDirectory
Thus refactor verifyDirectoryCompression to prepare test data using absolute
path while we will still use relative path to test FileUtils method will be
sufficient.
h3. Failed test case 2: FileUtilsTest.testDeleteDirectoryConcurrently
*Root cause:* FileUtils.deleteDirectory is not thread safe. This test case is
not always failing. In my local env(MacOS), it failed 7,8 times out of 10. The
test case is creating 3 level file hierarchy like below:
- a
- b
- e
- c
- d
Then the test case started 3 threads. All of them are trying to iterate from
root directory: a. Thus, race condition will happen. There will be a chance 2
threads trying to delete /a/b/e/ at the same time. When this happens the test
will file. Maybe I'm missing something given [this comment|#L248]]
*Solution:* make [deleteFileOrDirectoryInternal|#L317]] synchronized will
protect the test from failing. But it also means concurrent deletion is
disabled if 2 threads trying to delete from the same root.
If above proposal sounds reasonable to you, feel free to assign this issue to
me. I'm also more than glad to talk about both of them.
> FileUtilsTest fails on Mac OS
> -
>
> Key: FLINK-16198
> URL: https://issues.apache.org/jira/browse/FLINK-16198
> Project: Flink
> Issue Type: Bug
> Components: FileSystems, Tests
>Affects Versions: 1.11.0
>Reporter: Andrey Zagrebin
>Priority: Blocker
> Labels: starter
> Fix For: 1.11.0
>
>
> The following tests fail if run on Mac OS (IDE/maven).
>
> FileUtilsTest.testCompressionOnRelativePath:
> {code:java}
> java.nio.file.NoSuchFileException:
> ../../../../../var/folders/67/v4yp_42d21j6_n8k1h556h0cgn/T/junit6496651678375117676/compressDir/rootDirjava.nio.file.NoSuchFileException:
>
> ../../../../../var/folders/67/v4yp_42d21j6_n8k1h556h0cgn/T/junit6496651678375117676/compressDir/rootDir
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
> at java.nio.file.Files.createDirectory(Files.java:674) at
> org.apache.flink.util.FileUtilsTest.verifyDirectoryCompression(FileUtilsTest.java:440)
> at
> org.apache.flink.util.FileUtilsTest.testCompressionOnRelativePath(FileUtilsTest.java:261)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall
[jira] [Commented] (FLINK-16198) FileUtilsTest fails on Mac OS
[
https://issues.apache.org/jira/browse/FLINK-16198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098164#comment-17098164
]
Zhe Yu commented on FLINK-16198:
h3. Failed test case 1: FileUtilsTest.testCompressionOnRelativePath
*Root cause*: failed to create directory using relative path when preparing
file for testing.
*Solution*: Since goal of this test case is to verify correctness of
FileUtils.compressDirectory and FileUtils.expandDirectory
Thus refactor verifyDirectoryCompression to prepare test data using absolute
path while we will still use relative path to test FileUtils method will be
sufficient.
h3. Failed test case 2: FileUtilsTest.testDeleteDirectoryConcurrently
*Root cause:* FileUtils.deleteDirectory is not thread safe. This test case is
not always failing. In my local env(MacOS), it failed 7,8 times out of 10. The
test case is creating 3 level file hierarchy like below:
- a
- b
- e
- c
- d
Then the test case started 3 threads. All of them are trying to iterate from
root directory: a. Thus, race condition will happen. There will be a chance 2
threads trying to delete /a/b/e/ at the same time. When this happens the test
will file. Maybe I'm missing something given [this
comment|[https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/util/FileUtils.java#L248]]
*Solution:* make
[deleteFileOrDirectoryInternal|[https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/util/FileUtils.java#L317]]
synchronized will protect the test from failing. But it also means concurrent
deletion is disabled if 2 threads trying to delete from the same root.
If above proposal sounds reasonable to you, feel free to assign this issue to
me. I'm also more than glad to talk about both of them.
> FileUtilsTest fails on Mac OS
> -
>
> Key: FLINK-16198
> URL: https://issues.apache.org/jira/browse/FLINK-16198
> Project: Flink
> Issue Type: Bug
> Components: FileSystems, Tests
>Affects Versions: 1.11.0
>Reporter: Andrey Zagrebin
>Priority: Blocker
> Labels: starter
> Fix For: 1.11.0
>
>
> The following tests fail if run on Mac OS (IDE/maven).
>
> FileUtilsTest.testCompressionOnRelativePath:
> {code:java}
> java.nio.file.NoSuchFileException:
> ../../../../../var/folders/67/v4yp_42d21j6_n8k1h556h0cgn/T/junit6496651678375117676/compressDir/rootDirjava.nio.file.NoSuchFileException:
>
> ../../../../../var/folders/67/v4yp_42d21j6_n8k1h556h0cgn/T/junit6496651678375117676/compressDir/rootDir
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
> at java.nio.file.Files.createDirectory(Files.java:674) at
> org.apache.flink.util.FileUtilsTest.verifyDirectoryCompression(FileUtilsTest.java:440)
> at
> org.apache.flink.util.FileUtilsTest.testCompressionOnRelativePath(FileUtilsTest.java:261)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at
> org.junit.rules.RunRules.evaluate(RunRules.java:20) at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363) at
> org.junit.runner.JUnitCore.run(JUnitCore.java:137) at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunne
[jira] [Comment Edited] (FLINK-16198) FileUtilsTest fails on Mac OS
[
https://issues.apache.org/jira/browse/FLINK-16198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098164#comment-17098164
]
Zhe Yu edited comment on FLINK-16198 at 5/3/20, 1:48 AM:
-
h3. Failed test case 1: FileUtilsTest.testCompressionOnRelativePath
*Root cause*: failed to create directory using relative path when preparing
file for testing.
*Solution*: Since goal of this test case is to verify correctness of
FileUtils.compressDirectory and FileUtils.expandDirectory
Thus refactor verifyDirectoryCompression to prepare test data using absolute
path while we will still use relative path to test FileUtils method will be
sufficient.
h3. Failed test case 2: FileUtilsTest.testDeleteDirectoryConcurrently
*Root cause:* FileUtils.deleteDirectory is not thread safe. This test case is
not always failing. In my local env(MacOS), it failed 7,8 times out of 10. The
test case is creating 3 level file hierarchy like below:
- a
- b
- e
- c
- d
Then the test case started 3 threads. All of them are trying to iterate from
root directory: a. Thus, race condition will happen. There will be a chance 2
threads trying to delete /a/b/e/ at the same time. When this happens the test
will file. Maybe I'm missing something given [this comment|#L248]]
*Solution:* make [deleteFileOrDirectoryInternal|#L317]] synchronized will
protect the test from failing. But it also means concurrent deletion is
disabled if 2 threads trying to delete from the same root.
If above proposal sounds reasonable to you, feel free to assign this issue to
me. I'm also more than glad to talk about both of them.
was (Author: fishzhe):
h3. Failed test case 1: FileUtilsTest.testCompressionOnRelativePath
*Root cause*: failed to create directory using relative path when preparing
file for testing.
*Solution*: Since goal of this test case is to verify correctness of
FileUtils.compressDirectory and FileUtils.expandDirectory
Thus refactor verifyDirectoryCompression to prepare test data using absolute
path while we will still use relative path to test FileUtils method will be
sufficient.
h3. Failed test case 2: FileUtilsTest.testDeleteDirectoryConcurrently
*Root cause:* FileUtils.deleteDirectory is not thread safe. This test case is
not always failing. In my local env(MacOS), it failed 7,8 times out of 10. The
test case is creating 3 level file hierarchy like below:
- a
- b
- e
- c
- d
Then the test case started 3 threads. All of them are trying to iterate from
root directory: a. Thus, race condition will happen. There will be a chance 2
threads trying to delete /a/b/e/ at the same time. When this happens the test
will file. Maybe I'm missing something given [this
comment|[https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/util/FileUtils.java#L248]]
*Solution:* make
[deleteFileOrDirectoryInternal|[https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/util/FileUtils.java#L317]]
synchronized will protect the test from failing. But it also means concurrent
deletion is disabled if 2 threads trying to delete from the same root.
If above proposal sounds reasonable to you, feel free to assign this issue to
me. I'm also more than glad to talk about both of them.
> FileUtilsTest fails on Mac OS
> -
>
> Key: FLINK-16198
> URL: https://issues.apache.org/jira/browse/FLINK-16198
> Project: Flink
> Issue Type: Bug
> Components: FileSystems, Tests
>Affects Versions: 1.11.0
>Reporter: Andrey Zagrebin
>Priority: Blocker
> Labels: starter
> Fix For: 1.11.0
>
>
> The following tests fail if run on Mac OS (IDE/maven).
>
> FileUtilsTest.testCompressionOnRelativePath:
> {code:java}
> java.nio.file.NoSuchFileException:
> ../../../../../var/folders/67/v4yp_42d21j6_n8k1h556h0cgn/T/junit6496651678375117676/compressDir/rootDirjava.nio.file.NoSuchFileException:
>
> ../../../../../var/folders/67/v4yp_42d21j6_n8k1h556h0cgn/T/junit6496651678375117676/compressDir/rootDir
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
> at java.nio.file.Files.createDirectory(Files.java:674) at
> org.apache.flink.util.FileUtilsTest.verifyDirectoryCompression(FileUtilsTest.java:440)
> at
> org.apache.flink.util.FileUtilsTest.testCompressionOnRelativePath(FileUtilsTest.java:261)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
