Quanlong Huang created IMPALA-11720:
---------------------------------------
Summary: FileMetadataLoaderTest is flaky due to IOException:
Filesystem closed
Key: IMPALA-11720
URL: https://issues.apache.org/jira/browse/IMPALA-11720
Project: IMPALA
Issue Type: Bug
Components: Frontend
Reporter: Quanlong Huang
Assignee: Quanlong Huang
IMPALA-11699 recently reverts the changes in FileMetadataLoaderTest that
extends it to FrontendTestBase. This causes tests in it become flaky if run
individually. E.g. the following commands all fail:
{code:java}
(pushd fe && mvn test -Dtest=FileMetadataLoaderTest#testRecursiveLoading)
(pushd fe && mvn test -Dtest=FileMetadataLoaderTest#testHudiParquetLoading)
(pushd fe && mvn test -Dtest=FileMetadataLoaderTest#testIcebergLoading)
(pushd fe && mvn test
-Dtest=FileMetadataLoaderTest#testAcidMinorCompactionLoading)
(pushd fe && mvn test -Dtest=FileMetadataLoaderTest#testLoadMissingDirectory)
(pushd fe && mvn test
-Dtest=FileMetadataLoaderTest#testSkipHiddenDirectories){code}
and the failures are the same, i.e. due to java.io.IOException: Filesystem
closed
{noformat}
$ (pushd fe && mvn test -Dtest=FileMetadataLoaderTest#testLoadMissingDirectory)
...
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.601 s
<<< FAILURE! - in org.apache.impala.catalog.FileMetadataLoaderTest
[ERROR]
testLoadMissingDirectory(org.apache.impala.catalog.FileMetadataLoaderTest)
Time elapsed: 1.565 s <<< ERROR!
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:483)
at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1662)
at
org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1266)
at
org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1249)
at
org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1194)
at
org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1190)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:1208)
at
org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:2145)
at
org.apache.impala.common.FileSystemUtil.listLocatedStatusIterator(FileSystemUtil.java:804)
at
org.apache.impala.common.FileSystemUtil$RecursingIterator.<init>(FileSystemUtil.java:975)
at
org.apache.impala.common.FileSystemUtil$RecursingIterator.<init>(FileSystemUtil.java:958)
at
org.apache.impala.common.FileSystemUtil.listFiles(FileSystemUtil.java:768)
at
org.apache.impala.catalog.FileMetadataLoader.load(FileMetadataLoader.java:187)
at
org.apache.impala.catalog.FileMetadataLoaderTest.testLoadMissingDirectory(FileMetadataLoaderTest.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
{noformat}
However, any test runs after the failure will succeed. If we run the test twice
(by adding -Dsurefire.rerunFailingTestsCount=2), the second run also succeed:
{noformat}
$ (pushd fe && mvn test -Dtest=FileMetadataLoaderTest#testLoadMissingDirectory
-Dsurefire.rerunFailingTestsCount=2)
...
[WARNING] Flakes:
[WARNING]
org.apache.impala.catalog.FileMetadataLoaderTest.testLoadMissingDirectory(org.apache.impala.catalog.FileMetadataLoaderTest)
[ERROR] Run 1: FileMetadataLoaderTest.testLoadMissingDirectory:192 » IO
Filesystem closed
[INFO] Run 2: PASS
[INFO]
[INFO]
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
