[ https://issues.apache.org/jira/browse/HDFS-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wei-Chiu Chuang updated HDFS-9631: ---------------------------------- Description: I found a number of {{TestOpenFilesWithSnapshot}} tests failed quite frequently. These tests ({{testParentDirWithUCFileDeleteWithSnapshot}}, {{testOpenFilesWithRename}}, {{testWithCheckpoint}}) are unable to reconnect to the namenode after restart. It looks like the reconnection failed due to an EOFException between data node and the name node. {noformat} FAILED: org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot.testParentDirWithUCFileDeleteWithSnapShot Error Message: Timed out waiting for Mini HDFS Cluster to start Stack Trace: java.io.IOException: Timed out waiting for Mini HDFS Cluster to start at org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1345) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:2024) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1985) at org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot.testParentDirWithUCFileDeleteWithSnapShot(TestOpenFilesWithSnapshot.java:82) {noformat} It appears that these three tests all call {{doWriteAndAbort()}}, which creates files and then abort, and then set the parent directory with a snapshot, and then delete the parent directory. Interestingly, if the parent directory does not have a snapshot, the tests will not fail. The following test will fail intermittently: {code:java} public void testDeleteParentDirWithSnapShot() throws Exception { Path path = new Path("/test"); fs.mkdirs(path); fs.allowSnapshot(path); Path file = new Path("/test/test/test2"); FSDataOutputStream out = fs.create(file); for (int i = 0; i < 2; i++) { long count = 0; while (count < 1048576) { out.writeBytes("hell"); count += 4; } } ((DFSOutputStream) out.getWrappedStream()).hsync(EnumSet .of(SyncFlag.UPDATE_LENGTH)); DFSTestUtil.abortStream((DFSOutputStream) out.getWrappedStream()); Path file2 = new Path("/test/test/test3"); FSDataOutputStream out2 = fs.create(file2); for (int i = 0; i < 2; i++) { long count = 0; while (count < 1048576) { out2.writeBytes("hell"); count += 4; } } ((DFSOutputStream) out2.getWrappedStream()).hsync(EnumSet .of(SyncFlag.UPDATE_LENGTH)); DFSTestUtil.abortStream((DFSOutputStream) out2.getWrappedStream()); fs.createSnapshot(path, "s1"); // delete parent directory fs.delete(new Path("/test/test"), true); cluster.restartNameNode(); } {code} I am not sure if it's a test case issue, or something to do with snapshots. was: I found a number of {{TestOpenFilesWithSnapshot}} tests failed quite frequently. These tests ({{testParentDirWithUCFileDeleteWithSnapshot}}, {{testOpenFilesWithRename}}, {{testWithCheckpoint}}) are unable to reconnect to the namenode after restart. It looks like the reconnection failed due to an EOFException between data node and the name node. {noformat} FAILED: org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot.testParentDirWithUCFileDeleteWithSnapShot Error Message: Timed out waiting for Mini HDFS Cluster to start Stack Trace: java.io.IOException: Timed out waiting for Mini HDFS Cluster to start at org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1345) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:2024) at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1985) at org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot.testParentDirWithUCFileDeleteWithSnapShot(TestOpenFilesWithSnapshot.java:82) {noformat} It appears that these three tests all call doWriteAndAbort(), which creates files and then abort, and then set the parent directory with a snapshot, and then delete the parent directory. Interestingly, if the parent directory does not have a snapshot, the tests will not fail. The following test will fail intermittently: {code:java} public void testDeleteParentDirWithSnapShot() throws Exception { Path path = new Path("/test"); fs.mkdirs(path); fs.allowSnapshot(path); Path file = new Path("/test/test/test2"); FSDataOutputStream out = fs.create(file); for (int i = 0; i < 2; i++) { long count = 0; while (count < 1048576) { out.writeBytes("hell"); count += 4; } } ((DFSOutputStream) out.getWrappedStream()).hsync(EnumSet .of(SyncFlag.UPDATE_LENGTH)); DFSTestUtil.abortStream((DFSOutputStream) out.getWrappedStream()); Path file2 = new Path("/test/test/test3"); FSDataOutputStream out2 = fs.create(file2); for (int i = 0; i < 2; i++) { long count = 0; while (count < 1048576) { out2.writeBytes("hell"); count += 4; } } ((DFSOutputStream) out2.getWrappedStream()).hsync(EnumSet .of(SyncFlag.UPDATE_LENGTH)); DFSTestUtil.abortStream((DFSOutputStream) out2.getWrappedStream()); fs.createSnapshot(path, "s1"); // delete parent directory fs.delete(new Path("/test/test"), true); cluster.restartNameNode(); } {code} I am not sure if it's a test case issue, or something to do with snapshots. > Restarting namenode after deleting a directory with snapshot will fail > ---------------------------------------------------------------------- > > Key: HDFS-9631 > URL: https://issues.apache.org/jira/browse/HDFS-9631 > Project: Hadoop HDFS > Issue Type: Bug > Affects Versions: 3.0.0 > Reporter: Wei-Chiu Chuang > Assignee: Wei-Chiu Chuang > > I found a number of {{TestOpenFilesWithSnapshot}} tests failed quite > frequently. > These tests ({{testParentDirWithUCFileDeleteWithSnapshot}}, > {{testOpenFilesWithRename}}, {{testWithCheckpoint}}) are unable to reconnect > to the namenode after restart. It looks like the reconnection failed due to > an EOFException between data node and the name node. > {noformat} > FAILED: > org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot.testParentDirWithUCFileDeleteWithSnapShot > Error Message: > Timed out waiting for Mini HDFS Cluster to start > Stack Trace: > java.io.IOException: Timed out waiting for Mini HDFS Cluster to start > at > org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1345) > at > org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:2024) > at > org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1985) > at > org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot.testParentDirWithUCFileDeleteWithSnapShot(TestOpenFilesWithSnapshot.java:82) > {noformat} > It appears that these three tests all call {{doWriteAndAbort()}}, which > creates files and then abort, and then set the parent directory with a > snapshot, and then delete the parent directory. > Interestingly, if the parent directory does not have a snapshot, the tests > will not fail. > The following test will fail intermittently: > {code:java} > public void testDeleteParentDirWithSnapShot() throws Exception { > Path path = new Path("/test"); > fs.mkdirs(path); > fs.allowSnapshot(path); > Path file = new Path("/test/test/test2"); > FSDataOutputStream out = fs.create(file); > for (int i = 0; i < 2; i++) { > long count = 0; > while (count < 1048576) { > out.writeBytes("hell"); > count += 4; > } > } > ((DFSOutputStream) out.getWrappedStream()).hsync(EnumSet > .of(SyncFlag.UPDATE_LENGTH)); > DFSTestUtil.abortStream((DFSOutputStream) out.getWrappedStream()); > Path file2 = new Path("/test/test/test3"); > FSDataOutputStream out2 = fs.create(file2); > for (int i = 0; i < 2; i++) { > long count = 0; > while (count < 1048576) { > out2.writeBytes("hell"); > count += 4; > } > } > ((DFSOutputStream) out2.getWrappedStream()).hsync(EnumSet > .of(SyncFlag.UPDATE_LENGTH)); > DFSTestUtil.abortStream((DFSOutputStream) out2.getWrappedStream()); > fs.createSnapshot(path, "s1"); > // delete parent directory > fs.delete(new Path("/test/test"), true); > cluster.restartNameNode(); > } > {code} > I am not sure if it's a test case issue, or something to do with snapshots. -- This message was sent by Atlassian JIRA (v6.3.4#6332)