[jira] [Commented] (HDFS-15144) TestBlockStatsMXBean#testStorageTypeStatsWhenStorageFailed is incorrect
[ https://issues.apache.org/jira/browse/HDFS-15144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024741#comment-17024741 ] Íñigo Goiri commented on HDFS-15144: I have to say this is a very anti-intuitive piece of code. I thinks it might be worth clarifying this a little; either adding comments or making the restart method not move nodes back into a list in a different order. > TestBlockStatsMXBean#testStorageTypeStatsWhenStorageFailed is incorrect > --- > > Key: HDFS-15144 > URL: https://issues.apache.org/jira/browse/HDFS-15144 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: 2020-01-24-09-30-TestBlockStatsMXBean-output.txt, > HDFS-15144.001.patch > > > {{TestBlockStatsMXBean#testStorageTypeStatsWhenStorageFailed}} loops three > times to restart Datanodes. However, the code restart the DN-0 three times. > As a result, the JUnit does not really execute the scenario it was supposed > to. > {code:java} > DataNodeTestUtils.restoreDataDirFromFailure(dn1ArcVol1); > DataNodeTestUtils.restoreDataDirFromFailure(dn2ArcVol1); > DataNodeTestUtils.restoreDataDirFromFailure(dn3ArcVol1); > for (int i = 0; i < 3; i++) { > cluster.restartDataNode(0, true); > } > // wait for heartbeat > Thread.sleep(6000); > storageTypeStatsMap = cluster.getNamesystem().getBlockManager() > .getStorageTypeStats(); > storageTypeStats = storageTypeStatsMap.get(StorageType.RAM_DISK); > assertEquals(6, storageTypeStats.getNodesInService()); > {code} > When I changed the loop inner block to {{cluster.restartDataNode(i, true)}}, > the test did not pass with the stack trace below. I suspect that one of the > datanodes does not start properly after calling restart. > {code:bash} > [INFO] Running > org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean > [ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: > 28.805 s <<< FAILURE! - in > org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean > [ERROR] > testStorageTypeStatsWhenStorageFailed(org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean) > Time elapsed: 17.682 s <<< FAILURE! > java.lang.AssertionError: expected:<6> but was:<5> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at org.junit.Assert.assertEquals(Assert.java:631) > at > org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean.testStorageTypeStatsWhenStorageFailed(TestBlockStatsMXBean.java:213) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15144) TestBlockStatsMXBean#testStorageTypeStatsWhenStorageFailed is incorrect
[ https://issues.apache.org/jira/browse/HDFS-15144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024608#comment-17024608 ] Ahmed Hussein commented on HDFS-15144: -- [~ayushtkn], I think I was wrong in my first evaluation. MiniDFSCluster restart the node by removing it from a list, then adding it back. That's why I was confused that the index is not updated inside the loop. My bad. I will close the Jira. > TestBlockStatsMXBean#testStorageTypeStatsWhenStorageFailed is incorrect > --- > > Key: HDFS-15144 > URL: https://issues.apache.org/jira/browse/HDFS-15144 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: 2020-01-24-09-30-TestBlockStatsMXBean-output.txt, > HDFS-15144.001.patch > > > {{TestBlockStatsMXBean#testStorageTypeStatsWhenStorageFailed}} loops three > times to restart Datanodes. However, the code restart the DN-0 three times. > As a result, the JUnit does not really execute the scenario it was supposed > to. > {code:java} > DataNodeTestUtils.restoreDataDirFromFailure(dn1ArcVol1); > DataNodeTestUtils.restoreDataDirFromFailure(dn2ArcVol1); > DataNodeTestUtils.restoreDataDirFromFailure(dn3ArcVol1); > for (int i = 0; i < 3; i++) { > cluster.restartDataNode(0, true); > } > // wait for heartbeat > Thread.sleep(6000); > storageTypeStatsMap = cluster.getNamesystem().getBlockManager() > .getStorageTypeStats(); > storageTypeStats = storageTypeStatsMap.get(StorageType.RAM_DISK); > assertEquals(6, storageTypeStats.getNodesInService()); > {code} > When I changed the loop inner block to {{cluster.restartDataNode(i, true)}}, > the test did not pass with the stack trace below. I suspect that one of the > datanodes does not start properly after calling restart. > {code:bash} > [INFO] Running > org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean > [ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: > 28.805 s <<< FAILURE! - in > org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean > [ERROR] > testStorageTypeStatsWhenStorageFailed(org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean) > Time elapsed: 17.682 s <<< FAILURE! > java.lang.AssertionError: expected:<6> but was:<5> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at org.junit.Assert.assertEquals(Assert.java:631) > at > org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean.testStorageTypeStatsWhenStorageFailed(TestBlockStatsMXBean.java:213) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15144) TestBlockStatsMXBean#testStorageTypeStatsWhenStorageFailed is incorrect
[ https://issues.apache.org/jira/browse/HDFS-15144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024413#comment-17024413 ] Ahmed Hussein commented on HDFS-15144: -- [~ayushtkn], I think there is a bug in {{MiniDFSCluster}} that fails to restart datanodes with injected failures. I saw that couple of times in other test case (i.e., HDFS-13179). I am going to file a separate Jira with more analysis and stack traces. > TestBlockStatsMXBean#testStorageTypeStatsWhenStorageFailed is incorrect > --- > > Key: HDFS-15144 > URL: https://issues.apache.org/jira/browse/HDFS-15144 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: 2020-01-24-09-30-TestBlockStatsMXBean-output.txt, > HDFS-15144.001.patch > > > {{TestBlockStatsMXBean#testStorageTypeStatsWhenStorageFailed}} loops three > times to restart Datanodes. However, the code restart the DN-0 three times. > As a result, the JUnit does not really execute the scenario it was supposed > to. > {code:java} > DataNodeTestUtils.restoreDataDirFromFailure(dn1ArcVol1); > DataNodeTestUtils.restoreDataDirFromFailure(dn2ArcVol1); > DataNodeTestUtils.restoreDataDirFromFailure(dn3ArcVol1); > for (int i = 0; i < 3; i++) { > cluster.restartDataNode(0, true); > } > // wait for heartbeat > Thread.sleep(6000); > storageTypeStatsMap = cluster.getNamesystem().getBlockManager() > .getStorageTypeStats(); > storageTypeStats = storageTypeStatsMap.get(StorageType.RAM_DISK); > assertEquals(6, storageTypeStats.getNodesInService()); > {code} > When I changed the loop inner block to {{cluster.restartDataNode(i, true)}}, > the test did not pass with the stack trace below. I suspect that one of the > datanodes does not start properly after calling restart. > {code:bash} > [INFO] Running > org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean > [ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: > 28.805 s <<< FAILURE! - in > org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean > [ERROR] > testStorageTypeStatsWhenStorageFailed(org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean) > Time elapsed: 17.682 s <<< FAILURE! > java.lang.AssertionError: expected:<6> but was:<5> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at org.junit.Assert.assertEquals(Assert.java:631) > at > org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean.testStorageTypeStatsWhenStorageFailed(TestBlockStatsMXBean.java:213) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15144) TestBlockStatsMXBean#testStorageTypeStatsWhenStorageFailed is incorrect
[ https://issues.apache.org/jira/browse/HDFS-15144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023419#comment-17023419 ] Ayush Saxena commented on HDFS-15144: - Can you help me understand. How reversing the order is fixing this issue? > TestBlockStatsMXBean#testStorageTypeStatsWhenStorageFailed is incorrect > --- > > Key: HDFS-15144 > URL: https://issues.apache.org/jira/browse/HDFS-15144 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: 2020-01-24-09-30-TestBlockStatsMXBean-output.txt, > HDFS-15144.001.patch > > > {{TestBlockStatsMXBean#testStorageTypeStatsWhenStorageFailed}} loops three > times to restart Datanodes. However, the code restart the DN-0 three times. > As a result, the JUnit does not really execute the scenario it was supposed > to. > {code:java} > DataNodeTestUtils.restoreDataDirFromFailure(dn1ArcVol1); > DataNodeTestUtils.restoreDataDirFromFailure(dn2ArcVol1); > DataNodeTestUtils.restoreDataDirFromFailure(dn3ArcVol1); > for (int i = 0; i < 3; i++) { > cluster.restartDataNode(0, true); > } > // wait for heartbeat > Thread.sleep(6000); > storageTypeStatsMap = cluster.getNamesystem().getBlockManager() > .getStorageTypeStats(); > storageTypeStats = storageTypeStatsMap.get(StorageType.RAM_DISK); > assertEquals(6, storageTypeStats.getNodesInService()); > {code} > When I changed the loop inner block to {{cluster.restartDataNode(i, true)}}, > the test did not pass with the stack trace below. I suspect that one of the > datanodes does not start properly after calling restart. > {code:bash} > [INFO] Running > org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean > [ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: > 28.805 s <<< FAILURE! - in > org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean > [ERROR] > testStorageTypeStatsWhenStorageFailed(org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean) > Time elapsed: 17.682 s <<< FAILURE! > java.lang.AssertionError: expected:<6> but was:<5> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at org.junit.Assert.assertEquals(Assert.java:631) > at > org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean.testStorageTypeStatsWhenStorageFailed(TestBlockStatsMXBean.java:213) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15144) TestBlockStatsMXBean#testStorageTypeStatsWhenStorageFailed is incorrect
[ https://issues.apache.org/jira/browse/HDFS-15144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023391#comment-17023391 ] Hadoop QA commented on HDFS-15144: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 28s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 27s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 1s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 49s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 36s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}154m 23s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestDeadNodeDetection | | | hadoop.hdfs.TestReconstructStripedFile | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 | | JIRA Issue | HDFS-15144 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12991805/HDFS-15144.001.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 636c9d93df2b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 839e607 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_232 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/28710/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/28710/testReport/ | | Max. process+thread count | 4355 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/28710/console | | Powered by | Apache Yetus 0.8.0
[jira] [Commented] (HDFS-15144) TestBlockStatsMXBean#testStorageTypeStatsWhenStorageFailed is incorrect
[ https://issues.apache.org/jira/browse/HDFS-15144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023341#comment-17023341 ] Ahmed Hussein commented on HDFS-15144: -- There was a problem when restarting the datanodes. One of the DN won't receive the restart and it will stay as inactive following the injected disk failure. I reversed the order by which Datanodes are restarted and this seems to fix the issue. > TestBlockStatsMXBean#testStorageTypeStatsWhenStorageFailed is incorrect > --- > > Key: HDFS-15144 > URL: https://issues.apache.org/jira/browse/HDFS-15144 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: 2020-01-24-09-30-TestBlockStatsMXBean-output.txt > > > {{TestBlockStatsMXBean#testStorageTypeStatsWhenStorageFailed}} loops three > times to restart Datanodes. However, the code restart the DN-0 three times. > As a result, the JUnit does not really execute the scenario it was supposed > to. > {code:java} > DataNodeTestUtils.restoreDataDirFromFailure(dn1ArcVol1); > DataNodeTestUtils.restoreDataDirFromFailure(dn2ArcVol1); > DataNodeTestUtils.restoreDataDirFromFailure(dn3ArcVol1); > for (int i = 0; i < 3; i++) { > cluster.restartDataNode(0, true); > } > // wait for heartbeat > Thread.sleep(6000); > storageTypeStatsMap = cluster.getNamesystem().getBlockManager() > .getStorageTypeStats(); > storageTypeStats = storageTypeStatsMap.get(StorageType.RAM_DISK); > assertEquals(6, storageTypeStats.getNodesInService()); > {code} > When I changed the loop inner block to {{cluster.restartDataNode(i, true)}}, > the test did not pass with the stack trace below. I suspect that one of the > datanodes does not start properly after calling restart. > {code:bash} > [INFO] Running > org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean > [ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: > 28.805 s <<< FAILURE! - in > org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean > [ERROR] > testStorageTypeStatsWhenStorageFailed(org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean) > Time elapsed: 17.682 s <<< FAILURE! > java.lang.AssertionError: expected:<6> but was:<5> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at org.junit.Assert.assertEquals(Assert.java:631) > at > org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean.testStorageTypeStatsWhenStorageFailed(TestBlockStatsMXBean.java:213) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org