[jira] [Commented] (HDFS-8960) DFS client says "no more good datanodes being available to try" on a single drive failure
[
https://issues.apache.org/jira/browse/HDFS-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15119793#comment-15119793
]
Ruslan Dautkhanov commented on HDFS-8960:
-
We seems getting the same problem on a Hive job too:
{quote}
Error: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Failed
to replace a bad datanode on the existing pipeline due to no more good
datanodes being available to try. (Nodes:
current=[DatanodeInfoWithStorage[10.20.32.60:1004,DS-1cc9c7cd-f1f9-4cad-b6e2-c9821d644033,DISK]],
original=[DatanodeInfoWithStorage[10.20.32.60:1004,DS-1cc9c7cd-f1f9-4cad-b6e2-c9821d644033,DISK]]).
The current failed datanode replacement policy is DEFAULT, and a client may
configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy'
in its configuration. at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:265) at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444) at
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by:
org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Failed
to replace a bad datanode on the existing pipeline due to no more good
datanodes being available to try. (Nodes:
current=[DatanodeInfoWithStorage[10.20.32.60:1004,DS-1cc9c7cd-f1f9-4cad-b6e2-c9821d644033,DISK]],
original=[DatanodeInfoWithStorage[10.20.32.60:1004,DS-1cc9c7cd-f1f9-4cad-b6e2-c9821d644033,DISK]]).
The current failed datanode replacement policy is DEFAULT, and a client may
configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy'
in its configuration. at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:729)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815) at
org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1047)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.flushHashTable(GroupByOperator.java:1015)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:833)
at
{quote}
> DFS client says "no more good datanodes being available to try" on a single
> drive failure
> -
>
> Key: HDFS-8960
> URL: https://issues.apache.org/jira/browse/HDFS-8960
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.7.1
> Environment: openjdk version "1.8.0_45-internal"
> OpenJDK Runtime Environment (build 1.8.0_45-internal-b14)
> OpenJDK 64-Bit Server VM (build 25.45-b02, mixed mode)
>Reporter: Benoit Sigoure
> Attachments: blk_1073817519_77099.log, r12s13-datanode.log,
> r12s16-datanode.log
>
>
> Since we upgraded to 2.7.1 we regularly see single-drive failures cause
> widespread problems at the HBase level (with the default 3x replication
> target).
> Here's an example. This HBase RegionServer is r12s16 (172.24.32.16) and is
> writing its WAL to [172.24.32.16:10110, 172.24.32.8:10110,
> 172.24.32.13:10110] as can be seen by the following occasional messages:
> {code}
> 2015-08-23 06:28:40,272 INFO [sync.3] wal.FSHLog: Slow sync cost: 123 ms,
> current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110]
> {code}
> A bit later, the second node in the pipeline above is going to experience an
> HDD failure.
> {code}
> 2015-08-23 07:21:58,720 WARN [DataStreamer for file
> /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998
> block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099]
> hdfs.DFSClient: Error Recovery for block
> BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline
> 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.8:10110: bad datanode
> 172.24.32.8:10110
> {code}
> And then HBase will go like "omg I can't write to my WAL, let me commit
> suicide".
> {code}
> 2015-08-23 07:22:26,060 FATAL
> [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1]
> wal.FSHLog: Could not append. Requesting close of wal
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[172.24.32.16:10110, 172.24.32.13:10110],
> original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed
> datanode replacement policy is DEFAULT, and a client may co
[jira] [Commented] (HDFS-8960) DFS client says "no more good datanodes being available to try" on a single drive failure
[
https://issues.apache.org/jira/browse/HDFS-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14733444#comment-14733444
]
Hudson commented on HDFS-8960:
--
FAILURE: Integrated in HBase-1.3-IT #136 (See
[https://builds.apache.org/job/HBase-1.3-IT/136/])
HBASE-14317 Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL (stack: rev
bbafb47f7271449d46b46569ca9f0cb227b44c6e)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiVersionConcurrencyControl.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogReader.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/DamagedWALException.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSWALEntry.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MultiVersionConcurrencyControl.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SyncFuture.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALKey.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiVersionConcurrencyControlBasic.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFailedAppendAndSync.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWALLockup.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> DFS client says "no more good datanodes being available to try" on a single
> drive failure
> -
>
> Key: HDFS-8960
> URL: https://issues.apache.org/jira/browse/HDFS-8960
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.7.1
> Environment: openjdk version "1.8.0_45-internal"
> OpenJDK Runtime Environment (build 1.8.0_45-internal-b14)
> OpenJDK 64-Bit Server VM (build 25.45-b02, mixed mode)
>Reporter: Benoit Sigoure
> Attachments: blk_1073817519_77099.log, r12s13-datanode.log,
> r12s16-datanode.log
>
>
> Since we upgraded to 2.7.1 we regularly see single-drive failures cause
> widespread problems at the HBase level (with the default 3x replication
> target).
> Here's an example. This HBase RegionServer is r12s16 (172.24.32.16) and is
> writing its WAL to [172.24.32.16:10110, 172.24.32.8:10110,
> 172.24.32.13:10110] as can be seen by the following occasional messages:
> {code}
> 2015-08-23 06:28:40,272 INFO [sync.3] wal.FSHLog: Slow sync cost: 123 ms,
> current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110]
> {code}
> A bit later, the second node in the pipeline above is going to experience an
> HDD failure.
> {code}
> 2015-08-23 07:21:58,720 WARN [DataStreamer for file
> /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998
> block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099]
> hdfs.DFSClient: Error Recovery for block
> BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline
> 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.8:10110: bad datanode
> 172.24.32.8:10110
> {code}
> And then HBase will go like "omg I can't write to my WAL, let me commit
> suicide".
> {code}
> 2015-08-23 07:22:26,060 FATAL
> [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1]
> wal.FSHLog: Could not append. Requesting close of wal
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[172.24.32.16:10110, 172.24.32.13:10110],
> original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed
> datanode replacement policy is DEFAULT, and a client may configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)
> at
> org.apach
[jira] [Commented] (HDFS-8960) DFS client says "no more good datanodes being available to try" on a single drive failure
[
https://issues.apache.org/jira/browse/HDFS-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14733402#comment-14733402
]
Hudson commented on HDFS-8960:
--
SUCCESS: Integrated in HBase-1.2-IT #130 (See
[https://builds.apache.org/job/HBase-1.2-IT/130/])
HBASE-14317 Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL (stack: rev
990e3698a7ca7e95894150a2905ba4271eb371e9)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogReader.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFailedAppendAndSync.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MultiVersionConcurrencyControl.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWALLockup.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SyncFuture.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiVersionConcurrencyControl.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALKey.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSWALEntry.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/DamagedWALException.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiVersionConcurrencyControlBasic.java
> DFS client says "no more good datanodes being available to try" on a single
> drive failure
> -
>
> Key: HDFS-8960
> URL: https://issues.apache.org/jira/browse/HDFS-8960
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.7.1
> Environment: openjdk version "1.8.0_45-internal"
> OpenJDK Runtime Environment (build 1.8.0_45-internal-b14)
> OpenJDK 64-Bit Server VM (build 25.45-b02, mixed mode)
>Reporter: Benoit Sigoure
> Attachments: blk_1073817519_77099.log, r12s13-datanode.log,
> r12s16-datanode.log
>
>
> Since we upgraded to 2.7.1 we regularly see single-drive failures cause
> widespread problems at the HBase level (with the default 3x replication
> target).
> Here's an example. This HBase RegionServer is r12s16 (172.24.32.16) and is
> writing its WAL to [172.24.32.16:10110, 172.24.32.8:10110,
> 172.24.32.13:10110] as can be seen by the following occasional messages:
> {code}
> 2015-08-23 06:28:40,272 INFO [sync.3] wal.FSHLog: Slow sync cost: 123 ms,
> current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110]
> {code}
> A bit later, the second node in the pipeline above is going to experience an
> HDD failure.
> {code}
> 2015-08-23 07:21:58,720 WARN [DataStreamer for file
> /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998
> block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099]
> hdfs.DFSClient: Error Recovery for block
> BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline
> 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.8:10110: bad datanode
> 172.24.32.8:10110
> {code}
> And then HBase will go like "omg I can't write to my WAL, let me commit
> suicide".
> {code}
> 2015-08-23 07:22:26,060 FATAL
> [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1]
> wal.FSHLog: Could not append. Requesting close of wal
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[172.24.32.16:10110, 172.24.32.13:10110],
> original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed
> datanode replacement policy is DEFAULT, and a client may configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)
> at
> org.apach
[jira] [Commented] (HDFS-8960) DFS client says "no more good datanodes being available to try" on a single drive failure
[
https://issues.apache.org/jira/browse/HDFS-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14733356#comment-14733356
]
Hudson commented on HDFS-8960:
--
FAILURE: Integrated in HBase-1.2 #154 (See
[https://builds.apache.org/job/HBase-1.2/154/])
HBASE-14317 Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL (stack: rev
990e3698a7ca7e95894150a2905ba4271eb371e9)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSWALEntry.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/DamagedWALException.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiVersionConcurrencyControl.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MultiVersionConcurrencyControl.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SyncFuture.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogReader.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALKey.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFailedAppendAndSync.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiVersionConcurrencyControlBasic.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWALLockup.java
> DFS client says "no more good datanodes being available to try" on a single
> drive failure
> -
>
> Key: HDFS-8960
> URL: https://issues.apache.org/jira/browse/HDFS-8960
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.7.1
> Environment: openjdk version "1.8.0_45-internal"
> OpenJDK Runtime Environment (build 1.8.0_45-internal-b14)
> OpenJDK 64-Bit Server VM (build 25.45-b02, mixed mode)
>Reporter: Benoit Sigoure
> Attachments: blk_1073817519_77099.log, r12s13-datanode.log,
> r12s16-datanode.log
>
>
> Since we upgraded to 2.7.1 we regularly see single-drive failures cause
> widespread problems at the HBase level (with the default 3x replication
> target).
> Here's an example. This HBase RegionServer is r12s16 (172.24.32.16) and is
> writing its WAL to [172.24.32.16:10110, 172.24.32.8:10110,
> 172.24.32.13:10110] as can be seen by the following occasional messages:
> {code}
> 2015-08-23 06:28:40,272 INFO [sync.3] wal.FSHLog: Slow sync cost: 123 ms,
> current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110]
> {code}
> A bit later, the second node in the pipeline above is going to experience an
> HDD failure.
> {code}
> 2015-08-23 07:21:58,720 WARN [DataStreamer for file
> /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998
> block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099]
> hdfs.DFSClient: Error Recovery for block
> BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline
> 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.8:10110: bad datanode
> 172.24.32.8:10110
> {code}
> And then HBase will go like "omg I can't write to my WAL, let me commit
> suicide".
> {code}
> 2015-08-23 07:22:26,060 FATAL
> [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1]
> wal.FSHLog: Could not append. Requesting close of wal
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[172.24.32.16:10110, 172.24.32.13:10110],
> original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed
> datanode replacement policy is DEFAULT, and a client may configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)
> at
> org.apache.hado
[jira] [Commented] (HDFS-8960) DFS client says "no more good datanodes being available to try" on a single drive failure
[
https://issues.apache.org/jira/browse/HDFS-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1473#comment-1473
]
Hudson commented on HDFS-8960:
--
FAILURE: Integrated in HBase-1.3 #152 (See
[https://builds.apache.org/job/HBase-1.3/152/])
HBASE-14317 Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL (stack: rev
bbafb47f7271449d46b46569ca9f0cb227b44c6e)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/DamagedWALException.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFailedAppendAndSync.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiVersionConcurrencyControl.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiVersionConcurrencyControlBasic.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWALLockup.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSWALEntry.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALKey.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MultiVersionConcurrencyControl.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogReader.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SyncFuture.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> DFS client says "no more good datanodes being available to try" on a single
> drive failure
> -
>
> Key: HDFS-8960
> URL: https://issues.apache.org/jira/browse/HDFS-8960
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.7.1
> Environment: openjdk version "1.8.0_45-internal"
> OpenJDK Runtime Environment (build 1.8.0_45-internal-b14)
> OpenJDK 64-Bit Server VM (build 25.45-b02, mixed mode)
>Reporter: Benoit Sigoure
> Attachments: blk_1073817519_77099.log, r12s13-datanode.log,
> r12s16-datanode.log
>
>
> Since we upgraded to 2.7.1 we regularly see single-drive failures cause
> widespread problems at the HBase level (with the default 3x replication
> target).
> Here's an example. This HBase RegionServer is r12s16 (172.24.32.16) and is
> writing its WAL to [172.24.32.16:10110, 172.24.32.8:10110,
> 172.24.32.13:10110] as can be seen by the following occasional messages:
> {code}
> 2015-08-23 06:28:40,272 INFO [sync.3] wal.FSHLog: Slow sync cost: 123 ms,
> current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110]
> {code}
> A bit later, the second node in the pipeline above is going to experience an
> HDD failure.
> {code}
> 2015-08-23 07:21:58,720 WARN [DataStreamer for file
> /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998
> block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099]
> hdfs.DFSClient: Error Recovery for block
> BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline
> 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.8:10110: bad datanode
> 172.24.32.8:10110
> {code}
> And then HBase will go like "omg I can't write to my WAL, let me commit
> suicide".
> {code}
> 2015-08-23 07:22:26,060 FATAL
> [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1]
> wal.FSHLog: Could not append. Requesting close of wal
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[172.24.32.16:10110, 172.24.32.13:10110],
> original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed
> datanode replacement policy is DEFAULT, and a client may configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)
> at
> org.apache.hado
[jira] [Commented] (HDFS-8960) DFS client says "no more good datanodes being available to try" on a single drive failure
[
https://issues.apache.org/jira/browse/HDFS-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14730699#comment-14730699
]
Hudson commented on HDFS-8960:
--
FAILURE: Integrated in HBase-TRUNK #6780 (See
[https://builds.apache.org/job/HBase-TRUNK/6780/])
HBASE-14317 Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL; addendum2 --
found a fix testing the branch-1 patch (stack: rev
ec4d719f1927576d3de321c2e380e4c4acd099db)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
> DFS client says "no more good datanodes being available to try" on a single
> drive failure
> -
>
> Key: HDFS-8960
> URL: https://issues.apache.org/jira/browse/HDFS-8960
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.7.1
> Environment: openjdk version "1.8.0_45-internal"
> OpenJDK Runtime Environment (build 1.8.0_45-internal-b14)
> OpenJDK 64-Bit Server VM (build 25.45-b02, mixed mode)
>Reporter: Benoit Sigoure
> Attachments: blk_1073817519_77099.log, r12s13-datanode.log,
> r12s16-datanode.log
>
>
> Since we upgraded to 2.7.1 we regularly see single-drive failures cause
> widespread problems at the HBase level (with the default 3x replication
> target).
> Here's an example. This HBase RegionServer is r12s16 (172.24.32.16) and is
> writing its WAL to [172.24.32.16:10110, 172.24.32.8:10110,
> 172.24.32.13:10110] as can be seen by the following occasional messages:
> {code}
> 2015-08-23 06:28:40,272 INFO [sync.3] wal.FSHLog: Slow sync cost: 123 ms,
> current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110]
> {code}
> A bit later, the second node in the pipeline above is going to experience an
> HDD failure.
> {code}
> 2015-08-23 07:21:58,720 WARN [DataStreamer for file
> /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998
> block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099]
> hdfs.DFSClient: Error Recovery for block
> BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline
> 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.8:10110: bad datanode
> 172.24.32.8:10110
> {code}
> And then HBase will go like "omg I can't write to my WAL, let me commit
> suicide".
> {code}
> 2015-08-23 07:22:26,060 FATAL
> [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1]
> wal.FSHLog: Could not append. Requesting close of wal
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[172.24.32.16:10110, 172.24.32.13:10110],
> original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed
> datanode replacement policy is DEFAULT, and a client may configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)
> {code}
> Whereas this should be mostly a non-event as the DFS client should just drop
> the bad replica from the write pipeline.
> This is a small cluster but has 16 DNs so the failed DN in the pipeline
> should be easily replaced. I didn't set
> {{dfs.client.block.write.replace-datanode-on-failure.policy}} (so it's still
> {{DEFAULT}}) and didn't set
> {{dfs.client.block.write.replace-datanode-on-failure.enable}} (so it's still
> {{true}}).
> I don't see anything noteworthy in the NN log around the time of the failure,
> it just seems like the DFS client gave up or threw an exception back to HBase
> that it wasn't throwing before or something else, and that made this single
> drive failure lethal.
> We've occasionally be "unlucky" enough to have a single-drive failure cause
> multiple RegionServers to commit suicide because they had their WALs on that
> drive.
> We upgraded from 2.7.0 about a month ago, and I'm not sure whether we were
> seeing this with 2.7 or not – prior to that we were running in a quite
> different environment, but this is a fairly new deployment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8960) DFS client says "no more good datanodes being available to try" on a single drive failure
[
https://issues.apache.org/jira/browse/HDFS-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14730359#comment-14730359
]
Hudson commented on HDFS-8960:
--
FAILURE: Integrated in HBase-TRUNK #6778 (See
[https://builds.apache.org/job/HBase-TRUNK/6778/])
HBASE-14317 Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL (stack: rev
661faf6fe0833726d7ce7ad44a829eba3f8e3e45)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SyncFuture.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWALLockup.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogReader.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MultiVersionConcurrencyControl.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSWALEntry.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiVersionConcurrencyControlBasic.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiVersionConcurrencyControl.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALKey.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/DamagedWALException.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFailedAppendAndSync.java
HBASE-14317 Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL; addendum
(stack: rev 54717a6314ef6673f7607091e5f77321c202d49f)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
> DFS client says "no more good datanodes being available to try" on a single
> drive failure
> -
>
> Key: HDFS-8960
> URL: https://issues.apache.org/jira/browse/HDFS-8960
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.7.1
> Environment: openjdk version "1.8.0_45-internal"
> OpenJDK Runtime Environment (build 1.8.0_45-internal-b14)
> OpenJDK 64-Bit Server VM (build 25.45-b02, mixed mode)
>Reporter: Benoit Sigoure
> Attachments: blk_1073817519_77099.log, r12s13-datanode.log,
> r12s16-datanode.log
>
>
> Since we upgraded to 2.7.1 we regularly see single-drive failures cause
> widespread problems at the HBase level (with the default 3x replication
> target).
> Here's an example. This HBase RegionServer is r12s16 (172.24.32.16) and is
> writing its WAL to [172.24.32.16:10110, 172.24.32.8:10110,
> 172.24.32.13:10110] as can be seen by the following occasional messages:
> {code}
> 2015-08-23 06:28:40,272 INFO [sync.3] wal.FSHLog: Slow sync cost: 123 ms,
> current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110]
> {code}
> A bit later, the second node in the pipeline above is going to experience an
> HDD failure.
> {code}
> 2015-08-23 07:21:58,720 WARN [DataStreamer for file
> /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998
> block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099]
> hdfs.DFSClient: Error Recovery for block
> BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline
> 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.8:10110: bad datanode
> 172.24.32.8:10110
> {code}
> And then HBase will go like "omg I can't write to my WAL, let me commit
> suicide".
> {code}
> 2015-08-23 07:22:26,060 FATAL
> [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1]
> wal.FSHLog: Could not append. Requesting close of wal
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[172.24.32.16:10110, 172.24.32.13:10110],
> original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed
> datanode replacement policy is DEFAULT, and a client may configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hd
[jira] [Commented] (HDFS-8960) DFS client says "no more good datanodes being available to try" on a single drive failure
[
https://issues.apache.org/jira/browse/HDFS-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14718146#comment-14718146
]
Yongjun Zhang commented on HDFS-8960:
-
Yes, it's trying to do pipeline recovery, see below:
{code}
[yzhang@localhost Downloads]$ grep -B 3 blk_1073817519 r12s16-datanode.log |
grep firstbadlink
15/08/23 07:21:49 INFO datanode.DataNode: Datanode 2 got response for connect
ack from downstream datanode with firstbadlink as 172.24.32.5:10110
15/08/23 07:21:49 INFO datanode.DataNode: Datanode 2 forwarding connect ack to
upstream firstbadlink is 172.24.32.5:10110
15/08/23 07:21:52 INFO datanode.DataNode: Datanode 2 got response for connect
ack from downstream datanode with firstbadlink as 172.24.32.1:10110
15/08/23 07:21:52 INFO datanode.DataNode: Datanode 2 forwarding connect ack to
upstream firstbadlink is 172.24.32.1:10110
15/08/23 07:21:55 INFO datanode.DataNode: Datanode 2 got response for connect
ack from downstream datanode with firstbadlink as 172.24.32.6:10110
15/08/23 07:21:55 INFO datanode.DataNode: Datanode 2 forwarding connect ack to
upstream firstbadlink is 172.24.32.6:10110
15/08/23 07:21:58 INFO datanode.DataNode: Datanode 2 got response for connect
ack from downstream datanode with firstbadlink as 172.24.32.8:10110
15/08/23 07:21:58 INFO datanode.DataNode: Datanode 2 forwarding connect ack to
upstream firstbadlink is 172.24.32.8:10110
15/08/23 07:22:01 INFO datanode.DataNode: Datanode 2 got response for connect
ack from downstream datanode with firstbadlink as 172.24.32.14:10110
15/08/23 07:22:01 INFO datanode.DataNode: Datanode 2 forwarding connect ack to
upstream firstbadlink is 172.24.32.14:10110
15/08/23 07:22:04 INFO datanode.DataNode: Datanode 2 got response for connect
ack from downstream datanode with firstbadlink as 172.24.32.2:10110
15/08/23 07:22:04 INFO datanode.DataNode: Datanode 2 forwarding connect ack to
upstream firstbadlink is 172.24.32.2:10110
15/08/23 07:22:07 INFO datanode.DataNode: Datanode 2 got response for connect
ack from downstream datanode with firstbadlink as 172.24.32.9:10110
15/08/23 07:22:07 INFO datanode.DataNode: Datanode 2 forwarding connect ack to
upstream firstbadlink is 172.24.32.9:10110
15/08/23 07:22:10 INFO datanode.DataNode: Datanode 2 got response for connect
ack from downstream datanode with firstbadlink as 172.24.32.3:10110
15/08/23 07:22:10 INFO datanode.DataNode: Datanode 2 forwarding connect ack to
upstream firstbadlink is 172.24.32.3:10110
15/08/23 07:22:13 INFO datanode.DataNode: Datanode 2 got response for connect
ack from downstream datanode with firstbadlink as 172.24.32.7:10110
15/08/23 07:22:13 INFO datanode.DataNode: Datanode 2 forwarding connect ack to
upstream firstbadlink is 172.24.32.7:10110
15/08/23 07:22:16 INFO datanode.DataNode: Datanode 2 got response for connect
ack from downstream datanode with firstbadlink as 172.24.32.10:10110
15/08/23 07:22:16 INFO datanode.DataNode: Datanode 2 forwarding connect ack to
upstream firstbadlink is 172.24.32.10:10110
15/08/23 07:22:19 INFO datanode.DataNode: Datanode 2 got response for connect
ack from downstream datanode with firstbadlink as 172.24.32.12:10110
15/08/23 07:22:19 INFO datanode.DataNode: Datanode 2 forwarding connect ack to
upstream firstbadlink is 172.24.32.12:10110
15/08/23 07:22:23 INFO datanode.DataNode: Datanode 2 got response for connect
ack from downstream datanode with firstbadlink as 172.24.32.11:10110
15/08/23 07:22:23 INFO datanode.DataNode: Datanode 2 forwarding connect ack to
upstream firstbadlink is 172.24.32.11:10110
15/08/23 07:22:26 INFO datanode.DataNode: Datanode 2 got response for connect
ack from downstream datanode with firstbadlink as 172.24.32.15:10110
15/08/23 07:22:26 INFO datanode.DataNode: Datanode 2 forwarding connect ack to
upstream firstbadlink is 172.24.32.15:10110
{code}
It happened you loaded r12s13 which is not one of the node in the grepped
message (per your report, r12s13 is the last node in the initial pipeline), and
r12s16 is the source node.
Would you please upload a few more DN logs?
Thanks.
> DFS client says "no more good datanodes being available to try" on a single
> drive failure
> -
>
> Key: HDFS-8960
> URL: https://issues.apache.org/jira/browse/HDFS-8960
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.7.1
> Environment: openjdk version "1.8.0_45-internal"
> OpenJDK Runtime Environment (build 1.8.0_45-internal-b14)
> OpenJDK 64-Bit Server VM (build 25.45-b02, mixed mode)
>Reporter: Benoit Sigoure
> Attachments: blk_1073817519_77099.log, r12s13-datanode.log,
> r12s16-datanode.log
>
>
> Since we upgraded to 2.7.1 we regularly see single-drive failures cause
>
[jira] [Commented] (HDFS-8960) DFS client says "no more good datanodes being available to try" on a single drive failure
[
https://issues.apache.org/jira/browse/HDFS-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14717242#comment-14717242
]
Benoit Sigoure commented on HDFS-8960:
--
No, these were lost in the HDD failure.
Do you see any sign of pipeline recovery even trying to happen here? I'm kinda
confused because none of the logs I've looked at show any indication of it
happening. Am I missing something?
> DFS client says "no more good datanodes being available to try" on a single
> drive failure
> -
>
> Key: HDFS-8960
> URL: https://issues.apache.org/jira/browse/HDFS-8960
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.7.1
> Environment: openjdk version "1.8.0_45-internal"
> OpenJDK Runtime Environment (build 1.8.0_45-internal-b14)
> OpenJDK 64-Bit Server VM (build 25.45-b02, mixed mode)
>Reporter: Benoit Sigoure
> Attachments: blk_1073817519_77099.log, r12s13-datanode.log,
> r12s16-datanode.log
>
>
> Since we upgraded to 2.7.1 we regularly see single-drive failures cause
> widespread problems at the HBase level (with the default 3x replication
> target).
> Here's an example. This HBase RegionServer is r12s16 (172.24.32.16) and is
> writing its WAL to [172.24.32.16:10110, 172.24.32.8:10110,
> 172.24.32.13:10110] as can be seen by the following occasional messages:
> {code}
> 2015-08-23 06:28:40,272 INFO [sync.3] wal.FSHLog: Slow sync cost: 123 ms,
> current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110]
> {code}
> A bit later, the second node in the pipeline above is going to experience an
> HDD failure.
> {code}
> 2015-08-23 07:21:58,720 WARN [DataStreamer for file
> /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998
> block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099]
> hdfs.DFSClient: Error Recovery for block
> BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline
> 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.8:10110: bad datanode
> 172.24.32.8:10110
> {code}
> And then HBase will go like "omg I can't write to my WAL, let me commit
> suicide".
> {code}
> 2015-08-23 07:22:26,060 FATAL
> [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1]
> wal.FSHLog: Could not append. Requesting close of wal
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[172.24.32.16:10110, 172.24.32.13:10110],
> original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed
> datanode replacement policy is DEFAULT, and a client may configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)
> {code}
> Whereas this should be mostly a non-event as the DFS client should just drop
> the bad replica from the write pipeline.
> This is a small cluster but has 16 DNs so the failed DN in the pipeline
> should be easily replaced. I didn't set
> {{dfs.client.block.write.replace-datanode-on-failure.policy}} (so it's still
> {{DEFAULT}}) and didn't set
> {{dfs.client.block.write.replace-datanode-on-failure.enable}} (so it's still
> {{true}}).
> I don't see anything noteworthy in the NN log around the time of the failure,
> it just seems like the DFS client gave up or threw an exception back to HBase
> that it wasn't throwing before or something else, and that made this single
> drive failure lethal.
> We've occasionally be "unlucky" enough to have a single-drive failure cause
> multiple RegionServers to commit suicide because they had their WALs on that
> drive.
> We upgraded from 2.7.0 about a month ago, and I'm not sure whether we were
> seeing this with 2.7 or not – prior to that we were running in a quite
> different environment, but this is a fairly new deployment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8960) DFS client says "no more good datanodes being available to try" on a single drive failure
[
https://issues.apache.org/jira/browse/HDFS-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14717137#comment-14717137
]
Yongjun Zhang commented on HDFS-8960:
-
Thanks, would you please provide the DN log on r12s8 too?
> DFS client says "no more good datanodes being available to try" on a single
> drive failure
> -
>
> Key: HDFS-8960
> URL: https://issues.apache.org/jira/browse/HDFS-8960
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.7.1
> Environment: openjdk version "1.8.0_45-internal"
> OpenJDK Runtime Environment (build 1.8.0_45-internal-b14)
> OpenJDK 64-Bit Server VM (build 25.45-b02, mixed mode)
>Reporter: Benoit Sigoure
> Attachments: blk_1073817519_77099.log, r12s13-datanode.log,
> r12s16-datanode.log
>
>
> Since we upgraded to 2.7.1 we regularly see single-drive failures cause
> widespread problems at the HBase level (with the default 3x replication
> target).
> Here's an example. This HBase RegionServer is r12s16 (172.24.32.16) and is
> writing its WAL to [172.24.32.16:10110, 172.24.32.8:10110,
> 172.24.32.13:10110] as can be seen by the following occasional messages:
> {code}
> 2015-08-23 06:28:40,272 INFO [sync.3] wal.FSHLog: Slow sync cost: 123 ms,
> current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110]
> {code}
> A bit later, the second node in the pipeline above is going to experience an
> HDD failure.
> {code}
> 2015-08-23 07:21:58,720 WARN [DataStreamer for file
> /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998
> block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099]
> hdfs.DFSClient: Error Recovery for block
> BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline
> 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.8:10110: bad datanode
> 172.24.32.8:10110
> {code}
> And then HBase will go like "omg I can't write to my WAL, let me commit
> suicide".
> {code}
> 2015-08-23 07:22:26,060 FATAL
> [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1]
> wal.FSHLog: Could not append. Requesting close of wal
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[172.24.32.16:10110, 172.24.32.13:10110],
> original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed
> datanode replacement policy is DEFAULT, and a client may configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)
> {code}
> Whereas this should be mostly a non-event as the DFS client should just drop
> the bad replica from the write pipeline.
> This is a small cluster but has 16 DNs so the failed DN in the pipeline
> should be easily replaced. I didn't set
> {{dfs.client.block.write.replace-datanode-on-failure.policy}} (so it's still
> {{DEFAULT}}) and didn't set
> {{dfs.client.block.write.replace-datanode-on-failure.enable}} (so it's still
> {{true}}).
> I don't see anything noteworthy in the NN log around the time of the failure,
> it just seems like the DFS client gave up or threw an exception back to HBase
> that it wasn't throwing before or something else, and that made this single
> drive failure lethal.
> We've occasionally be "unlucky" enough to have a single-drive failure cause
> multiple RegionServers to commit suicide because they had their WALs on that
> drive.
> We upgraded from 2.7.0 about a month ago, and I'm not sure whether we were
> seeing this with 2.7 or not – prior to that we were running in a quite
> different environment, but this is a fairly new deployment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8960) DFS client says "no more good datanodes being available to try" on a single drive failure
[
https://issues.apache.org/jira/browse/HDFS-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716215#comment-14716215
]
Benoit Sigoure commented on HDFS-8960:
--
It's kinda hard for me to collect jstack of the involved DNs "when the issue is
happening". I'm going to attach the log of the DNs from the two surviving
nodes, for the 1h time period where the problem occurred. The logs contain
some complaints that mostly relate to r12s8's struggle and subsequent death.
I'm also attaching the output of {{grep blk_1073817519_77099}} from the NN log,
which shows that the blog above was allocated and then more than 3 days later
it underwent "recovery" as I finally killed the stuck RS. The RS was stuck
forever waiting on HDFS (see HBASE-14317 for that other issue).
The drive that failed was not on the node where the RS was running. The RS was
on r12s16 (172.24.32.16) while the drive failed on r12s8 (172.24.32.8). As
shown in my initial report, this one is the middle node of the three node
pipeline.
While checksum errors are possible here, the symptom observed was that the
drive simply stopped accepting any I/O and was subsequently forcefully
unmounted by the kernel.
> DFS client says "no more good datanodes being available to try" on a single
> drive failure
> -
>
> Key: HDFS-8960
> URL: https://issues.apache.org/jira/browse/HDFS-8960
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.7.1
> Environment: openjdk version "1.8.0_45-internal"
> OpenJDK Runtime Environment (build 1.8.0_45-internal-b14)
> OpenJDK 64-Bit Server VM (build 25.45-b02, mixed mode)
>Reporter: Benoit Sigoure
> Attachments: blk_1073817519_77099.log, r12s13-datanode.log,
> r12s16-datanode.log
>
>
> Since we upgraded to 2.7.1 we regularly see single-drive failures cause
> widespread problems at the HBase level (with the default 3x replication
> target).
> Here's an example. This HBase RegionServer is r12s16 (172.24.32.16) and is
> writing its WAL to [172.24.32.16:10110, 172.24.32.8:10110,
> 172.24.32.13:10110] as can be seen by the following occasional messages:
> {code}
> 2015-08-23 06:28:40,272 INFO [sync.3] wal.FSHLog: Slow sync cost: 123 ms,
> current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110]
> {code}
> A bit later, the second node in the pipeline above is going to experience an
> HDD failure.
> {code}
> 2015-08-23 07:21:58,720 WARN [DataStreamer for file
> /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998
> block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099]
> hdfs.DFSClient: Error Recovery for block
> BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline
> 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.8:10110: bad datanode
> 172.24.32.8:10110
> {code}
> And then HBase will go like "omg I can't write to my WAL, let me commit
> suicide".
> {code}
> 2015-08-23 07:22:26,060 FATAL
> [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1]
> wal.FSHLog: Could not append. Requesting close of wal
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[172.24.32.16:10110, 172.24.32.13:10110],
> original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed
> datanode replacement policy is DEFAULT, and a client may configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)
> {code}
> Whereas this should be mostly a non-event as the DFS client should just drop
> the bad replica from the write pipeline.
> This is a small cluster but has 16 DNs so the failed DN in the pipeline
> should be easily replaced. I didn't set
> {{dfs.client.block.write.replace-datanode-on-failure.policy}} (so it's still
> {{DEFAULT}}) and didn't set
> {{dfs.client.block.write.replace-datanode-on-failure.enable}} (so it's still
> {{true}}).
> I don't see anything noteworthy in the NN log around the time of the failure,
> it just seems li
[jira] [Commented] (HDFS-8960) DFS client says "no more good datanodes being available to try" on a single drive failure
[
https://issues.apache.org/jira/browse/HDFS-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716193#comment-14716193
]
Yongjun Zhang commented on HDFS-8960:
-
is the single drive that failed on the node where RS is running? or is it on a
middle node of the three node pipeline? Maybe the recovery process somehow used
the replica on the failed drive as the source to copy to a different node
attempted thus always fail (like HDFS-6937)?
Thanks.
> DFS client says "no more good datanodes being available to try" on a single
> drive failure
> -
>
> Key: HDFS-8960
> URL: https://issues.apache.org/jira/browse/HDFS-8960
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.7.1
> Environment: openjdk version "1.8.0_45-internal"
> OpenJDK Runtime Environment (build 1.8.0_45-internal-b14)
> OpenJDK 64-Bit Server VM (build 25.45-b02, mixed mode)
>Reporter: Benoit Sigoure
>
> Since we upgraded to 2.7.1 we regularly see single-drive failures cause
> widespread problems at the HBase level (with the default 3x replication
> target).
> Here's an example. This HBase RegionServer is r12s16 (172.24.32.16) and is
> writing its WAL to [172.24.32.16:10110, 172.24.32.8:10110,
> 172.24.32.13:10110] as can be seen by the following occasional messages:
> {code}
> 2015-08-23 06:28:40,272 INFO [sync.3] wal.FSHLog: Slow sync cost: 123 ms,
> current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110]
> {code}
> A bit later, the second node in the pipeline above is going to experience an
> HDD failure.
> {code}
> 2015-08-23 07:21:58,720 WARN [DataStreamer for file
> /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998
> block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099]
> hdfs.DFSClient: Error Recovery for block
> BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline
> 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.8:10110: bad datanode
> 172.24.32.8:10110
> {code}
> And then HBase will go like "omg I can't write to my WAL, let me commit
> suicide".
> {code}
> 2015-08-23 07:22:26,060 FATAL
> [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1]
> wal.FSHLog: Could not append. Requesting close of wal
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[172.24.32.16:10110, 172.24.32.13:10110],
> original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed
> datanode replacement policy is DEFAULT, and a client may configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)
> {code}
> Whereas this should be mostly a non-event as the DFS client should just drop
> the bad replica from the write pipeline.
> This is a small cluster but has 16 DNs so the failed DN in the pipeline
> should be easily replaced. I didn't set
> {{dfs.client.block.write.replace-datanode-on-failure.policy}} (so it's still
> {{DEFAULT}}) and didn't set
> {{dfs.client.block.write.replace-datanode-on-failure.enable}} (so it's still
> {{true}}).
> I don't see anything noteworthy in the NN log around the time of the failure,
> it just seems like the DFS client gave up or threw an exception back to HBase
> that it wasn't throwing before or something else, and that made this single
> drive failure lethal.
> We've occasionally be "unlucky" enough to have a single-drive failure cause
> multiple RegionServers to commit suicide because they had their WALs on that
> drive.
> We upgraded from 2.7.0 about a month ago, and I'm not sure whether we were
> seeing this with 2.7 or not – prior to that we were running in a quite
> different environment, but this is a fairly new deployment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8960) DFS client says "no more good datanodes being available to try" on a single drive failure
[
https://issues.apache.org/jira/browse/HDFS-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716191#comment-14716191
]
Yongjun Zhang commented on HDFS-8960:
-
would you please collect the logs and jstack of some of the involved DNs
(claimed "bad" during the pipeline recovering) when the issue is happening?
wonder what kind of errors/warns are reported there.
thanks.
> DFS client says "no more good datanodes being available to try" on a single
> drive failure
> -
>
> Key: HDFS-8960
> URL: https://issues.apache.org/jira/browse/HDFS-8960
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.7.1
> Environment: openjdk version "1.8.0_45-internal"
> OpenJDK Runtime Environment (build 1.8.0_45-internal-b14)
> OpenJDK 64-Bit Server VM (build 25.45-b02, mixed mode)
>Reporter: Benoit Sigoure
>
> Since we upgraded to 2.7.1 we regularly see single-drive failures cause
> widespread problems at the HBase level (with the default 3x replication
> target).
> Here's an example. This HBase RegionServer is r12s16 (172.24.32.16) and is
> writing its WAL to [172.24.32.16:10110, 172.24.32.8:10110,
> 172.24.32.13:10110] as can be seen by the following occasional messages:
> {code}
> 2015-08-23 06:28:40,272 INFO [sync.3] wal.FSHLog: Slow sync cost: 123 ms,
> current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110]
> {code}
> A bit later, the second node in the pipeline above is going to experience an
> HDD failure.
> {code}
> 2015-08-23 07:21:58,720 WARN [DataStreamer for file
> /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998
> block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099]
> hdfs.DFSClient: Error Recovery for block
> BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline
> 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.8:10110: bad datanode
> 172.24.32.8:10110
> {code}
> And then HBase will go like "omg I can't write to my WAL, let me commit
> suicide".
> {code}
> 2015-08-23 07:22:26,060 FATAL
> [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1]
> wal.FSHLog: Could not append. Requesting close of wal
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[172.24.32.16:10110, 172.24.32.13:10110],
> original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed
> datanode replacement policy is DEFAULT, and a client may configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)
> {code}
> Whereas this should be mostly a non-event as the DFS client should just drop
> the bad replica from the write pipeline.
> This is a small cluster but has 16 DNs so the failed DN in the pipeline
> should be easily replaced. I didn't set
> {{dfs.client.block.write.replace-datanode-on-failure.policy}} (so it's still
> {{DEFAULT}}) and didn't set
> {{dfs.client.block.write.replace-datanode-on-failure.enable}} (so it's still
> {{true}}).
> I don't see anything noteworthy in the NN log around the time of the failure,
> it just seems like the DFS client gave up or threw an exception back to HBase
> that it wasn't throwing before or something else, and that made this single
> drive failure lethal.
> We've occasionally be "unlucky" enough to have a single-drive failure cause
> multiple RegionServers to commit suicide because they had their WALs on that
> drive.
> We upgraded from 2.7.0 about a month ago, and I'm not sure whether we were
> seeing this with 2.7 or not – prior to that we were running in a quite
> different environment, but this is a fairly new deployment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8960) DFS client says "no more good datanodes being available to try" on a single drive failure
[
https://issues.apache.org/jira/browse/HDFS-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715635#comment-14715635
]
Benoit Sigoure commented on HDFS-8960:
--
I looked at the GC logs and don't see anything unusual (both on the RS/DN/NN
sides).
> DFS client says "no more good datanodes being available to try" on a single
> drive failure
> -
>
> Key: HDFS-8960
> URL: https://issues.apache.org/jira/browse/HDFS-8960
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.7.1
> Environment: openjdk version "1.8.0_45-internal"
> OpenJDK Runtime Environment (build 1.8.0_45-internal-b14)
> OpenJDK 64-Bit Server VM (build 25.45-b02, mixed mode)
>Reporter: Benoit Sigoure
>
> Since we upgraded to 2.7.1 we regularly see single-drive failures cause
> widespread problems at the HBase level (with the default 3x replication
> target).
> Here's an example. This HBase RegionServer is r12s16 (172.24.32.16) and is
> writing its WAL to [172.24.32.16:10110, 172.24.32.8:10110,
> 172.24.32.13:10110] as can be seen by the following occasional messages:
> {code}
> 2015-08-23 06:28:40,272 INFO [sync.3] wal.FSHLog: Slow sync cost: 123 ms,
> current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110]
> {code}
> A bit later, the second node in the pipeline above is going to experience an
> HDD failure.
> {code}
> 2015-08-23 07:21:58,720 WARN [DataStreamer for file
> /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998
> block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099]
> hdfs.DFSClient: Error Recovery for block
> BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline
> 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.8:10110: bad datanode
> 172.24.32.8:10110
> {code}
> And then HBase will go like "omg I can't write to my WAL, let me commit
> suicide".
> {code}
> 2015-08-23 07:22:26,060 FATAL
> [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1]
> wal.FSHLog: Could not append. Requesting close of wal
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[172.24.32.16:10110, 172.24.32.13:10110],
> original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed
> datanode replacement policy is DEFAULT, and a client may configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)
> {code}
> Whereas this should be mostly a non-event as the DFS client should just drop
> the bad replica from the write pipeline.
> This is a small cluster but has 16 DNs so the failed DN in the pipeline
> should be easily replaced. I didn't set
> {{dfs.client.block.write.replace-datanode-on-failure.policy}} (so it's still
> {{DEFAULT}}) and didn't set
> {{dfs.client.block.write.replace-datanode-on-failure.enable}} (so it's still
> {{true}}).
> I don't see anything noteworthy in the NN log around the time of the failure,
> it just seems like the DFS client gave up or threw an exception back to HBase
> that it wasn't throwing before or something else, and that made this single
> drive failure lethal.
> We've occasionally be "unlucky" enough to have a single-drive failure cause
> multiple RegionServers to commit suicide because they had their WALs on that
> drive.
> We upgraded from 2.7.0 about a month ago, and I'm not sure whether we were
> seeing this with 2.7 or not – prior to that we were running in a quite
> different environment, but this is a fairly new deployment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8960) DFS client says "no more good datanodes being available to try" on a single drive failure
[
https://issues.apache.org/jira/browse/HDFS-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715622#comment-14715622
]
Yongjun Zhang commented on HDFS-8960:
-
Yes, agree that we need to look further why a single drive failure cause all to
fail. Any big GC observed on different DNs? thanks.
> DFS client says "no more good datanodes being available to try" on a single
> drive failure
> -
>
> Key: HDFS-8960
> URL: https://issues.apache.org/jira/browse/HDFS-8960
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.7.1
> Environment: openjdk version "1.8.0_45-internal"
> OpenJDK Runtime Environment (build 1.8.0_45-internal-b14)
> OpenJDK 64-Bit Server VM (build 25.45-b02, mixed mode)
>Reporter: Benoit Sigoure
>
> Since we upgraded to 2.7.1 we regularly see single-drive failures cause
> widespread problems at the HBase level (with the default 3x replication
> target).
> Here's an example. This HBase RegionServer is r12s16 (172.24.32.16) and is
> writing its WAL to [172.24.32.16:10110, 172.24.32.8:10110,
> 172.24.32.13:10110] as can be seen by the following occasional messages:
> {code}
> 2015-08-23 06:28:40,272 INFO [sync.3] wal.FSHLog: Slow sync cost: 123 ms,
> current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110]
> {code}
> A bit later, the second node in the pipeline above is going to experience an
> HDD failure.
> {code}
> 2015-08-23 07:21:58,720 WARN [DataStreamer for file
> /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998
> block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099]
> hdfs.DFSClient: Error Recovery for block
> BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline
> 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.8:10110: bad datanode
> 172.24.32.8:10110
> {code}
> And then HBase will go like "omg I can't write to my WAL, let me commit
> suicide".
> {code}
> 2015-08-23 07:22:26,060 FATAL
> [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1]
> wal.FSHLog: Could not append. Requesting close of wal
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[172.24.32.16:10110, 172.24.32.13:10110],
> original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed
> datanode replacement policy is DEFAULT, and a client may configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)
> {code}
> Whereas this should be mostly a non-event as the DFS client should just drop
> the bad replica from the write pipeline.
> This is a small cluster but has 16 DNs so the failed DN in the pipeline
> should be easily replaced. I didn't set
> {{dfs.client.block.write.replace-datanode-on-failure.policy}} (so it's still
> {{DEFAULT}}) and didn't set
> {{dfs.client.block.write.replace-datanode-on-failure.enable}} (so it's still
> {{true}}).
> I don't see anything noteworthy in the NN log around the time of the failure,
> it just seems like the DFS client gave up or threw an exception back to HBase
> that it wasn't throwing before or something else, and that made this single
> drive failure lethal.
> We've occasionally be "unlucky" enough to have a single-drive failure cause
> multiple RegionServers to commit suicide because they had their WALs on that
> drive.
> We upgraded from 2.7.0 about a month ago, and I'm not sure whether we were
> seeing this with 2.7 or not – prior to that we were running in a quite
> different environment, but this is a fairly new deployment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8960) DFS client says "no more good datanodes being available to try" on a single drive failure
[
https://issues.apache.org/jira/browse/HDFS-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715584#comment-14715584
]
Benoit Sigoure commented on HDFS-8960:
--
I'm not sure that's a good idea. Why would I want to let the RS succeed if it
didn't manage to write the three replicas? My understanding from reading that
post is that if the block can be written to only one replica, then the write
would be considered successful. Now my data is at risk because if the drive on
which that block is goes bad, I have data loss, and data loss in the WAL would
be pretty bad.
Why doesn't HDFS simply recover from the bad DN in the pipeline? The blog post
confirms my understanding of how this recovery should happen, yet here it seems
that a single bad drive screws things up to the point that the pipeline doesn't
recover at all. Why?
> DFS client says "no more good datanodes being available to try" on a single
> drive failure
> -
>
> Key: HDFS-8960
> URL: https://issues.apache.org/jira/browse/HDFS-8960
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.7.1
> Environment: openjdk version "1.8.0_45-internal"
> OpenJDK Runtime Environment (build 1.8.0_45-internal-b14)
> OpenJDK 64-Bit Server VM (build 25.45-b02, mixed mode)
>Reporter: Benoit Sigoure
>
> Since we upgraded to 2.7.1 we regularly see single-drive failures cause
> widespread problems at the HBase level (with the default 3x replication
> target).
> Here's an example. This HBase RegionServer is r12s16 (172.24.32.16) and is
> writing its WAL to [172.24.32.16:10110, 172.24.32.8:10110,
> 172.24.32.13:10110] as can be seen by the following occasional messages:
> {code}
> 2015-08-23 06:28:40,272 INFO [sync.3] wal.FSHLog: Slow sync cost: 123 ms,
> current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110]
> {code}
> A bit later, the second node in the pipeline above is going to experience an
> HDD failure.
> {code}
> 2015-08-23 07:21:58,720 WARN [DataStreamer for file
> /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998
> block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099]
> hdfs.DFSClient: Error Recovery for block
> BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline
> 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.8:10110: bad datanode
> 172.24.32.8:10110
> {code}
> And then HBase will go like "omg I can't write to my WAL, let me commit
> suicide".
> {code}
> 2015-08-23 07:22:26,060 FATAL
> [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1]
> wal.FSHLog: Could not append. Requesting close of wal
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[172.24.32.16:10110, 172.24.32.13:10110],
> original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed
> datanode replacement policy is DEFAULT, and a client may configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)
> {code}
> Whereas this should be mostly a non-event as the DFS client should just drop
> the bad replica from the write pipeline.
> This is a small cluster but has 16 DNs so the failed DN in the pipeline
> should be easily replaced. I didn't set
> {{dfs.client.block.write.replace-datanode-on-failure.policy}} (so it's still
> {{DEFAULT}}) and didn't set
> {{dfs.client.block.write.replace-datanode-on-failure.enable}} (so it's still
> {{true}}).
> I don't see anything noteworthy in the NN log around the time of the failure,
> it just seems like the DFS client gave up or threw an exception back to HBase
> that it wasn't throwing before or something else, and that made this single
> drive failure lethal.
> We've occasionally be "unlucky" enough to have a single-drive failure cause
> multiple RegionServers to commit suicide because they had their WALs on that
> drive.
> We upgraded from 2.7.0 about a month ago, and I'm not sure whether we were
> seeing this with 2.7 or not – pri
[jira] [Commented] (HDFS-8960) DFS client says "no more good datanodes being available to try" on a single drive failure
[
https://issues.apache.org/jira/browse/HDFS-8960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715564#comment-14715564
]
Yongjun Zhang commented on HDFS-8960:
-
HI [~tsuna],
Thanks for reporting the issue. While the failure reason is yet to be found out
with different DNs, would you please try to set config
{{dfs.client.block.write.replace-datanode-on-failure.best-effort}} to true in
hdfs-default.xml, restarting Hbase RS?
For some more details, see
http://blog.cloudera.com/blog/2015/03/understanding-hdfs-recovery-processes-part-2/
Thanks.
> DFS client says "no more good datanodes being available to try" on a single
> drive failure
> -
>
> Key: HDFS-8960
> URL: https://issues.apache.org/jira/browse/HDFS-8960
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.7.1
> Environment: openjdk version "1.8.0_45-internal"
> OpenJDK Runtime Environment (build 1.8.0_45-internal-b14)
> OpenJDK 64-Bit Server VM (build 25.45-b02, mixed mode)
>Reporter: Benoit Sigoure
>
> Since we upgraded to 2.7.1 we regularly see single-drive failures cause
> widespread problems at the HBase level (with the default 3x replication
> target).
> Here's an example. This HBase RegionServer is r12s16 (172.24.32.16) and is
> writing its WAL to [172.24.32.16:10110, 172.24.32.8:10110,
> 172.24.32.13:10110] as can be seen by the following occasional messages:
> {code}
> 2015-08-23 06:28:40,272 INFO [sync.3] wal.FSHLog: Slow sync cost: 123 ms,
> current pipeline: [172.24.32.16:10110, 172.24.32.8:10110, 172.24.32.13:10110]
> {code}
> A bit later, the second node in the pipeline above is going to experience an
> HDD failure.
> {code}
> 2015-08-23 07:21:58,720 WARN [DataStreamer for file
> /hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1439917659071/r12s16.sjc.aristanetworks.com%2C9104%2C1439917659071.default.1440314434998
> block BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099]
> hdfs.DFSClient: Error Recovery for block
> BP-1466258523-172.24.32.1-1437768622582:blk_1073817519_77099 in pipeline
> 172.24.32.16:10110, 172.24.32.13:10110, 172.24.32.8:10110: bad datanode
> 172.24.32.8:10110
> {code}
> And then HBase will go like "omg I can't write to my WAL, let me commit
> suicide".
> {code}
> 2015-08-23 07:22:26,060 FATAL
> [regionserver/r12s16.sjc.aristanetworks.com/172.24.32.16:9104.append-pool1-t1]
> wal.FSHLog: Could not append. Requesting close of wal
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[172.24.32.16:10110, 172.24.32.13:10110],
> original=[172.24.32.16:10110, 172.24.32.13:10110]). The current failed
> datanode replacement policy is DEFAULT, and a client may configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:969)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1035)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1184)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)
> {code}
> Whereas this should be mostly a non-event as the DFS client should just drop
> the bad replica from the write pipeline.
> This is a small cluster but has 16 DNs so the failed DN in the pipeline
> should be easily replaced. I didn't set
> {{dfs.client.block.write.replace-datanode-on-failure.policy}} (so it's still
> {{DEFAULT}}) and didn't set
> {{dfs.client.block.write.replace-datanode-on-failure.enable}} (so it's still
> {{true}}).
> I don't see anything noteworthy in the NN log around the time of the failure,
> it just seems like the DFS client gave up or threw an exception back to HBase
> that it wasn't throwing before or something else, and that made this single
> drive failure lethal.
> We've occasionally be "unlucky" enough to have a single-drive failure cause
> multiple RegionServers to commit suicide because they had their WALs on that
> drive.
> We upgraded from 2.7.0 about a month ago, and I'm not sure whether we were
> seeing this with 2.7 or not – prior to that we were running in a quite
> different environment, but this is a fairly new deployment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
