[
https://issues.apache.org/jira/browse/HDFS-16601?focusedWorklogId=795437&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-795437
]
ASF GitHub Bot logged work on HDFS-16601:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 26/Jul/22 21:15
Start Date: 26/Jul/22 21:15
Worklog Time Spent: 10m
Work Description: jojochuang commented on code in PR #4369:
URL: https://github.com/apache/hadoop/pull/4369#discussion_r930415320
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestClientProtocolForPipelineRecovery.java:
##########
@@ -761,6 +762,131 @@ public void failPipeline(ReplicaInPipeline replicaInfo,
}
}
+
+ @Test
+ public void testPipelineRecoveryWithFailedTransferBlock() throws Exception {
+ final int chunkSize = 512;
+ final int oneWriteSize = 5000;
+ final int totalSize = 1024 * 1024;
+ final int errorInjectionPos = 512;
+ Configuration conf = new HdfsConfiguration();
+ // Need 5 datanodes to verify the replaceDatanode during pipeline recovery
+ final MiniDFSCluster cluster =
+ new MiniDFSCluster.Builder(conf).numDataNodes(5).build();
+ DataNodeFaultInjector old = DataNodeFaultInjector.get();
+
+ try {
+ DistributedFileSystem fs = cluster.getFileSystem();
+ Path fileName = new Path("/f");
+ FSDataOutputStream o = fs.create(fileName);
+ int count = 0;
+ // Flush to get the pipeline created.
+ o.writeBytes("hello");
+ o.hflush();
+ DFSOutputStream dfsO = (DFSOutputStream) o.getWrappedStream();
+ final DatanodeInfo[] pipeline = dfsO.getStreamer().getNodes();
+ final String firstDn = pipeline[0].getXferAddr(false);
+ final String secondDn = pipeline[1].getXferAddr(false);
+ final AtomicBoolean pipelineFailed = new AtomicBoolean(false);
+ final AtomicBoolean transferFailed = new AtomicBoolean(false);
+
+ DataNodeFaultInjector.set(new DataNodeFaultInjector() {
+ @Override
+ public void failPipeline(ReplicaInPipeline replicaInfo,
+ String mirror) throws IOException {
+ if (!secondDn.equals(mirror)) {
+ // Only fail for first DN
+ return;
+ }
+ if (!pipelineFailed.get() &&
+ (replicaInfo.getBytesAcked() > errorInjectionPos) &&
+ (replicaInfo.getBytesAcked() % chunkSize != 0)) {
+ int count = 0;
+ while (count < 10) {
+ // Fail the pipeline (Throw exception) when:
+ // 1. bytsAcked is not at chunk boundary (checked in the if
+ // statement above)
+ // 2. bytesOnDisk is bigger than bytesAcked and at least
+ // reaches (or go beyond) the end of the chunk that
+ // bytesAcked is in (checked in the if statement below).
+ // At this condition, transferBlock that happens during
+ // pipeline recovery would transfer extra bytes to make up to the
+ // end of the chunk. And this is when the block corruption
+ // described in HDFS-4660 would occur.
Review Comment:
Oh HDFS-4660 brought back my worst nightmare when I spent a month chasing
this bug.
Issue Time Tracking
-------------------
Worklog Id: (was: 795437)
Time Spent: 2h (was: 1h 50m)
> Failed to replace a bad datanode on the existing pipeline due to no more good
> datanodes being available to try
> --------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-16601
> URL: https://issues.apache.org/jira/browse/HDFS-16601
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: ZanderXu
> Assignee: ZanderXu
> Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> In our production environment, we found a bug and stack like:
> {code:java}
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[127.0.0.1:59687,DS-b803febc-7b22-4144-9b39-7bf521cdaa8d,DISK],
>
> DatanodeInfoWithStorage[127.0.0.1:59670,DS-0d652bc2-1784-430d-961f-750f80a290f1,DISK]],
>
> original=[DatanodeInfoWithStorage[127.0.0.1:59670,DS-0d652bc2-1784-430d-961f-750f80a290f1,DISK],
>
> DatanodeInfoWithStorage[127.0.0.1:59687,DS-b803febc-7b22-4144-9b39-7bf521cdaa8d,DISK]]).
> The current failed datanode replacement policy is DEFAULT, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DataStreamer.findNewDatanode(DataStreamer.java:1418)
> at
> org.apache.hadoop.hdfs.DataStreamer.addDatanode2ExistingPipeline(DataStreamer.java:1478)
> at
> org.apache.hadoop.hdfs.DataStreamer.handleDatanodeReplacement(DataStreamer.java:1704)
> at
> org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1605)
> at
> org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1587)
> at
> org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1371)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:674)
> {code}
> And the root cause is that DFSClient cannot perceive the exception of
> TransferBlock during PipelineRecovery. If failed during TransferBlock, the
> DFSClient will retry all datanodes in the cluster and then failed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
