[ https://issues.apache.org/jira/browse/HDFS-16601?focusedWorklogId=795437&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-795437 ]
ASF GitHub Bot logged work on HDFS-16601: ----------------------------------------- Author: ASF GitHub Bot Created on: 26/Jul/22 21:15 Start Date: 26/Jul/22 21:15 Worklog Time Spent: 10m Work Description: jojochuang commented on code in PR #4369: URL: https://github.com/apache/hadoop/pull/4369#discussion_r930415320 ########## hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestClientProtocolForPipelineRecovery.java: ########## @@ -761,6 +762,131 @@ public void failPipeline(ReplicaInPipeline replicaInfo, } } + + @Test + public void testPipelineRecoveryWithFailedTransferBlock() throws Exception { + final int chunkSize = 512; + final int oneWriteSize = 5000; + final int totalSize = 1024 * 1024; + final int errorInjectionPos = 512; + Configuration conf = new HdfsConfiguration(); + // Need 5 datanodes to verify the replaceDatanode during pipeline recovery + final MiniDFSCluster cluster = + new MiniDFSCluster.Builder(conf).numDataNodes(5).build(); + DataNodeFaultInjector old = DataNodeFaultInjector.get(); + + try { + DistributedFileSystem fs = cluster.getFileSystem(); + Path fileName = new Path("/f"); + FSDataOutputStream o = fs.create(fileName); + int count = 0; + // Flush to get the pipeline created. + o.writeBytes("hello"); + o.hflush(); + DFSOutputStream dfsO = (DFSOutputStream) o.getWrappedStream(); + final DatanodeInfo[] pipeline = dfsO.getStreamer().getNodes(); + final String firstDn = pipeline[0].getXferAddr(false); + final String secondDn = pipeline[1].getXferAddr(false); + final AtomicBoolean pipelineFailed = new AtomicBoolean(false); + final AtomicBoolean transferFailed = new AtomicBoolean(false); + + DataNodeFaultInjector.set(new DataNodeFaultInjector() { + @Override + public void failPipeline(ReplicaInPipeline replicaInfo, + String mirror) throws IOException { + if (!secondDn.equals(mirror)) { + // Only fail for first DN + return; + } + if (!pipelineFailed.get() && + (replicaInfo.getBytesAcked() > errorInjectionPos) && + (replicaInfo.getBytesAcked() % chunkSize != 0)) { + int count = 0; + while (count < 10) { + // Fail the pipeline (Throw exception) when: + // 1. bytsAcked is not at chunk boundary (checked in the if + // statement above) + // 2. bytesOnDisk is bigger than bytesAcked and at least + // reaches (or go beyond) the end of the chunk that + // bytesAcked is in (checked in the if statement below). + // At this condition, transferBlock that happens during + // pipeline recovery would transfer extra bytes to make up to the + // end of the chunk. And this is when the block corruption + // described in HDFS-4660 would occur. Review Comment: Oh HDFS-4660 brought back my worst nightmare when I spent a month chasing this bug. Issue Time Tracking ------------------- Worklog Id: (was: 795437) Time Spent: 2h (was: 1h 50m) > Failed to replace a bad datanode on the existing pipeline due to no more good > datanodes being available to try > -------------------------------------------------------------------------------------------------------------- > > Key: HDFS-16601 > URL: https://issues.apache.org/jira/browse/HDFS-16601 > Project: Hadoop HDFS > Issue Type: Bug > Reporter: ZanderXu > Assignee: ZanderXu > Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > In our production environment, we found a bug and stack like: > {code:java} > java.io.IOException: Failed to replace a bad datanode on the existing > pipeline due to no more good datanodes being available to try. (Nodes: > current=[DatanodeInfoWithStorage[127.0.0.1:59687,DS-b803febc-7b22-4144-9b39-7bf521cdaa8d,DISK], > > DatanodeInfoWithStorage[127.0.0.1:59670,DS-0d652bc2-1784-430d-961f-750f80a290f1,DISK]], > > original=[DatanodeInfoWithStorage[127.0.0.1:59670,DS-0d652bc2-1784-430d-961f-750f80a290f1,DISK], > > DatanodeInfoWithStorage[127.0.0.1:59687,DS-b803febc-7b22-4144-9b39-7bf521cdaa8d,DISK]]). > The current failed datanode replacement policy is DEFAULT, and a client may > configure this via > 'dfs.client.block.write.replace-datanode-on-failure.policy' in its > configuration. > at > org.apache.hadoop.hdfs.DataStreamer.findNewDatanode(DataStreamer.java:1418) > at > org.apache.hadoop.hdfs.DataStreamer.addDatanode2ExistingPipeline(DataStreamer.java:1478) > at > org.apache.hadoop.hdfs.DataStreamer.handleDatanodeReplacement(DataStreamer.java:1704) > at > org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1605) > at > org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1587) > at > org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1371) > at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:674) > {code} > And the root cause is that DFSClient cannot perceive the exception of > TransferBlock during PipelineRecovery. If failed during TransferBlock, the > DFSClient will retry all datanodes in the cluster and then failed. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org