[ https://issues.apache.org/jira/browse/HDFS-16533?focusedWorklogId=775614&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-775614 ]
ASF GitHub Bot logged work on HDFS-16533: ----------------------------------------- Author: ASF GitHub Bot Created on: 28/May/22 03:59 Start Date: 28/May/22 03:59 Worklog Time Spent: 10m Work Description: ZanderXu commented on PR #4155: URL: https://github.com/apache/hadoop/pull/4155#issuecomment-1140163775 Both org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery.testPipelineRecoveryOnRestartFailure and org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement.testSynchronousEviction are introduced by [HDFS-16534](https://issues.apache.org/jira/browse/HDFS-16534). And I have create two PRs to fix it. - [testSynchronousEviction](https://github.com/apache/hadoop/pull/4367) - [testPipelineRecoveryOnRestartFailure](https://github.com/apache/hadoop/pull/4366) @jojochuang please help me review it again. Issue Time Tracking ------------------- Worklog Id: (was: 775614) Time Spent: 3h 10m (was: 3h) > COMPOSITE_CRC failed between replicated file and striped file. > -------------------------------------------------------------- > > Key: HDFS-16533 > URL: https://issues.apache.org/jira/browse/HDFS-16533 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs, hdfs-client > Reporter: ZanderXu > Assignee: ZanderXu > Priority: Major > Labels: pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > After testing the COMPOSITE_CRC with some random length between replicated > file and striped file which has same data with replicated file, it failed. > Reproduce step like this: > {code:java} > @Test(timeout = 90000) > public void testStripedAndReplicatedFileChecksum2() throws Exception { > int abnormalSize = (dataBlocks * 2 - 2) * blockSize + > (int) (blockSize * 0.5); > prepareTestFiles(abnormalSize, new String[] {stripedFile1, replicatedFile}); > int loopNumber = 100; > while (loopNumber-- > 0) { > int verifyLength = ThreadLocalRandom.current() > .nextInt(10, abnormalSize); > FileChecksum stripedFileChecksum1 = getFileChecksum(stripedFile1, > verifyLength, false); > FileChecksum replicatedFileChecksum = getFileChecksum(replicatedFile, > verifyLength, false); > if (checksumCombineMode.equals(ChecksumCombineMode.COMPOSITE_CRC.name())) > { > Assert.assertEquals(stripedFileChecksum1, replicatedFileChecksum); > } else { > Assert.assertNotEquals(stripedFileChecksum1, replicatedFileChecksum); > } > } > } {code} > And after tracing the root cause, `FileChecksumHelper#makeCompositeCrcResult` > maybe compute an error `consumedLastBlockLength` when updating checksum for > the last block of the fixed length which maybe not the last block in the file. -- This message was sent by Atlassian Jira (v8.20.7#820007) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org