[ https://issues.apache.org/jira/browse/HDFS-17346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated HDFS-17346: ---------------------------------- Labels: pull-request-available (was: ) > Fix DirectoryScanner check mark the normal blocks as corrupt. > ------------------------------------------------------------- > > Key: HDFS-17346 > URL: https://issues.apache.org/jira/browse/HDFS-17346 > Project: Hadoop HDFS > Issue Type: Improvement > Reporter: Haiyang Hu > Assignee: Haiyang Hu > Priority: Major > Labels: pull-request-available > > DirectoryScanner check mark the normal blocks as corrupt and report to > namenode, it maybe cause some corrupted blocks, actually these are health. > This can happen if Appending and DirectoryScanner are running at the same > time, and the probability is very high. > *Root cause:* > * Create a file such as:blk_xxx_1001 and diskFile is > "file:/XXX/current/finalized/blk_xxx", diskMetaFile is > "file:/XXX/current/finalized/blk_xxx_1001.meta" > * Run DirectoryScanner, first will create BlockPoolReport.ScanInfo and record > blockFile is "file:/XXX/current/finalized/blk_xxx" and metaFile is > "file:/XXX/current/finalized/blk_xxx_1001.meta" > * Simultaneously other thread to complete append for blk_xxx, then the > diskFile "file:/XXX/current/finalized/blk_xxx", diskMetaFile > "file:/XXX/current/finalized/blk_xxx_1002.meta", memMetaFile > "file:/XXX/current/finalized/blk_xxx", memDataFile > "file:/XXX/current/finalized/blk_xxx_1002.meta" > * DirectoryScanner continue to run, due to the different generation stamps of > the metadata file in mem and metadata file in scanInfo will add the scanInfo > object to the list of differences > * Continue to run FsDatasetImpl#checkAndUpdate will traverse the list of > differences, due to current diskMetaFile > "/XXX/current/finalized/blk_xxx_1001.meta" is not exists, so isRegular as > false > {code:java} > final boolean isRegular = FileUtil.isRegularFile(diskMetaFile, false) && > FileUtil.isRegularFile(diskFile, false); > {code} > * Here will mark the normal blocks as corrupt and report to namenode > {code:java} > } else if (!isRegular) { > corruptBlock = new Block(memBlockInfo); > LOG.warn("Block:{} is not a regular file.", corruptBlock.getBlockId()); > } > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org