[ https://issues.apache.org/jira/browse/HDFS-6114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinayakumar B updated HDFS-6114: -------------------------------- Attachment: HDFS-6114.patch Updated patch with above comments. Please review > Block Scan log rolling will never happen if blocks written continuously > leading to huge size of dncp_block_verification.log.curr > -------------------------------------------------------------------------------------------------------------------------------- > > Key: HDFS-6114 > URL: https://issues.apache.org/jira/browse/HDFS-6114 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode > Affects Versions: 2.3.0, 2.4.0 > Reporter: Vinayakumar B > Assignee: Vinayakumar B > Priority: Critical > Attachments: HDFS-6114.patch, HDFS-6114.patch, HDFS-6114.patch, > HDFS-6114.patch > > > 1. {{BlockPoolSliceScanner#scan()}} will not return until all the blocks are > scanned. > 2. If the blocks (with size in several MBs) to datanode are written > continuously > then one iteration of {{BlockPoolSliceScanner#scan()}} will be continously > scanning the blocks > 3. These blocks will be deleted after some time (enough to get block scanned) > 4. As Block Scanning is throttled, So verification of all blocks will take so > much time. > 5. Rolling will never happen, so even though the total number of blocks in > datanode doesn't increases, entries ( which contains stale entries of deleted > blocks) in *dncp_block_verification.log.curr* continuously increases leading > to huge size. > In one of our env, it grown more than 1TB where total number of blocks were > only ~45k. -- This message was sent by Atlassian JIRA (v6.2#6252)