[ https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14529045#comment-14529045 ]
Hadoop QA commented on HDFS-7980: --------------------------------- \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:red}-1{color} | patch | 0m 0s | The patch command could not apply the patch during dryrun. | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12729034/HDFS-7980.004.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / ffce9a3 | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/10819/console | This message was automatically generated. > Incremental BlockReport will dramatically slow down the startup of a namenode > ------------------------------------------------------------------------------ > > Key: HDFS-7980 > URL: https://issues.apache.org/jira/browse/HDFS-7980 > Project: Hadoop HDFS > Issue Type: Bug > Reporter: Hui Zheng > Assignee: Walter Su > Attachments: HDFS-7980.001.patch, HDFS-7980.002.patch, > HDFS-7980.003.patch, HDFS-7980.004.patch > > > In the current implementation the datanode will call the > reportReceivedDeletedBlocks() method that is a IncrementalBlockReport before > calling the bpNamenode.blockReport() method. So in a large(several thousands > of datanodes) and busy cluster it will slow down(more than one hour) the > startup of namenode. > {code} > List<DatanodeCommand> blockReport() throws IOException { > // send block report if timer has expired. > final long startTime = now(); > if (startTime - lastBlockReport <= dnConf.blockReportInterval) { > return null; > } > final ArrayList<DatanodeCommand> cmds = new ArrayList<DatanodeCommand>(); > // Flush any block information that precedes the block report. Otherwise > // we have a chance that we will miss the delHint information > // or we will report an RBW replica after the BlockReport already reports > // a FINALIZED one. > reportReceivedDeletedBlocks(); > lastDeletedReport = startTime; > ......... > // Send the reports to the NN. > int numReportsSent = 0; > int numRPCs = 0; > boolean success = false; > long brSendStartTime = now(); > try { > if (totalBlockCount < dnConf.blockReportSplitThreshold) { > // Below split threshold, send all reports in a single message. > DatanodeCommand cmd = bpNamenode.blockReport( > bpRegistration, bpos.getBlockPoolId(), reports); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)