[ https://issues.apache.org/jira/browse/HDFS-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226939#comment-16226939 ]
Yiqun Lin edited comment on HDFS-12443 at 10/31/17 3:00 PM: ------------------------------------------------------------ Thanks for the response, [~cheersyang]. I agree with what you said. As you mentioned one key point: DelLog has no notion of DN. For this, If we can add datanode info into delLog, then the data records seem like: ||TxID ||ContainerName||DatanodeID ||BlockList|| |id0|containerName0|datanodeID0|B1,B2,B3| |id1|containerName1|datanodeID1|B4,B5| By this way, we should query container store for getting node info, then write this to delLog as well. Actually this won't increase the total IO operations since {{SCMBlockDeletingService$DeletedBlockTransactionScanner#getToDeleteContainerBlocks()}} also do this query. I mean the following lines in {{getToDeleteContainerBlocks}} can be removed after change: {code} for (DeletedBlocksTransaction tx : txs) { try { ContainerInfo info = mappingService .getContainer(tx.getContainerName()); // Find out the datanode where this TX is supposed to send to. info.getPipeline().getMachines() .forEach(entry -> dnTXs.addTransaction(entry, tx)); } catch (IOException e) { LOG.warn("Container {} not found, continue to process next", tx.getContainerName(), e); } } {code} In {{SCMBlockDeletingService}}, I plan to use the following way: * Calculate the BlockDeleteLimitSize (The maximum container number for each node) * Scan the entire delLog from the beginning to end, getting blocks list info for each node. If one node reach maximum container number, then its record will be skipped. If not, keep scanning log until it reach the maximum value. How do you think now, [~cheersyang]? was (Author: linyiqun): Thanks for the response, [~cheersyang]. I agree with what you said. As you mentioned one key point: DelLog has no notion of DN. For this, If we can add datanode info into delLog, then the data records seem like: ||TxID ||ContainerName||DatanodeID ||BlockList|| |id0|containerName0|datanodeID0|B1,B2,B3| |id1|containerName1|datanodeID1|B1,B2| By this way, we should query container store for getting node info, then write this to delLog as well. Actually this won't increase the total IO operations since {{SCMBlockDeletingService$DeletedBlockTransactionScanner#getToDeleteContainerBlocks()}} also do this query. I mean the following lines in {{getToDeleteContainerBlocks}} can be removed after change: {code} for (DeletedBlocksTransaction tx : txs) { try { ContainerInfo info = mappingService .getContainer(tx.getContainerName()); // Find out the datanode where this TX is supposed to send to. info.getPipeline().getMachines() .forEach(entry -> dnTXs.addTransaction(entry, tx)); } catch (IOException e) { LOG.warn("Container {} not found, continue to process next", tx.getContainerName(), e); } } {code} In {{SCMBlockDeletingService}}, I plan to use the following way: * Calculate the BlockDeleteLimitSize (The maximum container number for each node) * Scan the entire delLog from the beginning to end, getting blocks list info for each node. If one node reach maximum container number, then its record will be skipped. How do you think now, [~cheersyang]? > Ozone: Improve SCM block deletion throttling algorithm > ------------------------------------------------------- > > Key: HDFS-12443 > URL: https://issues.apache.org/jira/browse/HDFS-12443 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, scm > Reporter: Weiwei Yang > Assignee: Yiqun Lin > Labels: OzonePostMerge > Attachments: HDFS-12443-HDFS-7240.001.patch, > HDFS-12443-HDFS-7240.002.patch, HDFS-12443-HDFS-7240.002.patch, > HDFS-12443-SCM-blockdeletion-throttle.pdf > > > Currently SCM scans delLog to send deletion transactions to datanode > periodically, the throttling algorithm is simple, it scans at most > {{BLOCK_DELETE_TX_PER_REQUEST_LIMIT}} (by default 50) at a time. This is > non-optimal, worst case it might cache 50 TXs for 50 different DNs so each DN > will only get 1 TX to proceed in an interval, this will make the deletion > slow. An improvement to this is to make this throttling by datanode, e.g 50 > TXs per datanode per interval. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org