[
https://issues.apache.org/jira/browse/HDFS-12443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226939#comment-16226939
]
Yiqun Lin commented on HDFS-12443:
----------------------------------
Thanks for the response, [~cheersyang].
I agree with what you said. As you mentioned one key point: DelLog has no
notion of DN. For this, If we can add datanode info into delLog, then the data
records seem like:
||TxID ||ContainerName||DatanodeID ||BlockList||
|id0|containerName0|datanodeID0|B1,B2,B3|
|id1|containerName1|datanodeID1|B1,B2|
By this way, we should query container store for getting node info, then write
this to delLog as well. Actually this won't increase the total IO operations
since
{{SCMBlockDeletingService$DeletedBlockTransactionScanner#getToDeleteContainerBlocks()}}
also do this query. I mean the following lines in
{{getToDeleteContainerBlocks}} can be removed after change:
{code}
for (DeletedBlocksTransaction tx : txs) {
try {
ContainerInfo info = mappingService
.getContainer(tx.getContainerName());
// Find out the datanode where this TX is supposed to send to.
info.getPipeline().getMachines()
.forEach(entry -> dnTXs.addTransaction(entry, tx));
} catch (IOException e) {
LOG.warn("Container {} not found, continue to process next",
tx.getContainerName(), e);
}
}
{code}
In {{SCMBlockDeletingService}}, I plan to use the following way:
* Calculate the BlockDeleteLimitSize (The maximum container number for each
node)
* Scan the entire delLog from the beginning to end, getting blocks list info
for each node. If one node reach maximum container number, then its record will
be skipped.
How do you think now, [~cheersyang]?
> Ozone: Improve SCM block deletion throttling algorithm
> -------------------------------------------------------
>
> Key: HDFS-12443
> URL: https://issues.apache.org/jira/browse/HDFS-12443
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, scm
> Reporter: Weiwei Yang
> Assignee: Yiqun Lin
> Labels: OzonePostMerge
> Attachments: HDFS-12443-HDFS-7240.001.patch,
> HDFS-12443-HDFS-7240.002.patch, HDFS-12443-HDFS-7240.002.patch,
> HDFS-12443-SCM-blockdeletion-throttle.pdf
>
>
> Currently SCM scans delLog to send deletion transactions to datanode
> periodically, the throttling algorithm is simple, it scans at most
> {{BLOCK_DELETE_TX_PER_REQUEST_LIMIT}} (by default 50) at a time. This is
> non-optimal, worst case it might cache 50 TXs for 50 different DNs so each DN
> will only get 1 TX to proceed in an interval, this will make the deletion
> slow. An improvement to this is to make this throttling by datanode, e.g 50
> TXs per datanode per interval.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
