[
https://issues.apache.org/jira/browse/HDFS-15584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17198220#comment-17198220
]
zhuqi commented on HDFS-15584:
------------------------------
Hi [~sodonnell]
Yeah, i agree with you that we should sleep.
And the good default for "dfs.namenode.block.deletion.lock.time.threshold", we
should test how much it should be. If the sleep time is 1ms, the
"dfs.namenode.block.deletion.lock.time.threshold" may should be tens of ms, but
i set the it 100ms for init.
Also i add the threshold is to avoid sleep when not heavy deletion, to reduce
the time it holds the lock. You are right.
When pending deletion too many, the datanode will be heavy to deal with too
many deletion. Also the speed namenode put into will be slower, which affect
the performance.
Thanks for your quickly reply.
> Improve HDFS large deletion cause namenode lockqueue boom and pending
> deletion boom.
> ------------------------------------------------------------------------------------
>
> Key: HDFS-15584
> URL: https://issues.apache.org/jira/browse/HDFS-15584
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 3.4.0
> Reporter: zhuqi
> Assignee: zhuqi
> Priority: Major
> Attachments: HDFS-15584.001.patch
>
>
> In our production cluster, the large deletion will boom the namenode lock
> queue, also will lead to the boom of pending deletion in invalidate blocks.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
