[
https://issues.apache.org/jira/browse/HDFS-17865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18075918#comment-18075918
]
ASF GitHub Bot commented on HDFS-17865:
---------------------------------------
ZanderXu commented on code in PR #8144:
URL: https://github.com/apache/hadoop/pull/8144#discussion_r3136051718
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java:
##########
@@ -162,6 +163,14 @@ public boolean isOfType(StorageDirType type) {
* Name directories size for metric.
*/
private Map<String, Long> nameDirSizeMap = new HashMap<>();
+ /**
+ * Service to update {@link #nameDirSizeMap} asynchronously instead of
blocking.
+ */
+ private final NameDirSizeMapUpdateService metricsUpdateService;
Review Comment:
@kokonguyen191
After deep thinking, I feel that introducing a new thread might be a bit
unnecessary and it's expensive. Maybe we can let getNNDirectorySize lazily
update the directory size when it is dirty.
There is a potential race condition: RollingEditsLog may mark the directory
size as dirty while getNNDirectorySize is computing the directory sizes one by
one. In that case, the directories computed earlier should be recomputed.
Maybe we can introduce a version counter to handle this without using a lock.
> Prevent updateNameDirSize() from blocking rollEditLog()
> -------------------------------------------------------
>
> Key: HDFS-17865
> URL: https://issues.apache.org/jira/browse/HDFS-17865
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Felix N
> Assignee: Felix N
> Priority: Minor
> Labels: pull-request-available
>
> updateNameDirSize() has only metrics functionality (i.e. not very critical)
> but has a fair bit of performance impact in one of the more critical methods
> (rollEditLog). It's better to either reduce the metrics update frequency or
> make it update asynchronously to avoid interference with rollEditLog().
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
