[
https://issues.apache.org/jira/browse/HDFS-16432?focusedWorklogId=722709&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-722709
]
ASF GitHub Bot logged work on HDFS-16432:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 08/Feb/22 10:35
Start Date: 08/Feb/22 10:35
Worklog Time Spent: 10m
Work Description: sodonnel commented on pull request #3907:
URL: https://github.com/apache/hadoop/pull/3907#issuecomment-1032457941
I am also concerned about yielding the lock in the middle of processing a
storage. I would be interested to understand where the time is spend doing the
block report processing inside the namenode, to see if we can improve the
performance of the code and avoid yielding the lock.
Have you tried reproducing this problem on a small otherwise idle cluster
and attaching the async profiler to the namenode process to see where the code
is taking time during block reporting? I think that analysis would be valuable
before we change the locking, as there may be something which can be improved
around the performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 722709)
Time Spent: 1h 40m (was: 1.5h)
> Namenode block report add yield to avoid holding write lock too long
> --------------------------------------------------------------------
>
> Key: HDFS-16432
> URL: https://issues.apache.org/jira/browse/HDFS-16432
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: qinyuren
> Priority: Major
> Labels: pull-request-available
> Attachments: image-2022-01-20-15-19-28-384.png
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> !image-2022-01-20-15-19-28-384.png|width=683,height=132!
> In our cluster, namenode block report will held write lock for a long time if
> the storage block number more than 100000. So we want to add a yield
> mechanism in block reporting process to avoid holding write lock too long.
> # Ensure that the processing of the same block is in the same write lock.
> # Because StorageInfo.addBlock will moves the block to the head of
> blockList, so we can collect blocks that have not been reported by delimiter
> block.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
