[ https://issues.apache.org/jira/browse/HDFS-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shilun Fan updated HDFS-15621: ------------------------------ Hadoop Flags: Reviewed > Datanode DirectoryScanner uses excessive memory > ----------------------------------------------- > > Key: HDFS-15621 > URL: https://issues.apache.org/jira/browse/HDFS-15621 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode > Affects Versions: 3.4.0 > Reporter: Stephen O'Donnell > Assignee: Stephen O'Donnell > Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0, 3.2.3 > > Attachments: Screenshot 2020-10-09 at 14.11.36.png, Screenshot > 2020-10-09 at 15.20.56.png > > Time Spent: 1h 20m > Remaining Estimate: 0h > > We generally work a rule of 1GB heap on a datanode per 1M blocks. For nodes > with a lot of blocks, this can mean a lot of heap. > We recently captured a heapdump of a DN with about 22M blocks and found only > about 1.5GB was occupied by the ReplicaMap. Another 9GB of the heap is taken > by the DirectoryScanner ScanInfo objects. Most of this memory was alloated to > strings. > Checking the strings in question, we can see two strings per scanInfo, > looking like: > {code} > /current/BP-671271071-10.163.205.13-1552020401842/current/finalized/subdir28/subdir17/blk_1180438785 > _106716708.meta > {code} > I will update a screen shot from MAT showing this. > For the first string especially, the part > "/current/BP-671271071-10.163.205.13-1552020401842/current/finalized/" will > be the same for every block in the block pool as the scanner is only > concerned about finalized blocks. > We can probably also store just the subdir indexes "28" and "27" rather than > "subdir28/subdir17" and then construct the path when it is requested via the > getter. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org