[ https://issues.apache.org/jira/browse/HDFS-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17212127#comment-17212127 ]
Karthik Palanisamy edited comment on HDFS-15621 at 10/12/20, 5:30 AM: ---------------------------------------------------------------------- [~sodonnell] I think we may only store block info. Can't we re-construct Blockpool and other constant parts? Blockpool can be identified before initializing the scanner right and only need to think of "finalized", "rbw",etc.. /current/BP-671271071-10.163.205.13-1552020401842/current/finalized/subdir28/subdir17 was (Author: kpalanisamy): [~sodonnell] I think we may only store block info. Can't we re-construct Blockpool and other constant parts? Blackpool can be identified before initializing the scanner right and only need to think of "finalized", "rbw",etc.. /current/BP-671271071-10.163.205.13-1552020401842/current/finalized/subdir28/subdir17 > Datanode DirectoryScanner uses excessive memory > ----------------------------------------------- > > Key: HDFS-15621 > URL: https://issues.apache.org/jira/browse/HDFS-15621 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode > Affects Versions: 3.4.0 > Reporter: Stephen O'Donnell > Assignee: Stephen O'Donnell > Priority: Major > Attachments: Screenshot 2020-10-09 at 14.11.36.png, Screenshot > 2020-10-09 at 15.20.56.png > > > We generally work a rule of 1GB heap on a datanode per 1M blocks. For nodes > with a lot of blocks, this can mean a lot of heap. > We recently captured a heapdump of a DN with about 22M blocks and found only > about 1.5GB was occupied by the ReplicaMap. Another 9GB of the heap is taken > by the DirectoryScanner ScanInfo objects. Most of this memory was alloated to > strings. > Checking the strings in question, we can see two strings per scanInfo, > looking like: > {code} > /current/BP-671271071-10.163.205.13-1552020401842/current/finalized/subdir28/subdir17/blk_1180438785 > _106716708.meta > {code} > I will update a screen shot from MAT showing this. > For the first string especially, the part > "/current/BP-671271071-10.163.205.13-1552020401842/current/finalized/" will > be the same for every block in the block pool as the scanner is only > concerned about finalized blocks. > We can probably also store just the subdir indexes "28" and "27" rather than > "subdir28/subdir17" and then construct the path when it is requested via the > getter. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org