[
https://issues.apache.org/jira/browse/HDFS-13700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899605#comment-16899605
]
He Xiaoqiao commented on HDFS-13700:
------------------------------------
HDFS-14617 try to upgrade fsimage loading to pararallel, which is very similar
with HDFS-7784. I think it could solve this issue, or any chance to merge this
thought together?
> The process of loading image can be done in a pipeline model
> ------------------------------------------------------------
>
> Key: HDFS-13700
> URL: https://issues.apache.org/jira/browse/HDFS-13700
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: zhouyingchao
> Assignee: Lisheng Sun
> Priority: Major
> Attachments: HDFS-13700-001.patch
>
>
> The process of loading a file system image involves reading inodes section,
> deserializing inodes, initializing inodes, adding inodes to the global map,
> reading directories section, adding inodes to their parents' map, cache name
> etc. These steps can be done in a pipeline model to reduce the total
> duration.
> Test the patch against a fsimage of a 70PB 2.4 cluster (200million files and
> 300million blocks, the fsimage is around 22GB), the image loading time be
> reduced from 1210 seconds to 739 seconds.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
