[
https://issues.apache.org/jira/browse/HDFS-16128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko resolved HDFS-16128.
----------------------------------------
Fix Version/s: Fine-Grained Locking
Hadoop Flags: Reviewed
Resolution: Fixed
I just committed this. Thank you [~xinglin].
> [FGL] Add support for saving/loading an FS Image for PartitionedGSet
> --------------------------------------------------------------------
>
> Key: HDFS-16128
> URL: https://issues.apache.org/jira/browse/HDFS-16128
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs, namenode
> Reporter: Xing Lin
> Assignee: Xing Lin
> Priority: Major
> Labels: pull-request-available
> Fix For: Fine-Grained Locking
>
>
> Add support to save Inodes stored in PartitionedGSet when saving an FS image
> and load Inodes into PartitionedGSet from a saved FS image.
> h1. Saving FSImage
> *Original HDFS design*: iterate every inode in inodeMap and save them into
> the FSImage file.
> *FGL*: no change is needed here, since PartitionedGSet also provides an
> iterator interface, to iterate over inodes stored in partitions.
> h1. Loading an HDFS
> *Original HDFS design*: it first loads the FSImage files and then loads edit
> logs for recent changes. FSImage files contain different sections, including
> INodeSections and INodeDirectorySections. An InodeSection contains serialized
> Inodes objects and the INodeDirectorySection contains the parent inode for an
> Inode. When loading an FSImage, the system first loads INodeSections and then
> load the INodeDirectorySections, to set the parent inode for each inode.
> After FSImage files are loaded, edit logs are then loaded. Edit log contains
> recent changes to the filesystem, including Inodes creation/deletion. For a
> newly created INode, the parent inode is set before it is added to the
> inodeMap.
> *FGL*: when adding an Inode into the partitionedGSet, we need the parent
> inode of an inode, in order to determine which partition to store that inode,
> when NAMESPACE_KEY_DEPTH = 2. Thus, in FGL, when loading FSImage files, we
> used a temporary LightweightGSet (inodeMapTemp), to store inodes. When
> LoadFSImage is done, the parent inode for all existing inodes in FSImage
> files is set. We can now move the inodes into a partitionedGSet. Load edit
> logs can work as usual, as the parent inode for an inode is set before it is
> added to the inodeMap.
> In theory, PartitionedGSet can support to store inodes without setting its
> parent inodes. All these inodes will be stored in the 0th partition. However,
> we decide to use a temporary LightweightGSet (inodeMapTemp) to store these
> inodes, to make this case more transparent.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
