[
https://issues.apache.org/jira/browse/HDFS-16128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xing Lin updated HDFS-16128:
----------------------------
Description:
Add support to save Inodes stored in PartitionedGSet when saving an FS image
and load Inodes into PartitionedGSet from a saved FS image.
h1. Saving FSImage
*Original HDFS design*: iterate every inode in inodeMap and save them into the
FSImage file.
*FGL*: no change is needed here, since PartitionedGSet also provides an
iterator interface, to iterate over inodes stored in partitions.
h1. Loading an HDFS
*Original HDFS design*: it first loads the FSImage files and then loads edit
logs for recent changes. FSImage files contain different sections, including
INodeSections and INodeDirectorySections. An InodeSection contains serialized
Inodes objects and the INodeDirectorySection contains the parent inode for an
Inode. When loading an FSImage, the system first loads INodeSections and then
load the INodeDirectorySections, to set the parent inode for each inode.
After FSImage files are loaded, edit logs are then loaded. Edit log contains
recent changes to the filesystem, including Inodes creation/deletion. For a
newly created INode, the parent inode is set before it is added to the inodeMap.
*FGL*: when adding an Inode into the partitionedGSet, we need the parent inode
of an inode, in order to determine which partition to store that inode, when
NAMESPACE_KEY_DEPTH = 2. Thus, in FGL, when loading FSImage files, we used a
temporary LightweightGSet (inodeMapTemp), to store inodes. When LoadFSImage is
done, the parent inode for all existing inodes in FSImage files is set. We can
now move the inodes into a partitionedGSet. Load edit logs can work as usual,
as the parent inode for an inode is set before it is added to the inodeMap.
In theory, PartitionedGSet can support to store inodes without setting its
parent inodes. All these inodes will be stored in the 0th partition. However,
we decide to use a temporary LightweightGSet (inodeMapTemp) to store these
inodes, to make this case more transparent.
was:Add support to save Inodes stored in PartitionedGSet when saving an FS
image and load Inodes into PartitionedGSet from a saved FS image.
> [FGL] Add support for saving/loading an FS Image for PartitionedGSet
> --------------------------------------------------------------------
>
> Key: HDFS-16128
> URL: https://issues.apache.org/jira/browse/HDFS-16128
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs, namenode
> Reporter: Xing Lin
> Assignee: Xing Lin
> Priority: Major
> Labels: pull-request-available
>
> Add support to save Inodes stored in PartitionedGSet when saving an FS image
> and load Inodes into PartitionedGSet from a saved FS image.
> h1. Saving FSImage
> *Original HDFS design*: iterate every inode in inodeMap and save them into
> the FSImage file.
> *FGL*: no change is needed here, since PartitionedGSet also provides an
> iterator interface, to iterate over inodes stored in partitions.
> h1. Loading an HDFS
> *Original HDFS design*: it first loads the FSImage files and then loads edit
> logs for recent changes. FSImage files contain different sections, including
> INodeSections and INodeDirectorySections. An InodeSection contains serialized
> Inodes objects and the INodeDirectorySection contains the parent inode for an
> Inode. When loading an FSImage, the system first loads INodeSections and then
> load the INodeDirectorySections, to set the parent inode for each inode.
> After FSImage files are loaded, edit logs are then loaded. Edit log contains
> recent changes to the filesystem, including Inodes creation/deletion. For a
> newly created INode, the parent inode is set before it is added to the
> inodeMap.
> *FGL*: when adding an Inode into the partitionedGSet, we need the parent
> inode of an inode, in order to determine which partition to store that inode,
> when NAMESPACE_KEY_DEPTH = 2. Thus, in FGL, when loading FSImage files, we
> used a temporary LightweightGSet (inodeMapTemp), to store inodes. When
> LoadFSImage is done, the parent inode for all existing inodes in FSImage
> files is set. We can now move the inodes into a partitionedGSet. Load edit
> logs can work as usual, as the parent inode for an inode is set before it is
> added to the inodeMap.
> In theory, PartitionedGSet can support to store inodes without setting its
> parent inodes. All these inodes will be stored in the 0th partition. However,
> we decide to use a temporary LightweightGSet (inodeMapTemp) to store these
> inodes, to make this case more transparent.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
