[
https://issues.apache.org/jira/browse/HDFS-13079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16443191#comment-16443191
]
Hanisha Koneru commented on HDFS-13079:
---------------------------------------
Thanks for working on this [~shashikant].
bq. Please note that in case a checkpoint has already happened and the
requested transaction id has been subsumed in an FSImage, then the namenode
will be started with the next nearest transaction id. Further FSImage files and
edits will be ignored.
In case the requested tx id falls within the latest fsImage , do we want to
load the said fsImage or fallback to a previous fsimage with lastTxId <
requested txId. IMO, we should load the fsImage with the endTxId <= requested
txId.
* In {{FsImage#loadFSImage}}, the check for whether we should load a fsImage is
made after the image is already being loaded. The line {{loader.load(curFile,
requireSameLayoutVersion)}} loads the fsImage transactions into the NN.
{code}
FSImageFormat.LoaderDelegator loader = FSImageFormat.newLoader(conf, target);
loader.load(curFile, requireSameLayoutVersion);
long lastTxIdToLoad = target.getLastTxidToLoad();
long txId = loader.getLoadedImageTxId();
if (lastTxIdToLoad != HdfsServerConstants.INVALID_TXID && txId >
lastTxIdToLoad) {
{code}
* When we skip loading the latest fsImage, we should keep falling back to try
and load the next latest fsImage. For example, say we have the 2 fsImages -
fsimage_00090 and fsimage_00150. Now say we want to start the namenode in
safemode upto txId 120. We first check fsimage_00150 and reject it. After this,
the NN should attempt to load the next latest fsimage i.e. fsimage_00090.
We can throw an exception when skipping an fsImage and catch that exception in
following code path in {{FSImage#loadFsImage}}. This way the next latest
fsimage will be loaded.
{code}
721 FSImageFile imageFile = null;
722 for (int i = 0; i < imageFiles.size(); i++) {
723 try {
724 imageFile = imageFiles.get(i);
725 loadFSImageFile(target, recovery, imageFile, startOpt);
726 break;
....
{code}
* What do we do when there are no fsImages with endTxId <= requested txId?
IMO, we should stop the NN and throw an error.
> Provide a config to start namenode in safemode state upto a certain
> transaction id
> ----------------------------------------------------------------------------------
>
> Key: HDFS-13079
> URL: https://issues.apache.org/jira/browse/HDFS-13079
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: Mukul Kumar Singh
> Assignee: Shashikant Banerjee
> Priority: Major
> Attachments: HDFS-13079.001.patch, HDFS-13079.002.patch
>
>
> In some cases it necessary to rollback the Namenode back to a certain
> transaction id. This is especially needed when the user issues a {{rm -Rf
> -skipTrash}} by mistake.
> Rolling back to a transaction id helps in taking a peek at the filesystem at
> a particular instant. This jira proposes to provide a configuration variable
> using which the namenode can be started upto a certain transaction id. The
> filesystem will be in a readonly safemode which cannot be overridden
> manually. It will only be overridden by removing the config value from the
> config file. Please also note that this will not cause any changes in the
> filesystem state, the filesystem will be in safemode state and no changes to
> the filesystem state will be allowed.
> Please note that in case a checkpoint has already happened and the requested
> transaction id has been subsumed in an FSImage, then the namenode will be
> started with the next nearest transaction id. Further FSImage files and edits
> will be ignored.
> If the checkpoint hasn't happen then the namenode will be started with the
> exact transaction id.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
