[
https://issues.apache.org/jira/browse/HDFS-17524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18022939#comment-18022939
]
ASF GitHub Bot commented on HDFS-17524:
---------------------------------------
github-actions[bot] commented on PR #6820:
URL: https://github.com/apache/hadoop/pull/6820#issuecomment-3336332180
We're closing this stale PR because it has been open for 100 days with no
activity. This isn't a judgement on the merit of the PR in any way. It's just a
way of keeping the PR queue manageable.
If you feel like this was a mistake, or you would like to continue working
on it, please feel free to re-open it and ask for a committer to remove the
stale tag and review again.
Thanks all for your contribution.
> OIV: add Transformed processor which reconstructs an fsimage from another
> fsimage file
> --------------------------------------------------------------------------------------
>
> Key: HDFS-17524
> URL: https://issues.apache.org/jira/browse/HDFS-17524
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
> Affects Versions: 3.2.0, 3.3.4
> Reporter: Xiaobao Wu
> Priority: Major
> Labels: pull-request-available
>
> *Background：*
> The Image file generated by the existing Hadoop 3.3.4 version cannot be
> forward compatible . In the high version of HDFS, the fsimage file conversion
> tool is provided to support the generation of forward compatible fsimage file
> to support the downgrade operation.
> {*}Description{*}：
> Because there are differences in the structure and loading methods of some
> Sections between high and low versions of fsimage files, especially the
> StringTable Section. This will make it impossible to downgrade to a lower
> version of HDFS ( e.g., 3.1.1 ) in higher versions ( e.g., 3.3.4 ), because
> when the lower version of HDFS loads the fsimage file generated by the higher
> version of HDFS, there will be an ArrayIndexOutOfBoundsException.
>
> The code differences are as follows：
> {code:java}
> // 3.3.4
> static SerialNumberManager.StringTable loadStringTable(InputStream in)
> throws IOException {
> ··· ···
> SerialNumberManager.StringTable stringTable =
> SerialNumberManager.newStringTable(s.getNumEntry(), s.getMaskBits());
> for (int i = 0; i < s.getNumEntry(); ++i) {
> FsImageProto.StringTableSection.Entry e = FsImageProto
> .StringTableSection.Entry.parseDelimitedFrom(in);
> stringTable.put(e.getId(), e.getStr());
> }
> return stringTable;
> }
> // 3.1.1
> static String[] loadStringTable(InputStream in) throws IOException {
> ··· ···
> String[] stringTable = new String[s.getNumEntry() + 1];
> for (int i = 0; i < s.getNumEntry(); ++i) {
> FsImageProto.StringTableSection.Entry e = FsImageProto
> .StringTableSection.Entry.parseDelimitedFrom(in);
> // ArrayIndexOutOfBoundsException is triggered when loading a higher
> version of the fsimage file.
> stringTable[e.getId()] = e.getStr();
> }
> return stringTable;
> }{code}
> {*}Solution{*}：
> Solution Reference from HDFS-17463
> !http://www.kdocs.cn/api/v3/office/copy/Mm0rd3BzNEx2Y29zaUdsQkczVnRUV2JwR2RvVWNVdk9aT3dRc2czUXRYdit1ekZ4UmN3UWFLN0hwOTZidnJ1L2ZxaW5PaUNHRmU1bGNyS3lRUGZRbE1vR2I4MlQvS0ppOUZxbVRnQ2o2SUNJZGFoeVNzMUFjR2tKTStsTjZpUTFwanpmcTRML0JFTDJHcXV4aGpESVFXS1RTeEkyZk5sb25LOEEyT0lHbDJydVlIZEJ2dXlyYVozM2pkZGdacEtWQnR3SUQ0MXUwV1RINTMyaDluV2FRTWNjS2p5Nm0rZngzbGNGdEd4cFpLdjFpWUtWK2UyMDZhVVFYUWVHZXlwZEQ0c25MWU93NFY0PQ==/attach/object/K3TLVNAYAAQFQ?|width=693!
> From the figure, it can be seen that the Id arrangement of StringTable in the
> fsimage file has changed from a compact arrangement to a decentralized
> arrangement, that is, USER, GROUP and XATTR are no longer mixed. The
> arrangement is divided into different storage areas and arranged separately.
> * With the sub-sections feature introduced in HDFS-14617, Protobuf can
> support compatible reading.
> * When saving fsimage files in high and low versions, the main difference is
> the arrangement of Entry（e.g., USER, GROUP, and XATTR ） in StringTable.
> * We will add a conversion tool to convert the Id arrangement of the high
> version fsimage file StringTable to a compact arrangement, so that the low
> version can be compatible with this format fsimage file.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
