[
https://issues.apache.org/jira/browse/HDFS-17524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18023233#comment-18023233
]
ASF GitHub Bot commented on HDFS-17524:
---------------------------------------
github-actions[bot] closed pull request #6820: HDFS-17524. OIV: add Transformed
processor which reconstructs an fsimage from another fsimage file
URL: https://github.com/apache/hadoop/pull/6820
> OIV: add Transformed processor which reconstructs an fsimage from another
> fsimage file
> --------------------------------------------------------------------------------------
>
> Key: HDFS-17524
> URL: https://issues.apache.org/jira/browse/HDFS-17524
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
> Affects Versions: 3.2.0, 3.3.4
> Reporter: Xiaobao Wu
> Priority: Major
> Labels: pull-request-available
>
> *Background：*
> The Image file generated by the existing Hadoop 3.3.4 version cannot be
> forward compatible . In the high version of HDFS, the fsimage file conversion
> tool is provided to support the generation of forward compatible fsimage file
> to support the downgrade operation.
> {*}Description{*}：
> Because there are differences in the structure and loading methods of some
> Sections between high and low versions of fsimage files, especially the
> StringTable Section. This will make it impossible to downgrade to a lower
> version of HDFS ( e.g., 3.1.1 ) in higher versions ( e.g., 3.3.4 ), because
> when the lower version of HDFS loads the fsimage file generated by the higher
> version of HDFS, there will be an ArrayIndexOutOfBoundsException.
>
> The code differences are as follows：
> {code:java}
> // 3.3.4
> static SerialNumberManager.StringTable loadStringTable(InputStream in)
> throws IOException {
> ··· ···
> SerialNumberManager.StringTable stringTable =
> SerialNumberManager.newStringTable(s.getNumEntry(), s.getMaskBits());
> for (int i = 0; i < s.getNumEntry(); ++i) {
> FsImageProto.StringTableSection.Entry e = FsImageProto
> .StringTableSection.Entry.parseDelimitedFrom(in);
> stringTable.put(e.getId(), e.getStr());
> }
> return stringTable;
> }
> // 3.1.1
> static String[] loadStringTable(InputStream in) throws IOException {
> ··· ···
> String[] stringTable = new String[s.getNumEntry() + 1];
> for (int i = 0; i < s.getNumEntry(); ++i) {
> FsImageProto.StringTableSection.Entry e = FsImageProto
> .StringTableSection.Entry.parseDelimitedFrom(in);
> // ArrayIndexOutOfBoundsException is triggered when loading a higher
> version of the fsimage file.
> stringTable[e.getId()] = e.getStr();
> }
> return stringTable;
> }{code}
> {*}Solution{*}：
> Solution Reference from HDFS-17463
> !http://www.kdocs.cn/api/v3/office/copy/Mm0rd3BzNEx2Y29zaUdsQkczVnRUV2JwR2RvVWNVdk9aT3dRc2czUXRYdit1ekZ4UmN3UWFLN0hwOTZidnJ1L2ZxaW5PaUNHRmU1bGNyS3lRUGZRbE1vR2I4MlQvS0ppOUZxbVRnQ2o2SUNJZGFoeVNzMUFjR2tKTStsTjZpUTFwanpmcTRML0JFTDJHcXV4aGpESVFXS1RTeEkyZk5sb25LOEEyT0lHbDJydVlIZEJ2dXlyYVozM2pkZGdacEtWQnR3SUQ0MXUwV1RINTMyaDluV2FRTWNjS2p5Nm0rZngzbGNGdEd4cFpLdjFpWUtWK2UyMDZhVVFYUWVHZXlwZEQ0c25MWU93NFY0PQ==/attach/object/K3TLVNAYAAQFQ?|width=693!
> From the figure, it can be seen that the Id arrangement of StringTable in the
> fsimage file has changed from a compact arrangement to a decentralized
> arrangement, that is, USER, GROUP and XATTR are no longer mixed. The
> arrangement is divided into different storage areas and arranged separately.
> * With the sub-sections feature introduced in HDFS-14617, Protobuf can
> support compatible reading.
> * When saving fsimage files in high and low versions, the main difference is
> the arrangement of Entry（e.g., USER, GROUP, and XATTR ） in StringTable.
> * We will add a conversion tool to convert the Id arrangement of the high
> version fsimage file StringTable to a compact arrangement, so that the low
> version can be compatible with this format fsimage file.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
