Xiaobao Wu created HDFS-17524:
---------------------------------
Summary: OIV: add Transformed processor which reconstructs an
fsimage from another fsimage file
Key: HDFS-17524
URL: https://issues.apache.org/jira/browse/HDFS-17524
Project: Hadoop HDFS
Issue Type: Improvement
Components: tools
Affects Versions: 3.3.4, 3.2.0
Reporter: Xiaobao Wu
*Background：*
The Image file generated by the existing Hadoop 3.3.4 version cannot be forward
compatible . In the high version of HDFS, the fsimage file conversion tool is
provided to support the generation of forward compatible fsimage file to
support the downgrade operation.
{*}Description{*}：
Because there are differences in the structure and loading methods of some
Sections between high and low versions of fsimage files, especially the
StringTable Section. This will make it impossible to downgrade to a lower
version of HDFS ( e.g., 3.1.1 ) in higher versions ( e.g., 3.3.4 ), because
when the lower version of HDFS loads the fsimage file generated by the higher
version of HDFS, there will be an ArrayIndexOutOfBoundsException.
The code differences are as follows：
{code:java}
// 3.3.4
static SerialNumberManager.StringTable loadStringTable(InputStream in)
throws IOException {
··· ···
SerialNumberManager.StringTable stringTable =
SerialNumberManager.newStringTable(s.getNumEntry(), s.getMaskBits());
for (int i = 0; i < s.getNumEntry(); ++i) {
FsImageProto.StringTableSection.Entry e = FsImageProto
.StringTableSection.Entry.parseDelimitedFrom(in);
stringTable.put(e.getId(), e.getStr());
}
return stringTable;
}
// 3.1.1
static String[] loadStringTable(InputStream in) throws IOException {
··· ···
String[] stringTable = new String[s.getNumEntry() + 1];
for (int i = 0; i < s.getNumEntry(); ++i) {
FsImageProto.StringTableSection.Entry e = FsImageProto
.StringTableSection.Entry.parseDelimitedFrom(in);
// ArrayIndexOutOfBoundsException is triggered when loading a higher
version of the fsimage file.
stringTable[e.getId()] = e.getStr();
}
return stringTable;
}{code}
{*}Solution{*}：
Solution Reference from HDFS-17463
!http://www.kdocs.cn/api/v3/office/copy/Mm0rd3BzNEx2Y29zaUdsQkczVnRUV2JwR2RvVWNVdk9aT3dRc2czUXRYdit1ekZ4UmN3UWFLN0hwOTZidnJ1L2ZxaW5PaUNHRmU1bGNyS3lRUGZRbE1vR2I4MlQvS0ppOUZxbVRnQ2o2SUNJZGFoeVNzMUFjR2tKTStsTjZpUTFwanpmcTRML0JFTDJHcXV4aGpESVFXS1RTeEkyZk5sb25LOEEyT0lHbDJydVlIZEJ2dXlyYVozM2pkZGdacEtWQnR3SUQ0MXUwV1RINTMyaDluV2FRTWNjS2p5Nm0rZngzbGNGdEd4cFpLdjFpWUtWK2UyMDZhVVFYUWVHZXlwZEQ0c25MWU93NFY0PQ==/attach/object/K3TLVNAYAAQFQ?|width=693!
>From the figure, it can be seen that the Id arrangement of StringTable in the
>fsimage file has changed from a compact arrangement to a decentralized
>arrangement, that is, USER, GROUP and XATTR are no longer mixed. The
>arrangement is divided into different storage areas and arranged separately.
* With the sub-sections feature introduced in HDFS-14617, Protobuf can support
compatible reading.
* When saving fsimage files in high and low versions, the main difference is
the arrangement of Entry（e.g., USER, GROUP, and XATTR ） in StringTable.
* We will add a conversion tool to convert the Id arrangement of the high
version fsimage file StringTable to a compact arrangement, so that the low
version can be compatible with this format fsimage file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
