[
https://issues.apache.org/jira/browse/HDFS-11382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887022#comment-15887022
]
Hudson commented on HDFS-11382:
-------------------------------
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11314 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11314/])
HDFS-11382. Persist Erasure Coding Policy ID in a new optional field in (wang:
rev 55c07bbed2f475f7b584a86112ee1b6fe0221e98)
* (edit) hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FSImageLoader.java
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/fsimage.proto
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingPolicies.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStripedINodeFile.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsFileStatus.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewerWithStripedBlocks.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileAttributes.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageXmlWriter.java
> Persist Erasure Coding Policy ID in a new optional field in INodeFile in
> FSImage
> --------------------------------------------------------------------------------
>
> Key: HDFS-11382
> URL: https://issues.apache.org/jira/browse/HDFS-11382
> Project: Hadoop HDFS
> Issue Type: Task
> Components: hdfs
> Affects Versions: 3.0.0-alpha1
> Reporter: Manoj Govindassamy
> Assignee: Manoj Govindassamy
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-11382.01.patch, HDFS-11382.02.patch,
> HDFS-11382.03.patch, HDFS-11382.04.patch, HDFS-11382.05.patch
>
>
> For Erasure Coded files, replication field in INodeFile message is re-used
> to store the EC Policy ID.
> *FSDirWriteFileOp#addFile*
> {noformat}
> private static INodesInPath addFile(
> FSDirectory fsd, INodesInPath existing, byte[] localName,
> PermissionStatus permissions, short replication, long
> preferredBlockSize,
> String clientName, String clientMachine)
> throws IOException {
> .. .. ..
> try {
> ErasureCodingPolicy ecPolicy = FSDirErasureCodingOp.
> getErasureCodingPolicy(fsd.getFSNamesystem(), existing);
> if (ecPolicy != null) {
> replication = ecPolicy.getId(); <===
> }
> final BlockType blockType = ecPolicy != null?
> BlockType.STRIPED : BlockType.CONTIGUOUS;
> INodeFile newNode = newINodeFile(fsd.allocateNewInodeId(), permissions,
> modTime, modTime, replication, preferredBlockSize, blockType);
> newNode.setLocalName(localName);
> newNode.toUnderConstruction(clientName, clientMachine);
> newiip = fsd.addINode(existing, newNode, permissions.getPermission());
> {noformat}
> With HDFS-11268 fix, {{FSImageFormatPBINode#Loader#loadInodeFile}} is rightly
> getting the EC ID from the replication field and then uses the right Policy
> to construct the blocks.
> *FSImageFormatPBINode#Loader#loadInodeFile*
> {noformat}
> ErasureCodingPolicy ecPolicy = (blockType == BlockType.STRIPED) ?
> ErasureCodingPolicyManager.getPolicyByPolicyID((byte) replication) :
> null;
> {noformat}
> The original intention was to re-use the replication field so the in-memory
> representation would be compact. But, this isn't necessary for the on-disk
> representation. replication is an optional field, and if we add another
> optional field for the EC policy, it won't be any extra space.
> Also, we need to make sure to have the appropriate asserts in place to make
> sure both fields aren't set for the same INodeField.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
