[jira] [Created] (HDFS-9496) Erasure coding: an erasure codec throughput benchmark tool
Hui Zheng created HDFS-9496: --- Summary: Erasure coding: an erasure codec throughput benchmark tool Key: HDFS-9496 URL: https://issues.apache.org/jira/browse/HDFS-9496 Project: Hadoop HDFS Issue Type: Sub-task Components: erasure-coding, test Reporter: Hui Zheng We need a tool which can help us decide/benchmark an Erasure Codec and schema. Considering HDFS-8968 has implemented an I/O throughput benchmark tool.Maybe we could simply add encode/decode operation to it or implement another tool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9496) Erasure coding: an erasure codec throughput benchmark tool
[ https://issues.apache.org/jira/browse/HDFS-9496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037240#comment-15037240 ] Hui Zheng commented on HDFS-9496: - HADOOP-11588 is good enough to me.Thanks [~lirui]. > Erasure coding: an erasure codec throughput benchmark tool > -- > > Key: HDFS-9496 > URL: https://issues.apache.org/jira/browse/HDFS-9496 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding, test >Reporter: Hui Zheng > > We need a tool which can help us decide/benchmark an Erasure Codec and schema. > Considering HDFS-8968 has implemented an I/O throughput benchmark tool.Maybe > we could simply add encode/decode operation to it or implement another tool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8805) Archival Storage: it is unable to get StoragePolicy of a directory
Hui Zheng created HDFS-8805:
---
Summary: Archival Storage: it is unable to get StoragePolicy of a
directory
Key: HDFS-8805
URL: https://issues.apache.org/jira/browse/HDFS-8805
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Hui Zheng
The result of getStoragePolicy command is always 'unspecified' even we has set
a StoragePolicy on a directory.But the real placement of blocks is correct.
The result of fsck is not correct either.
{code}
$ hdfs storagepolicies -setStoragePolicy -path /tmp/cold -policy COLD
Set storage policy COLD on /tmp/cold
$ hdfs storagepolicies -getStoragePolicy -path /tmp/cold
The storage policy of /tmp/cold is unspecified
$ hdfs fsck -storagepolicies /tmp/cold
Blocks NOT satisfying the specified storage policy:
Storage Policy Specified Storage Policy # of blocks
% of blocks
ARCHIVE:4(COLD) HOT 5
55.5556%
ARCHIVE:3(COLD) HOT 4
44.%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8805) Archival Storage: getStoragePolicy
[
https://issues.apache.org/jira/browse/HDFS-8805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui Zheng updated HDFS-8805:
Summary: Archival Storage: getStoragePolicy (was: Archival Storage: it is
unable to get StoragePolicy of a directory)
Archival Storage: getStoragePolicy
--
Key: HDFS-8805
URL: https://issues.apache.org/jira/browse/HDFS-8805
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover, namenode
Reporter: Hui Zheng
Assignee: Brahma Reddy Battula
Fix For: 2.6.0
The result of getStoragePolicy command is always 'unspecified' even we has
set a StoragePolicy on a directory.But the real placement of blocks is
correct.
The result of fsck is not correct either.
{code}
$ hdfs storagepolicies -setStoragePolicy -path /tmp/cold -policy COLD
Set storage policy COLD on /tmp/cold
$ hdfs storagepolicies -getStoragePolicy -path /tmp/cold
The storage policy of /tmp/cold is unspecified
$ hdfs fsck -storagepolicies /tmp/cold
Blocks NOT satisfying the specified storage policy:
Storage Policy Specified Storage Policy # of blocks
% of blocks
ARCHIVE:4(COLD) HOT 5
55.5556%
ARCHIVE:3(COLD) HOT 4
44.%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8805) Archival Storage: getStoragePolicy should not need superuser privilege
[
https://issues.apache.org/jira/browse/HDFS-8805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14638056#comment-14638056
]
Hui Zheng commented on HDFS-8805:
-
Hi [~jingzhao]
I tried it by super user then found it works well.
So this issue should become getStoragePolicy should not need superuser
privilege.
Archival Storage: getStoragePolicy should not need superuser privilege
--
Key: HDFS-8805
URL: https://issues.apache.org/jira/browse/HDFS-8805
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover, namenode
Reporter: Hui Zheng
Assignee: Brahma Reddy Battula
Fix For: 2.6.0
The result of getStoragePolicy command is always 'unspecified' even we has
set a StoragePolicy on a directory.But the real placement of blocks is
correct.
The result of fsck is not correct either.
{code}
$ hdfs storagepolicies -setStoragePolicy -path /tmp/cold -policy COLD
Set storage policy COLD on /tmp/cold
$ hdfs storagepolicies -getStoragePolicy -path /tmp/cold
The storage policy of /tmp/cold is unspecified
$ hdfs fsck -storagepolicies /tmp/cold
Blocks NOT satisfying the specified storage policy:
Storage Policy Specified Storage Policy # of blocks
% of blocks
ARCHIVE:4(COLD) HOT 5
55.5556%
ARCHIVE:3(COLD) HOT 4
44.%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8805) Archival Storage: getStoragePolicy should not need superuser privilege
[
https://issues.apache.org/jira/browse/HDFS-8805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui Zheng updated HDFS-8805:
Summary: Archival Storage: getStoragePolicy should not need superuser
privilege (was: Archival Storage: getStoragePolicy)
Archival Storage: getStoragePolicy should not need superuser privilege
--
Key: HDFS-8805
URL: https://issues.apache.org/jira/browse/HDFS-8805
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover, namenode
Reporter: Hui Zheng
Assignee: Brahma Reddy Battula
Fix For: 2.6.0
The result of getStoragePolicy command is always 'unspecified' even we has
set a StoragePolicy on a directory.But the real placement of blocks is
correct.
The result of fsck is not correct either.
{code}
$ hdfs storagepolicies -setStoragePolicy -path /tmp/cold -policy COLD
Set storage policy COLD on /tmp/cold
$ hdfs storagepolicies -getStoragePolicy -path /tmp/cold
The storage policy of /tmp/cold is unspecified
$ hdfs fsck -storagepolicies /tmp/cold
Blocks NOT satisfying the specified storage policy:
Storage Policy Specified Storage Policy # of blocks
% of blocks
ARCHIVE:4(COLD) HOT 5
55.5556%
ARCHIVE:3(COLD) HOT 4
44.%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8468) 2 RPC calls for every file read in DFSClient#open(..) resulting in double Audit log entries
[ https://issues.apache.org/jira/browse/HDFS-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14605337#comment-14605337 ] Hui Zheng commented on HDFS-8468: - hi [~vinayrpet] Are you working on this issue? If not please assign to me. 2 RPC calls for every file read in DFSClient#open(..) resulting in double Audit log entries --- Key: HDFS-8468 URL: https://issues.apache.org/jira/browse/HDFS-8468 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Vinayakumar B Assignee: Vinayakumar B In HDFS-7285 branch, To determine whether file is striped/not and get the Schema for the file, 2 RPCs done to Namenode. This is resulting in double audit logs for every file read for both striped/non-striped. This will be a major impact in size of audit logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7398) Reset cached thread-local FSEditLogOp's on every FSEditLog#logEdit
[ https://issues.apache.org/jira/browse/HDFS-7398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Zheng updated HDFS-7398: Fix Version/s: (was: 2.7.0) Reset cached thread-local FSEditLogOp's on every FSEditLog#logEdit -- Key: HDFS-7398 URL: https://issues.apache.org/jira/browse/HDFS-7398 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 2.6.0 Reporter: Gera Shegalov Assignee: Gera Shegalov Attachments: HDFS-7398.v01.patch, HDFS-7398.v02.patch This is a follow-up on HDFS-7385. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7398) Reset cached thread-local FSEditLogOp's on every FSEditLog#logEdit
[ https://issues.apache.org/jira/browse/HDFS-7398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Zheng updated HDFS-7398: Fix Version/s: 2.7.0 Reset cached thread-local FSEditLogOp's on every FSEditLog#logEdit -- Key: HDFS-7398 URL: https://issues.apache.org/jira/browse/HDFS-7398 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 2.6.0 Reporter: Gera Shegalov Assignee: Gera Shegalov Fix For: 2.7.0 Attachments: HDFS-7398.v01.patch, HDFS-7398.v02.patch This is a follow-up on HDFS-7385. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8167) BlockManager.addBlockCollectionWithCheck should check if the block is a striped block
[ https://issues.apache.org/jira/browse/HDFS-8167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502321#comment-14502321 ] Hui Zheng commented on HDFS-8167: - Hi Walter This jira has been committed. Are you working on TestReplicationPolicy? If yes could you correct it? Otherwise I will file another jira to resolve it. -Hui BlockManager.addBlockCollectionWithCheck should check if the block is a striped block - Key: HDFS-8167 URL: https://issues.apache.org/jira/browse/HDFS-8167 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo Nicholas Sze Assignee: Hui Zheng Attachments: HDFS-8167_001.patch, HDFS-8167_002.patch This JIRA is to address [~zhz]'s [comment|https://issues.apache.org/jira/browse/HDFS-7994?focusedCommentId=14498894page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14498894]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7994) Detect if resevered EC Block ID is already used
[ https://issues.apache.org/jira/browse/HDFS-7994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499105#comment-14499105 ] Hui Zheng commented on HDFS-7994: - Thanks [~zhz] and [~szetszwo] I will work on [HDFS-8167|https://issues.apache.org/jira/browse/HDFS-8167] Detect if resevered EC Block ID is already used --- Key: HDFS-7994 URL: https://issues.apache.org/jira/browse/HDFS-7994 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo Nicholas Sze Assignee: Hui Zheng Fix For: HDFS-7285 Attachments: HDFS-7994_001.patch, HDFS-7994_002.patch Since random block IDs were supported by some early version of HDFS, the block ID reserved for EC blocks could be already used by some existing blocks in a cluster. During NameNode startup, it detects if there are reserved EC block IDs used by non-EC blocks. If it is the case, NameNode will do an additional blocksMap lookup when there is a miss in a blockGroupsMap lookup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8167) BlockManager.addBlockCollectionWithCheck should check if the block is a striped block
[ https://issues.apache.org/jira/browse/HDFS-8167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499283#comment-14499283 ] Hui Zheng commented on HDFS-8167: - Seems to be a little weird to have a setter and a getter with the same name. ... hasNonEcBlockUsingStripedID(true) is only used it in BlockManager. Let's simply use hasNonEcBlockUsingStripedID = true. Yes,I will delete the method. But for test I will keep the getter method hasNonEcBlockUsingStripedID(). Since we plan to support EC with non-striping layout in phase II, non-EC blocks should be contiguous blocks. We can use an abbreviation of contiguous in naming Let's change the name at that time. hasNonEcBlockUsingStripedID is consistent with BlockIdManager.isStripedBlockID(..). I agree let it until that time. BTW I think maybe hasNocEcBlockUsingEcBlockID is good. But I think we also need some BlockIdManager.isECBlockID() and Block.isEcBlock().It is a bit complicated in this jira. ... block passed to addBlockCollectionWithCheck is a striped block ... Good point. We should also check !block.isStriped(). I agree. BlockManager.addBlockCollectionWithCheck should check if the block is a striped block - Key: HDFS-8167 URL: https://issues.apache.org/jira/browse/HDFS-8167 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Tsz Wo Nicholas Sze Assignee: Hui Zheng This JIRA is to address [~zhz]'s [comment|https://issues.apache.org/jira/browse/HDFS-7994?focusedCommentId=14498894page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14498894]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8167) BlockManager.addBlockCollectionWithCheck should check if the block is a striped block
[ https://issues.apache.org/jira/browse/HDFS-8167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Zheng updated HDFS-8167: Attachment: HDFS-8167_001.patch I attached a patch. BlockManager.addBlockCollectionWithCheck should check if the block is a striped block - Key: HDFS-8167 URL: https://issues.apache.org/jira/browse/HDFS-8167 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Tsz Wo Nicholas Sze Assignee: Hui Zheng Attachments: HDFS-8167_001.patch This JIRA is to address [~zhz]'s [comment|https://issues.apache.org/jira/browse/HDFS-7994?focusedCommentId=14498894page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14498894]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8167) BlockManager.addBlockCollectionWithCheck should check if the block is a striped block
[
https://issues.apache.org/jira/browse/HDFS-8167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499555#comment-14499555
]
Hui Zheng commented on HDFS-8167:
-
This is because the test case used a StripedId(which 0) for a
BlockInfoContiguous.
After [HDFS-7994|https://issues.apache.org/jira/browse/HDFS-7994] ,it will fail
if the BlockManager doesn't correctly initialize the
hasNonEcBlockUsingStripedID by loading fsimage or editlog.
I will update this patch as following:
{code}
@@ -1202,7 +1202,7 @@ public void
testAddStoredBlockDoesNotCauseSkippedReplication()
BlockManager bm = new BlockManager(mockNS, new HdfsConfiguration());
UnderReplicatedBlocks underReplicatedBlocks = bm.neededReplications;
-BlockInfo block1 = genBlockInfo(random.nextLong());
+ BlockInfo block1 = genBlockInfo(Math.abs(random.nextLong()));
BlockInfo block2 = genBlockInfo(random.nextLong());
{code}
BlockManager.addBlockCollectionWithCheck should check if the block is a
striped block
-
Key: HDFS-8167
URL: https://issues.apache.org/jira/browse/HDFS-8167
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Tsz Wo Nicholas Sze
Assignee: Hui Zheng
Attachments: HDFS-8167_001.patch
This JIRA is to address [~zhz]'s
[comment|https://issues.apache.org/jira/browse/HDFS-7994?focusedCommentId=14498894page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14498894].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7994) Detect if resevered EC Block ID is already used
[ https://issues.apache.org/jira/browse/HDFS-7994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Zheng updated HDFS-7994: Attachment: HDFS-7994_002.patch Thanks [~szetszwo] for reviewing! Consider we are using BlockManager to manage hasNonEcBlockUsingStripID,I added a method addBlockCollectionWithCheck which do some additional check and update the hasNonEcBlockUsingStripID and reuse addBlockCollection into BlockManager. Detect if resevered EC Block ID is already used --- Key: HDFS-7994 URL: https://issues.apache.org/jira/browse/HDFS-7994 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo Nicholas Sze Assignee: Hui Zheng Attachments: HDFS-7994_001.patch, HDFS-7994_002.patch Since random block IDs were supported by some early version of HDFS, the block ID reserved for EC blocks could be already used by some existing blocks in a cluster. During NameNode startup, it detects if there are reserved EC block IDs used by non-EC blocks. If it is the case, NameNode will do an additional blocksMap lookup when there is a miss in a blockGroupsMap lookup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7994) Detect if resevered EC Block ID is already used
[ https://issues.apache.org/jira/browse/HDFS-7994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Zheng updated HDFS-7994: Attachment: HDFS-7994_001.patch Hi Nicholas Thank you for clarification. I attached a patch for the purpose. Detect if resevered EC Block ID is already used --- Key: HDFS-7994 URL: https://issues.apache.org/jira/browse/HDFS-7994 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo Nicholas Sze Assignee: Hui Zheng Attachments: HDFS-7994_001.patch Since random block IDs were supported by some early version of HDFS, the block ID reserved for EC blocks could be already used by some existing blocks in a cluster. During NameNode startup, it detects if there are reserved EC block IDs used by non-EC blocks. If it is the case, NameNode will do an additional blocksMap lookup when there is a miss in a blockGroupsMap lookup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7994) Detect if resevered EC Block ID is already used
[ https://issues.apache.org/jira/browse/HDFS-7994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490659#comment-14490659 ] Hui Zheng commented on HDFS-7994: - Suppose that there are some blocks using StripedBlockID when namenode loads fsimage or editlog,so the hasNonEcBlockUsingStripID is true. But after some time user may delete some or all of these blocks by deleting files. If the number of these blocks is 0, the hasNonEcBlockUsingStripID will become false. Detect if resevered EC Block ID is already used --- Key: HDFS-7994 URL: https://issues.apache.org/jira/browse/HDFS-7994 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo Nicholas Sze Assignee: Hui Zheng Since random block IDs were supported by some early version of HDFS, the block ID reserved for EC blocks could be already used by some existing blocks in a cluster. During NameNode startup, it detects if there are reserved EC block IDs used by non-EC blocks. If it is the case, NameNode will do an additional blocksMap lookup when there is a miss in a blockGroupsMap lookup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8109) ECManager should be able to manage multiple ECSchemas
Hui Zheng created HDFS-8109: --- Summary: ECManager should be able to manage multiple ECSchemas Key: HDFS-8109 URL: https://issues.apache.org/jira/browse/HDFS-8109 Project: Hadoop HDFS Issue Type: Improvement Reporter: Hui Zheng [HDFS-8074|https://issues.apache.org/jira/browse/HDFS-8074] has implemented a default EC Schema. But a user may use another predefined schema when he creates an EC zone. Maybe we need to implement to get a ECSchema from ECManager by its schema name. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8109) ECManager should be able to manage multiple ECSchemas
[ https://issues.apache.org/jira/browse/HDFS-8109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Zheng updated HDFS-8109: Issue Type: Sub-task (was: Improvement) Parent: HDFS-8031 ECManager should be able to manage multiple ECSchemas - Key: HDFS-8109 URL: https://issues.apache.org/jira/browse/HDFS-8109 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Hui Zheng [HDFS-8074|https://issues.apache.org/jira/browse/HDFS-8074] has implemented a default EC Schema. But a user may use another predefined schema when he creates an EC zone. Maybe we need to implement to get a ECSchema from ECManager by its schema name. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7994) Detect if resevered EC Block ID is already used
[
https://issues.apache.org/jira/browse/HDFS-7994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14487162#comment-14487162
]
Hui Zheng commented on HDFS-7994:
-
Hi [~szetszwo]
I want to confirm my understanding about this issue.Is it correct?
When the NameNode loads a fsimage,it doesn't care about whether the block is an
EC Block during putting the blocks into the BlocksMap.
But it should take different lookup actions for non-EC/EC blocks during
processing BlockReport.
For non-EC blocks it only does a lookup in the blocksmap by blockid.
For EC blocks it must convert the blockid to EC-Blockid(the blockid of the
first block of a EC block group).
I think the current implementation has complete this function.
{code}
public BlockInfo getStoredBlock(Block block) {
BlockInfo info = null;
if (BlockIdManager.isStripedBlockID(block.getBlockId())) {
info = blocksMap.getStoredBlock(
new Block(BlockIdManager.convertToStripedID(block.getBlockId(;
//AAA
}
if (info == null) {
info = blocksMap.getStoredBlock(block); // BBB
}
return info;
}
{code}
1. A StripedBlockId(whether or not the low 4 bits are not '') comes and the
block is really EC block, it can be found by AAA.
2. A StripedBlockId whose low 4 bits are '' comes but the block is not EC,
it can also be found by AAA.
3. A StripedBlockId whose low 4 bits are NOT '' comes but the block is not
EC, it can be found by BBB.
3. A non-StripedBlockId comes,it can be found by BBB.
So I don't think we need to process this issue.
Detect if resevered EC Block ID is already used
---
Key: HDFS-7994
URL: https://issues.apache.org/jira/browse/HDFS-7994
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Reporter: Tsz Wo Nicholas Sze
Assignee: Hui Zheng
Since random block IDs were supported by some early version of HDFS, the
block ID reserved for EC blocks could be already used by some existing blocks
in a cluster. During NameNode startup, it detects if there are reserved EC
block IDs used by non-EC blocks. If it is the case, NameNode will do an
additional blocksMap lookup when there is a miss in a blockGroupsMap lookup.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HDFS-7994) Detect if resevered EC Block ID is already used
[ https://issues.apache.org/jira/browse/HDFS-7994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Zheng reassigned HDFS-7994: --- Assignee: Hui Zheng (was: Tsz Wo Nicholas Sze) Detect if resevered EC Block ID is already used --- Key: HDFS-7994 URL: https://issues.apache.org/jira/browse/HDFS-7994 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Tsz Wo Nicholas Sze Assignee: Hui Zheng Since random block IDs were supported by some early version of HDFS, the block ID reserved for EC blocks could be already used by some existing blocks in a cluster. During NameNode startup, it detects if there are reserved EC block IDs used by non-EC blocks. If it is the case, NameNode will do an additional blocksMap lookup when there is a miss in a blockGroupsMap lookup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7994) Detect if resevered EC Block ID is already used
[ https://issues.apache.org/jira/browse/HDFS-7994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484823#comment-14484823 ] Hui Zheng commented on HDFS-7994: - Hi Nicholas I would like to work on this issue. Detect if resevered EC Block ID is already used --- Key: HDFS-7994 URL: https://issues.apache.org/jira/browse/HDFS-7994 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Tsz Wo Nicholas Sze Assignee: Hui Zheng Since random block IDs were supported by some early version of HDFS, the block ID reserved for EC blocks could be already used by some existing blocks in a cluster. During NameNode startup, it detects if there are reserved EC block IDs used by non-EC blocks. If it is the case, NameNode will do an additional blocksMap lookup when there is a miss in a blockGroupsMap lookup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7617) Add unit tests for editlog transactions for EC
[ https://issues.apache.org/jira/browse/HDFS-7617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14481024#comment-14481024 ] Hui Zheng commented on HDFS-7617: - I am sorry for late reply. Add unit tests for editlog transactions for EC -- Key: HDFS-7617 URL: https://issues.apache.org/jira/browse/HDFS-7617 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo Nicholas Sze Assignee: Hui Zheng Fix For: HDFS-7285 Attachments: HDFS-7617.001.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7617) Add unit tests for editlog transactions for EC
[ https://issues.apache.org/jira/browse/HDFS-7617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14481023#comment-14481023 ] Hui Zheng commented on HDFS-7617: - I am sorry for reply.Thanks [~zhz]. Add unit tests for editlog transactions for EC -- Key: HDFS-7617 URL: https://issues.apache.org/jira/browse/HDFS-7617 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo Nicholas Sze Assignee: Hui Zheng Fix For: HDFS-7285 Attachments: HDFS-7617.001.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode
[
https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377756#comment-14377756
]
Hui Zheng commented on HDFS-7980:
-
Hi [~walter.k.su]
dfs.blockreport.initailDelay is 600 (10minutes).
dfs.blockreport.intervalmsec is 3600 (10 hours).
Incremental BlockReport will dramatically slow down the startup of a namenode
--
Key: HDFS-7980
URL: https://issues.apache.org/jira/browse/HDFS-7980
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Hui Zheng
Assignee: Walter Su
In the current implementation the datanode will call the
reportReceivedDeletedBlocks() method that is a IncrementalBlockReport before
calling the bpNamenode.blockReport() method. So in a large(several thousands
of datanodes) and busy cluster it will slow down(more than one hour) the
startup of namenode.
{code}
ListDatanodeCommand blockReport() throws IOException {
// send block report if timer has expired.
final long startTime = now();
if (startTime - lastBlockReport = dnConf.blockReportInterval) {
return null;
}
final ArrayListDatanodeCommand cmds = new ArrayListDatanodeCommand();
// Flush any block information that precedes the block report. Otherwise
// we have a chance that we will miss the delHint information
// or we will report an RBW replica after the BlockReport already reports
// a FINALIZED one.
reportReceivedDeletedBlocks();
lastDeletedReport = startTime;
.
// Send the reports to the NN.
int numReportsSent = 0;
int numRPCs = 0;
boolean success = false;
long brSendStartTime = now();
try {
if (totalBlockCount dnConf.blockReportSplitThreshold) {
// Below split threshold, send all reports in a single message.
DatanodeCommand cmd = bpNamenode.blockReport(
bpRegistration, bpos.getBlockPoolId(), reports);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode
Hui Zheng created HDFS-7980:
---
Summary: Incremental BlockReport will dramatically slow down the
startup of a namenode
Key: HDFS-7980
URL: https://issues.apache.org/jira/browse/HDFS-7980
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Hui Zheng
In the current implementation the datanode will call the
reportReceivedDeletedBlocks() method that is a IncrementalBlockReport before
calling the bpNamenode.blockReport() method. So in a large(several thousands of
datanodes) and busy cluster it will slow down(more than one hour) the startup
of namenode.
{code}
ListDatanodeCommand blockReport() throws IOException {
// send block report if timer has expired.
final long startTime = now();
if (startTime - lastBlockReport = dnConf.blockReportInterval) {
return null;
}
final ArrayListDatanodeCommand cmds = new ArrayListDatanodeCommand();
// Flush any block information that precedes the block report. Otherwise
// we have a chance that we will miss the delHint information
// or we will report an RBW replica after the BlockReport already reports
// a FINALIZED one.
reportReceivedDeletedBlocks();
lastDeletedReport = startTime;
.
// Send the reports to the NN.
int numReportsSent = 0;
int numRPCs = 0;
boolean success = false;
long brSendStartTime = now();
try {
if (totalBlockCount dnConf.blockReportSplitThreshold) {
// Below split threshold, send all reports in a single message.
DatanodeCommand cmd = bpNamenode.blockReport(
bpRegistration, bpos.getBlockPoolId(), reports);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode
[
https://issues.apache.org/jira/browse/HDFS-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377689#comment-14377689
]
Hui Zheng commented on HDFS-7980:
-
In our environment there are over 3000 datanodes, 100 million file and 150
million blocks.
When all the datanodes are working, it will take more than one hour to restart
the namenode(the time is almost used for block report).
But if we
first stop all the datanodes,
then restart the namenode,
last start all the datanodes after the namenode finishes loading editlog,
it will only take around 20 minutes.
Incremental BlockReport will dramatically slow down the startup of a namenode
--
Key: HDFS-7980
URL: https://issues.apache.org/jira/browse/HDFS-7980
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Hui Zheng
Assignee: Walter Su
In the current implementation the datanode will call the
reportReceivedDeletedBlocks() method that is a IncrementalBlockReport before
calling the bpNamenode.blockReport() method. So in a large(several thousands
of datanodes) and busy cluster it will slow down(more than one hour) the
startup of namenode.
{code}
ListDatanodeCommand blockReport() throws IOException {
// send block report if timer has expired.
final long startTime = now();
if (startTime - lastBlockReport = dnConf.blockReportInterval) {
return null;
}
final ArrayListDatanodeCommand cmds = new ArrayListDatanodeCommand();
// Flush any block information that precedes the block report. Otherwise
// we have a chance that we will miss the delHint information
// or we will report an RBW replica after the BlockReport already reports
// a FINALIZED one.
reportReceivedDeletedBlocks();
lastDeletedReport = startTime;
.
// Send the reports to the NN.
int numReportsSent = 0;
int numRPCs = 0;
boolean success = false;
long brSendStartTime = now();
try {
if (totalBlockCount dnConf.blockReportSplitThreshold) {
// Below split threshold, send all reports in a single message.
DatanodeCommand cmd = bpNamenode.blockReport(
bpRegistration, bpos.getBlockPoolId(), reports);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7617) Add editlog transactions for EC
[ https://issues.apache.org/jira/browse/HDFS-7617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Zheng updated HDFS-7617: Attachment: HDFS-7617.001.patch Add editlog transactions for EC --- Key: HDFS-7617 URL: https://issues.apache.org/jira/browse/HDFS-7617 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo Nicholas Sze Assignee: Hui Zheng Attachments: HDFS-7617.001.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7617) Add editlog transactions for EC
[ https://issues.apache.org/jira/browse/HDFS-7617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375569#comment-14375569 ] Hui Zheng commented on HDFS-7617: - Thanks [~zhz] and [~jingzhao] The implement of this function in HDFS-7749 is good to me.So I simply add two testcases for this jira. Add editlog transactions for EC --- Key: HDFS-7617 URL: https://issues.apache.org/jira/browse/HDFS-7617 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo Nicholas Sze Assignee: Hui Zheng Attachments: HDFS-7617.001.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375309#comment-14375309 ] Hui Zheng commented on HDFS-7827: - Hi Jing Thank you for your help.The 004 patch looks good to me. Also I have known where is not following the coding convention by the patch. Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng Attachments: HDFS-7827.000.patch, HDFS-7827.002.patch, HDFS-7827.003.patch, HDFS-7827.004.patch HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work stopped] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-7827 stopped by Hui Zheng. --- Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng Attachments: HDFS-7827.000.patch, HDFS-7827.002.patch, HDFS-7827.003.patch, HDFS-7827.004.patch HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7617) Add editlog transactions for EC
[ https://issues.apache.org/jira/browse/HDFS-7617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366788#comment-14366788 ] Hui Zheng commented on HDFS-7617: - Hi Jing Are you going to move the dataBlockNum and parityBlockNum from BlockInfoStriped to the XAttr of INode? If we can get the informations from the XAttr, it will be able to save much memory instead of saving them per BlockInfoStriped. Add editlog transactions for EC --- Key: HDFS-7617 URL: https://issues.apache.org/jira/browse/HDFS-7617 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo Nicholas Sze Assignee: Hui Zheng -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HDFS-7617) Add editlog transactions for EC
[ https://issues.apache.org/jira/browse/HDFS-7617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-7617 started by Hui Zheng. --- Add editlog transactions for EC --- Key: HDFS-7617 URL: https://issues.apache.org/jira/browse/HDFS-7617 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo Nicholas Sze Assignee: Hui Zheng -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7617) Add editlog transactions for EC
[ https://issues.apache.org/jira/browse/HDFS-7617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366768#comment-14366768 ] Hui Zheng commented on HDFS-7617: - [HDFS-7749|https://issues.apache.org/jira/browse/HDFS-7749] has already added the functionality in editlog for striped blocks.But it doesn't serialize the dataBlockNum and parityBlockNum . So we need to add it. Add editlog transactions for EC --- Key: HDFS-7617 URL: https://issues.apache.org/jira/browse/HDFS-7617 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo Nicholas Sze Assignee: Hui Zheng -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-7827 started by Hui Zheng. --- Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng Attachments: HDFS-7827.000.patch, HDFS-7827.002.patch, HDFS-7827.003.patch HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7949) WebImageViewer need support file size calculation with striped blocks
Hui Zheng created HDFS-7949: --- Summary: WebImageViewer need support file size calculation with striped blocks Key: HDFS-7949 URL: https://issues.apache.org/jira/browse/HDFS-7949 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Hui Zheng Priority: Minor The file size calculation should be changed when the blocks of the file are striped in WebImageViewer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Zheng updated HDFS-7827: Attachment: (was: HDFS-7827.001.patch) Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng Attachments: HDFS-7827.000.patch, HDFS-7827.002.patch HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Zheng updated HDFS-7827: Attachment: (was: HDFS-7285.003.patch) Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng Attachments: HDFS-7827.000.patch, HDFS-7827.002.patch HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Zheng updated HDFS-7827: Attachment: HDFS-7827.003.patch Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng Attachments: HDFS-7827.000.patch, HDFS-7827.002.patch, HDFS-7827.003.patch HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[
https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui Zheng updated HDFS-7827:
Attachment: HDFS-7285.003.patch
Hi Jing
I updated the patch,please review it.
1) Corrected the
{code}writeINodeUnderConstruction/readINodeUnderConstruction{code} method
2. Added a testcase for save/locad a INodeFileUnderConstruction.
3. Corrected the coding format. But I'm not sure whether they are all correct.
If there are still code that is not according to the conventions please tell me
the details.
4. About WebImageViewer, I create a new
[HDFS-7949|https://issues.apache.org/jira/browse/HDFS-7949]
Erasure Coding: support striped blocks in non-protobuf fsimage
--
Key: HDFS-7827
URL: https://issues.apache.org/jira/browse/HDFS-7827
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Jing Zhao
Assignee: Hui Zheng
Attachments: HDFS-7285.003.patch, HDFS-7827.000.patch,
HDFS-7827.001.patch, HDFS-7827.002.patch
HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage.
We should also add this support to the non-protobuf fsimage since it is still
used for use cases like offline image processing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14362853#comment-14362853 ] Hui Zheng commented on HDFS-7827: - I have add a testcase for saving and loading a INodeFile. Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng Attachments: HDFS-7827.000.patch, HDFS-7827.001.patch, HDFS-7827.002.patch HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Zheng updated HDFS-7827: Attachment: HDFS-7827.002.patch Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng Attachments: HDFS-7827.000.patch, HDFS-7827.001.patch, HDFS-7827.002.patch HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14362863#comment-14362863 ] Hui Zheng commented on HDFS-7827: - Thanks [~wheat9] Is there any plan to remove the non-protobuf fsimage? Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng Attachments: HDFS-7827.000.patch, HDFS-7827.001.patch, HDFS-7827.002.patch HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Zheng updated HDFS-7827: Attachment: HDFS-7827.001.patch Hi Jing I updated the patch. But how should we test it without EC client? Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng Attachments: HDFS-7827.000.patch, HDFS-7827.001.patch HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HDFS-7617) Add editlog transactions for EC
[ https://issues.apache.org/jira/browse/HDFS-7617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Zheng reassigned HDFS-7617: --- Assignee: Hui Zheng (was: Hui Zheng) Add editlog transactions for EC --- Key: HDFS-7617 URL: https://issues.apache.org/jira/browse/HDFS-7617 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Tsz Wo Nicholas Sze Assignee: Hui Zheng -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358398#comment-14358398 ] Hui Zheng commented on HDFS-7827: - Hi Jing I made a mistake and found that the INodeFile does not serialize its header.So we need to record the type of the blocks in the fsimage as you said. Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng Attachments: HDFS-7827.000.patch HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356515#comment-14356515 ] Hui Zheng commented on HDFS-7827: - Hi Jing I agree that we need a flag to identify the striped block, but could we use the STORAGE_POLICY_ID in header of INodeFile? Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng Attachments: HDFS-7827.000.patch HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354549#comment-14354549 ] Hui Zheng commented on HDFS-7827: - I think we don't need to implement the loading of a non-protobuf fsimage in Namenode,so it is not nessary to implement readFields method ofthe Writable inface. But we need to support it in tools(ImageLoaderCurrent.java). Is that right? Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng Attachments: HDFS-7827.000.patch HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Zheng updated HDFS-7827: Attachment: HDFS-7827.000.patch Hi Jing I upload the patch which override the write method of BlockInfoStriped class. Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng Attachments: HDFS-7827.000.patch HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352435#comment-14352435 ] Hui Zheng commented on HDFS-7827: - To support striped block in no-protobuf fsimage we only need to persist striped blocks in non-protobuf format. The BlockInfo class uses writable interface to persist itself in non-protobuf format. So we can override the write method to implement it. Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HDFS-7749) Erasure Coding: Add striped block support in INodeFile
[ https://issues.apache.org/jira/browse/HDFS-7749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Zheng reassigned HDFS-7749: --- Assignee: Hui Zheng (was: Jing Zhao) Erasure Coding: Add striped block support in INodeFile -- Key: HDFS-7749 URL: https://issues.apache.org/jira/browse/HDFS-7749 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Hui Zheng Attachments: HDFS-7749.000.patch This jira plan to add a new INodeFile feature to store the stripped blocks information in case that the INodeFile is erasure coded. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[ https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14336117#comment-14336117 ] Hui Zheng commented on HDFS-7827: - Hi Jing I would like to work on this jira. Could you assign it to me? Erasure Coding: support striped blocks in non-protobuf fsimage -- Key: HDFS-7827 URL: https://issues.apache.org/jira/browse/HDFS-7827 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Jing Zhao HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage. We should also add this support to the non-protobuf fsimage since it is still used for use cases like offline image processing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
