[jira] [Commented] (HDFS-11687) Add new public encryption APIs required by Hive
[ https://issues.apache.org/jira/browse/HDFS-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15980659#comment-15980659 ] Rui Li commented on HDFS-11687: --- Thanks [~ngangam]. Besides isHDFSEncryptionEnabled, we also need APIs related to KeyProvider. [~Ferd], [~xuefuz] I think you know better about this. > Add new public encryption APIs required by Hive > --- > > Key: HDFS-11687 > URL: https://issues.apache.org/jira/browse/HDFS-11687 > Project: Hadoop HDFS > Issue Type: Improvement > Components: encryption >Affects Versions: 2.6.5 >Reporter: Andrew Wang > > As discovered on HADOOP-14333, Hive is using reflection to get a DFSClient > for its encryption shim. We should provide proper public APIs for getting > this information. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9962) Erasure Coding: need a way to test multiple EC policies
[ https://issues.apache.org/jira/browse/HDFS-9962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935670#comment-15935670 ] Rui Li commented on HDFS-9962: -- [~tasanuma0829], feel free to take the JIRA. Thanks. > Erasure Coding: need a way to test multiple EC policies > --- > > Key: HDFS-9962 > URL: https://issues.apache.org/jira/browse/HDFS-9962 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rui Li > Labels: hdfs-ec-3.0-nice-to-have > > Now that we support multiple EC policies, we need a way test it to catch > potential issues. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9603) Erasure Coding: Use ErasureCoder to encode/decode a block group
[
https://issues.apache.org/jira/browse/HDFS-9603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15345826#comment-15345826
]
Rui Li commented on HDFS-9603:
--
Hi [~drankye], per our discussion, Wei Zhou would help with this task. But I
can't re-assign the JIRA. Please feel free to assign it to Wei if you can edit.
> Erasure Coding: Use ErasureCoder to encode/decode a block group
> ---
>
> Key: HDFS-9603
> URL: https://issues.apache.org/jira/browse/HDFS-9603
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HDFS-9603.1.patch, HDFS-9603.2.patch, HDFS-9603.3.patch
>
>
> According to design, {{ErasureCoder}} is responsible to encode/decode a block
> group. Currently however, we directly use {{RawErasureCoder}} to do the work,
> e.g. in {{DFSStripedOutputStream}}. This task attempts to encapsulate
> {{RawErasureCoder}} to comply with the design.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9962) Erasure Coding: need a way to test multiple EC policies
[ https://issues.apache.org/jira/browse/HDFS-9962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15194788#comment-15194788 ] Rui Li commented on HDFS-9962: -- As I proposed in HDFS-7866, one possible solution is to randomly choose a policy in each test. Kai also suggested to make sure the default policy always gets tested. [~drankye], [~zhz], let's use this JIRA to reach an agreement. Would like to know your opinions. Thanks. > Erasure Coding: need a way to test multiple EC policies > --- > > Key: HDFS-9962 > URL: https://issues.apache.org/jira/browse/HDFS-9962 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li > > Now that we support multiple EC policies, we need a way test it to catch > potential issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9962) Erasure Coding: need a way to test multiple EC policies
Rui Li created HDFS-9962: Summary: Erasure Coding: need a way to test multiple EC policies Key: HDFS-9962 URL: https://issues.apache.org/jira/browse/HDFS-9962 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Rui Li Assignee: Rui Li Now that we support multiple EC policies, we need a way test it to catch potential issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages multiple erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186623#comment-15186623 ] Rui Li commented on HDFS-7866: -- Thanks guys for the review! > Erasure coding: NameNode manages multiple erasure coding policies > - > > Key: HDFS-7866 > URL: https://issues.apache.org/jira/browse/HDFS-7866 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > Fix For: 3.0.0 > > Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, > HDFS-7866-v3.patch, HDFS-7866.10.patch, HDFS-7866.11.patch, > HDFS-7866.12.patch, HDFS-7866.13.patch, HDFS-7866.4.patch, HDFS-7866.5.patch, > HDFS-7866.6.patch, HDFS-7866.7.patch, HDFS-7866.8.patch, HDFS-7866.9.patch > > > This is to extend NameNode to load, list and sync predefine EC schemas in > authorized and controlled approach. The provided facilities will be used to > implement DFSAdmin commands so admin can list available EC schemas, then > could choose some of them for target EC zones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages multiple erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186314#comment-15186314 ] Rui Li commented on HDFS-7866: -- Thanks [~jingzhao] and [~zhz] for the great discussions. So we're good to leave the field as required right? I also checked the test failures and they cannot be reproduced locally. > Erasure coding: NameNode manages multiple erasure coding policies > - > > Key: HDFS-7866 > URL: https://issues.apache.org/jira/browse/HDFS-7866 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, > HDFS-7866-v3.patch, HDFS-7866.10.patch, HDFS-7866.11.patch, > HDFS-7866.12.patch, HDFS-7866.13.patch, HDFS-7866.4.patch, HDFS-7866.5.patch, > HDFS-7866.6.patch, HDFS-7866.7.patch, HDFS-7866.8.patch, HDFS-7866.9.patch > > > This is to extend NameNode to load, list and sync predefine EC schemas in > authorized and controlled approach. The provided facilities will be used to > implement DFSAdmin commands so admin can list available EC schemas, then > could choose some of them for target EC zones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7866) Erasure coding: NameNode manages multiple erasure coding policies
[
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HDFS-7866:
-
Attachment: HDFS-7866.13.patch
I realized that {{TestEditLog}} fails on trunk as well. So it's not related to
the patch here.
The v13 patch mainly addresses Walter's comments:
# Adds the JavaDoc.
# Moves LAYOUT_BIT_WIDTH and MAX_REDUNDANCY to {{HeaderFormat}}.
# Adds method {{getECPolicyID}} to {{HeaderFormat}} so that we don't have to
reuse {{getReplication}} to get the policy id. Currently we only have setter
for replication but not for EC policy, so don't need to take care of the setter.
# Updates {{TestErasureCodingPolicies#testReplication}} to make sure setting
replication on EC file doesn't mess up with the policy.
> Erasure coding: NameNode manages multiple erasure coding policies
> -
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch,
> HDFS-7866-v3.patch, HDFS-7866.10.patch, HDFS-7866.11.patch,
> HDFS-7866.12.patch, HDFS-7866.13.patch, HDFS-7866.4.patch, HDFS-7866.5.patch,
> HDFS-7866.6.patch, HDFS-7866.7.patch, HDFS-7866.8.patch, HDFS-7866.9.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in
> authorized and controlled approach. The provided facilities will be used to
> implement DFSAdmin commands so admin can list available EC schemas, then
> could choose some of them for target EC zones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages multiple erasure coding policies
[
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184300#comment-15184300
]
Rui Li commented on HDFS-7866:
--
Thanks Walter for the suggestions.
bq. separating the logic of manipulation of the 12-bits. 2 sets of set/get
methods for them.
It makes sense to have 2 separate sets of get/set methods. But if we do it
here, we'll be just repeating the code. What do you think?
bq. We are sure policyID is strictly <=7-bits right?
I think 128 policies is enough for now, and we can just consider negative ID as
invalid. If we have more policies in the future, we can support negative IDs
which means we can have 256 different IDs. Then we can think about what to
return if we call getErasureCodingPolicyID for non-EC files. Maybe throw an
exception?
[~zhz], please hold on the patch. The latest {{TestEditLog}} failure may be
related. I'll look into it.
> Erasure coding: NameNode manages multiple erasure coding policies
> -
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch,
> HDFS-7866-v3.patch, HDFS-7866.10.patch, HDFS-7866.11.patch,
> HDFS-7866.12.patch, HDFS-7866.4.patch, HDFS-7866.5.patch, HDFS-7866.6.patch,
> HDFS-7866.7.patch, HDFS-7866.8.patch, HDFS-7866.9.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in
> authorized and controlled approach. The provided facilities will be used to
> implement DFSAdmin commands so admin can list available EC schemas, then
> could choose some of them for target EC zones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7866) Erasure coding: NameNode manages multiple erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HDFS-7866: - Attachment: HDFS-7866.12.patch Thanks Zhe for the clarifications! Patch updated to address comments. > Erasure coding: NameNode manages multiple erasure coding policies > - > > Key: HDFS-7866 > URL: https://issues.apache.org/jira/browse/HDFS-7866 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, > HDFS-7866-v3.patch, HDFS-7866.10.patch, HDFS-7866.11.patch, > HDFS-7866.12.patch, HDFS-7866.4.patch, HDFS-7866.5.patch, HDFS-7866.6.patch, > HDFS-7866.7.patch, HDFS-7866.8.patch, HDFS-7866.9.patch > > > This is to extend NameNode to load, list and sync predefine EC schemas in > authorized and controlled approach. The provided facilities will be used to > implement DFSAdmin commands so admin can list available EC schemas, then > could choose some of them for target EC zones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages multiple erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179541#comment-15179541 ] Rui Li commented on HDFS-7866: -- Thanks Zhe for your comments. 1. We're using 11 bits to store the policy ID, which means it could excess a byte (replication factor is also a short). What do you think? 2. This is to make the tests use the new policy to find any potential issues. If there's no further comments, I'll revert it to RS-6-3 in next update. I also would like to know your opinions about randomly choosing a policy in the tests. You can refer to my previous [discussions|https://issues.apache.org/jira/browse/HDFS-7866?focusedCommentId=15171320&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15171320] with Kai. 3. I agree we can do this in a follow-on. I think essentially it's hacky because we allow user to specify the replication factor when creating a file but then overwrite it to store the EC policy ID. Fixing this requires changing the API, which I think is unacceptable. > Erasure coding: NameNode manages multiple erasure coding policies > - > > Key: HDFS-7866 > URL: https://issues.apache.org/jira/browse/HDFS-7866 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, > HDFS-7866-v3.patch, HDFS-7866.10.patch, HDFS-7866.11.patch, > HDFS-7866.4.patch, HDFS-7866.5.patch, HDFS-7866.6.patch, HDFS-7866.7.patch, > HDFS-7866.8.patch, HDFS-7866.9.patch > > > This is to extend NameNode to load, list and sync predefine EC schemas in > authorized and controlled approach. The provided facilities will be used to > implement DFSAdmin commands so admin can list available EC schemas, then > could choose some of them for target EC zones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages multiple erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177063#comment-15177063 ] Rui Li commented on HDFS-7866: -- Test failures cannot be reproduced. I'll fix the checkstyle in next update. (we need at least revert the default policy back to RS-6-3 and make sure the tests pass) > Erasure coding: NameNode manages multiple erasure coding policies > - > > Key: HDFS-7866 > URL: https://issues.apache.org/jira/browse/HDFS-7866 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, > HDFS-7866-v3.patch, HDFS-7866.10.patch, HDFS-7866.11.patch, > HDFS-7866.4.patch, HDFS-7866.5.patch, HDFS-7866.6.patch, HDFS-7866.7.patch, > HDFS-7866.8.patch, HDFS-7866.9.patch > > > This is to extend NameNode to load, list and sync predefine EC schemas in > authorized and controlled approach. The provided facilities will be used to > implement DFSAdmin commands so admin can list available EC schemas, then > could choose some of them for target EC zones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7866) Erasure coding: NameNode manages multiple erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HDFS-7866: - Attachment: HDFS-7866.11.patch Thanks Zhe for the explanations! Update patch accordingly. > Erasure coding: NameNode manages multiple erasure coding policies > - > > Key: HDFS-7866 > URL: https://issues.apache.org/jira/browse/HDFS-7866 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, > HDFS-7866-v3.patch, HDFS-7866.10.patch, HDFS-7866.11.patch, > HDFS-7866.4.patch, HDFS-7866.5.patch, HDFS-7866.6.patch, HDFS-7866.7.patch, > HDFS-7866.8.patch, HDFS-7866.9.patch > > > This is to extend NameNode to load, list and sync predefine EC schemas in > authorized and controlled approach. The provided facilities will be used to > implement DFSAdmin commands so admin can list available EC schemas, then > could choose some of them for target EC zones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages multiple erasure coding policies
[
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15174861#comment-15174861
]
Rui Li commented on HDFS-7866:
--
Thanks Zhe for the review and comments!
Forgive my ignorance but what do you mean by ASCII art?
As to the policy IDs, I think we can put them in {{HdfsConstants}} and name as
{{RS_6_3_POLICY_ID}}. Sounds good?
> Erasure coding: NameNode manages multiple erasure coding policies
> -
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch,
> HDFS-7866-v3.patch, HDFS-7866.10.patch, HDFS-7866.4.patch, HDFS-7866.5.patch,
> HDFS-7866.6.patch, HDFS-7866.7.patch, HDFS-7866.8.patch, HDFS-7866.9.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in
> authorized and controlled approach. The provided facilities will be used to
> implement DFSAdmin commands so admin can list available EC schemas, then
> could choose some of them for target EC zones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages multiple erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15171824#comment-15171824 ] Rui Li commented on HDFS-7866: -- The latest failures are not related. > Erasure coding: NameNode manages multiple erasure coding policies > - > > Key: HDFS-7866 > URL: https://issues.apache.org/jira/browse/HDFS-7866 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, > HDFS-7866-v3.patch, HDFS-7866.10.patch, HDFS-7866.4.patch, HDFS-7866.5.patch, > HDFS-7866.6.patch, HDFS-7866.7.patch, HDFS-7866.8.patch, HDFS-7866.9.patch > > > This is to extend NameNode to load, list and sync predefine EC schemas in > authorized and controlled approach. The provided facilities will be used to > implement DFSAdmin commands so admin can list available EC schemas, then > could choose some of them for target EC zones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7866) Erasure coding: NameNode manages multiple erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HDFS-7866: - Attachment: HDFS-7866.10.patch Thanks Kai for the explanations! Rebase the patch and fix the checkstyle issue. > Erasure coding: NameNode manages multiple erasure coding policies > - > > Key: HDFS-7866 > URL: https://issues.apache.org/jira/browse/HDFS-7866 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, > HDFS-7866-v3.patch, HDFS-7866.10.patch, HDFS-7866.4.patch, HDFS-7866.5.patch, > HDFS-7866.6.patch, HDFS-7866.7.patch, HDFS-7866.8.patch, HDFS-7866.9.patch > > > This is to extend NameNode to load, list and sync predefine EC schemas in > authorized and controlled approach. The provided facilities will be used to > implement DFSAdmin commands so admin can list available EC schemas, then > could choose some of them for target EC zones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7866) Erasure coding: NameNode manages multiple erasure coding policies
[
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HDFS-7866:
-
Attachment: HDFS-7866.9.patch
Fix test. I don't quite understand the checkstyle issue. The methods in
{{INodeFileAttributes}} are all public so not sure why it's complaining the
public is redundant.
I'm also thinking about how to test multiple policies. One way may be to make
all the tests get the EC policy from {{StripedFileTestUtil}}, which in turn
picks a policy randomly from all system policies. This can be done in a
separate JIRA of course. What do you guys think?
> Erasure coding: NameNode manages multiple erasure coding policies
> -
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch,
> HDFS-7866-v3.patch, HDFS-7866.4.patch, HDFS-7866.5.patch, HDFS-7866.6.patch,
> HDFS-7866.7.patch, HDFS-7866.8.patch, HDFS-7866.9.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in
> authorized and controlled approach. The provided facilities will be used to
> implement DFSAdmin commands so admin can list available EC schemas, then
> could choose some of them for target EC zones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7866) Erasure coding: NameNode manages multiple erasure coding policies
[
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HDFS-7866:
-
Attachment: HDFS-7866.8.patch
Update the patch as all the changes to tests are committed.
The v8 patch uses hard coded policy IDs and combines IS_STRIPED and REPLICATION
as Zhe suggested.
{{TestStripedBlockUtil}}, {{TestStripedINodeFile}} and {{TestBlockRecovery}}
use hard coded policy (RS-6-3) to work with hard coded test suite. Other tests
still use the default system policy which is set to RS-3-2 in the patch. Again
this is only temporary to test if multiple policy really works.
> Erasure coding: NameNode manages multiple erasure coding policies
> -
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch,
> HDFS-7866-v3.patch, HDFS-7866.4.patch, HDFS-7866.5.patch, HDFS-7866.6.patch,
> HDFS-7866.7.patch, HDFS-7866.8.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in
> authorized and controlled approach. The provided facilities will be used to
> implement DFSAdmin commands so admin can list available EC schemas, then
> could choose some of them for target EC zones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9816) Erasure Coding: allow to use multiple EC policies in striping related tests [Part 3]
[ https://issues.apache.org/jira/browse/HDFS-9816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15160029#comment-15160029 ] Rui Li commented on HDFS-9816: -- Thanks Zhe and Walter for the review! > Erasure Coding: allow to use multiple EC policies in striping related tests > [Part 3] > > > Key: HDFS-9816 > URL: https://issues.apache.org/jira/browse/HDFS-9816 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding, test >Reporter: Rui Li >Assignee: Rui Li > Fix For: 3.0.0 > > Attachments: HDFS-9816.1.patch, HDFS-9816.2.patch, HDFS-9816.3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9816) Erasure Coding: allow to use multiple EC policies in striping related tests [Part 3]
[ https://issues.apache.org/jira/browse/HDFS-9816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156816#comment-15156816 ] Rui Li commented on HDFS-9816: -- The failures are not related. > Erasure Coding: allow to use multiple EC policies in striping related tests > [Part 3] > > > Key: HDFS-9816 > URL: https://issues.apache.org/jira/browse/HDFS-9816 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding, test >Reporter: Rui Li >Assignee: Rui Li > Attachments: HDFS-9816.1.patch, HDFS-9816.2.patch, HDFS-9816.3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9816) Erasure Coding: allow to use multiple EC policies in striping related tests [Part 3]
[
https://issues.apache.org/jira/browse/HDFS-9816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HDFS-9816:
-
Attachment: HDFS-9816.3.patch
The v3 patch reverts changes to {{TestStripedINodeFile}} and
{{TestStripedBlockUtil}}.
> Erasure Coding: allow to use multiple EC policies in striping related tests
> [Part 3]
>
>
> Key: HDFS-9816
> URL: https://issues.apache.org/jira/browse/HDFS-9816
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding, test
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HDFS-9816.1.patch, HDFS-9816.2.patch, HDFS-9816.3.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9816) Erasure Coding: allow to use multiple EC policies in striping related tests [Part 3]
[ https://issues.apache.org/jira/browse/HDFS-9816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15156362#comment-15156362 ] Rui Li commented on HDFS-9816: -- Yeah that makes sense. Thanks Zhe. I'll update the patch. > Erasure Coding: allow to use multiple EC policies in striping related tests > [Part 3] > > > Key: HDFS-9816 > URL: https://issues.apache.org/jira/browse/HDFS-9816 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding, test >Reporter: Rui Li >Assignee: Rui Li > Attachments: HDFS-9816.1.patch, HDFS-9816.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9816) Erasure Coding: allow to use multiple EC policies in striping related tests [Part 3]
[ https://issues.apache.org/jira/browse/HDFS-9816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15154129#comment-15154129 ] Rui Li commented on HDFS-9816: -- Test failures cannot be reproduced locally. > Erasure Coding: allow to use multiple EC policies in striping related tests > [Part 3] > > > Key: HDFS-9816 > URL: https://issues.apache.org/jira/browse/HDFS-9816 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding, test >Reporter: Rui Li >Assignee: Rui Li > Attachments: HDFS-9816.1.patch, HDFS-9816.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9816) Erasure Coding: allow to use multiple EC policies in striping related tests [Part 3]
[
https://issues.apache.org/jira/browse/HDFS-9816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HDFS-9816:
-
Attachment: HDFS-9816.2.patch
Update patch to address comments.
The new static method is added to {{StripedBlockUtil}}.
{{TestBlockRecovery}} now uses hard-coded suite. I'll make it use hard-coded EC
policy in HDFS-7866.
{{TestLeaseRecoveryStriped}} still uses computed suite with randomness.
> Erasure Coding: allow to use multiple EC policies in striping related tests
> [Part 3]
>
>
> Key: HDFS-9816
> URL: https://issues.apache.org/jira/browse/HDFS-9816
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding, test
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HDFS-9816.1.patch, HDFS-9816.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9816) Erasure Coding: allow to use multiple EC policies in striping related tests [Part 3]
[
https://issues.apache.org/jira/browse/HDFS-9816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15153579#comment-15153579
]
Rui Li commented on HDFS-9816:
--
Thanks [~zhz] for the clarifications. How about putting the static method in
{{StripedBlockUtil}}? We're already using
{{StripedBlockUtil#getInternalBlockLength}} in {{RecoveryTaskStriped}}.
> Erasure Coding: allow to use multiple EC policies in striping related tests
> [Part 3]
>
>
> Key: HDFS-9816
> URL: https://issues.apache.org/jira/browse/HDFS-9816
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding, test
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HDFS-9816.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9816) Erasure Coding: allow to use multiple EC policies in striping related tests [Part 3]
[
https://issues.apache.org/jira/browse/HDFS-9816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15151803#comment-15151803
]
Rui Li commented on HDFS-9816:
--
Thanks Walter for your inputs.
As to reusing production code, {{TestLeaseRecoveryStriped#computeSafeLen}}
borrows the logic from {{RecoveryTaskStriped#getSafeLength}}. Shall I add a
public static method in {{RecoveryTaskStriped}} so that we can avoid repeating
the code?
> Erasure Coding: allow to use multiple EC policies in striping related tests
> [Part 3]
>
>
> Key: HDFS-9816
> URL: https://issues.apache.org/jira/browse/HDFS-9816
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding, test
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HDFS-9816.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9816) Erasure Coding: allow to use multiple EC policies in striping related tests [Part 3]
[
https://issues.apache.org/jira/browse/HDFS-9816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15151653#comment-15151653
]
Rui Li commented on HDFS-9816:
--
[~zhz], I think you mean {{TestBlockRecovery#testSafeLength}} right? If we want
to use hard-coded suite for this test, we should also make it use a hard-coded
schema, i.e. RS-6-3. Then we can move the current hard coded suite to
{{TestBlockRecovery}}. What do you think?
> Erasure Coding: allow to use multiple EC policies in striping related tests
> [Part 3]
>
>
> Key: HDFS-9816
> URL: https://issues.apache.org/jira/browse/HDFS-9816
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding, test
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HDFS-9816.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9816) Erasure Coding: allow to use multiple EC policies in striping related tests [Part 3]
[
https://issues.apache.org/jira/browse/HDFS-9816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15151609#comment-15151609
]
Rui Li commented on HDFS-9816:
--
Thanks guys for the discussion. I added
{{TestLeaseRecoveryStriped#getBlockLengthsSuite}} because with different EC
schemas, we'll have different number of internal block lengths. So it's
difficult to use hard-coded test suite. Please let me know if I misunderstand
about the test suite.
[~zhz], regarding your last comment, which test do you mean by
{{testSafeLength}}? I don't see such a test in {{TestLeaseRecoveryStriped}}.
> Erasure Coding: allow to use multiple EC policies in striping related tests
> [Part 3]
>
>
> Key: HDFS-9816
> URL: https://issues.apache.org/jira/browse/HDFS-9816
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding, test
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HDFS-9816.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9816) Erasure Coding: allow to use multiple EC policies in striping related tests [Part 3]
[
https://issues.apache.org/jira/browse/HDFS-9816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15149691#comment-15149691
]
Rui Li commented on HDFS-9816:
--
Thanks Zhe for the review.
{{TestLeaseRecoveryStriped}} failed because of {{java.net.BindException:
Address already in use}} when restarting namenode. So I think it's not related
to the patch here.
> Erasure Coding: allow to use multiple EC policies in striping related tests
> [Part 3]
>
>
> Key: HDFS-9816
> URL: https://issues.apache.org/jira/browse/HDFS-9816
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding, test
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HDFS-9816.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9816) Erasure Coding: allow to use multiple EC policies in striping related tests [Part 3]
[
https://issues.apache.org/jira/browse/HDFS-9816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HDFS-9816:
-
Attachment: HDFS-9816.1.patch
{{TestStripedINodeFile}} and {{TestFSImage}} don't have all the changes in
HDFS-7866 because we miss the API change here. These two will be updated later
with HDFS-7866.
cc [~zhz], [~drankye]
> Erasure Coding: allow to use multiple EC policies in striping related tests
> [Part 3]
>
>
> Key: HDFS-9816
> URL: https://issues.apache.org/jira/browse/HDFS-9816
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding, test
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HDFS-9816.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9816) Erasure Coding: allow to use multiple EC policies in striping related tests [Part 3]
Rui Li created HDFS-9816: Summary: Erasure Coding: allow to use multiple EC policies in striping related tests [Part 3] Key: HDFS-9816 URL: https://issues.apache.org/jira/browse/HDFS-9816 Project: Hadoop HDFS Issue Type: Sub-task Components: erasure-coding, test Reporter: Rui Li Assignee: Rui Li -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9755) Erasure Coding: allow to use multiple EC policies in striping related tests [Part 2]
[ https://issues.apache.org/jira/browse/HDFS-9755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15148010#comment-15148010 ] Rui Li commented on HDFS-9755: -- Thanks Zhe for reviewing and committing the patch. > Erasure Coding: allow to use multiple EC policies in striping related tests > [Part 2] > > > Key: HDFS-9755 > URL: https://issues.apache.org/jira/browse/HDFS-9755 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding, test >Reporter: Rui Li >Assignee: Rui Li > Fix For: 3.0.0 > > Attachments: HDFS-9755.1.patch, HDFS-9755.2.patch, HDFS-9755.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9755) Erasure Coding: allow to use multiple EC policies in striping related tests [Part 2]
[ https://issues.apache.org/jira/browse/HDFS-9755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HDFS-9755: - Attachment: HDFS-9755.2.patch HDFS-9755.2.patch Thanks Zhe for the review! I added the assert in v2 patch. Let's handle the follow-on task after we make all the tests work with multiple EC schema. > Erasure Coding: allow to use multiple EC policies in striping related tests > [Part 2] > > > Key: HDFS-9755 > URL: https://issues.apache.org/jira/browse/HDFS-9755 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding, test >Reporter: Rui Li >Assignee: Rui Li > Attachments: HDFS-9755.1.patch, HDFS-9755.2.patch, HDFS-9755.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9755) Erasure Coding: allow to use multiple EC policies in striping related tests [Part 2]
[ https://issues.apache.org/jira/browse/HDFS-9755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15142113#comment-15142113 ] Rui Li commented on HDFS-9755: -- Thanks for clarifying. > Erasure Coding: allow to use multiple EC policies in striping related tests > [Part 2] > > > Key: HDFS-9755 > URL: https://issues.apache.org/jira/browse/HDFS-9755 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding, test >Reporter: Rui Li >Assignee: Rui Li > Attachments: HDFS-9755.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9755) Erasure Coding: allow to use multiple EC policies in striping related tests [Part 2]
[
https://issues.apache.org/jira/browse/HDFS-9755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15132172#comment-15132172
]
Rui Li commented on HDFS-9755:
--
{{TestDatanodeStartupFixesLegacyStorageIDs}} fails on trunk as well. Other
failures cannot be reproduced.
cc [~drankye] [~zhz]
> Erasure Coding: allow to use multiple EC policies in striping related tests
> [Part 2]
>
>
> Key: HDFS-9755
> URL: https://issues.apache.org/jira/browse/HDFS-9755
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HDFS-9755.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9755) Erasure Coding: allow to use multiple EC policies in striping related tests [Part 2]
[ https://issues.apache.org/jira/browse/HDFS-9755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HDFS-9755: - Status: Patch Available (was: Open) > Erasure Coding: allow to use multiple EC policies in striping related tests > [Part 2] > > > Key: HDFS-9755 > URL: https://issues.apache.org/jira/browse/HDFS-9755 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li > Attachments: HDFS-9755.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9755) Erasure Coding: allow to use multiple EC policies in striping related tests [Part 2]
[ https://issues.apache.org/jira/browse/HDFS-9755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HDFS-9755: - Attachment: HDFS-9755.1.patch > Erasure Coding: allow to use multiple EC policies in striping related tests > [Part 2] > > > Key: HDFS-9755 > URL: https://issues.apache.org/jira/browse/HDFS-9755 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li > Attachments: HDFS-9755.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9755) Erasure Coding: allow to use multiple EC policies in striping related tests [Part 2]
Rui Li created HDFS-9755: Summary: Erasure Coding: allow to use multiple EC policies in striping related tests [Part 2] Key: HDFS-9755 URL: https://issues.apache.org/jira/browse/HDFS-9755 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Rui Li Assignee: Rui Li -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9658) Erasure Coding: allow to use multiple EC policies in striping related tests
[ https://issues.apache.org/jira/browse/HDFS-9658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15131520#comment-15131520 ] Rui Li commented on HDFS-9658: -- Thanks Zhe for the review! > Erasure Coding: allow to use multiple EC policies in striping related tests > --- > > Key: HDFS-9658 > URL: https://issues.apache.org/jira/browse/HDFS-9658 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li > Fix For: 3.0.0 > > Attachments: HDFS-9658.1.patch, HDFS-9658.2.patch > > > Currently many of the EC-related tests assume we're using RS-6-3 > schema/policy. There're lots of hard coded fields as well as computations > based on that. To support multiple EC policies, we need to remove these hard > coded logic and make the tests more flexible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9658) Erasure Coding: allow to use multiple EC policies in striping related tests
[ https://issues.apache.org/jira/browse/HDFS-9658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129957#comment-15129957 ] Rui Li commented on HDFS-9658: -- Failures are not related. > Erasure Coding: allow to use multiple EC policies in striping related tests > --- > > Key: HDFS-9658 > URL: https://issues.apache.org/jira/browse/HDFS-9658 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li > Attachments: HDFS-9658.1.patch, HDFS-9658.2.patch > > > Currently many of the EC-related tests assume we're using RS-6-3 > schema/policy. There're lots of hard coded fields as well as computations > based on that. To support multiple EC policies, we need to remove these hard > coded logic and make the tests more flexible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9658) Erasure Coding: allow to use multiple EC policies in striping related tests
[
https://issues.apache.org/jira/browse/HDFS-9658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HDFS-9658:
-
Attachment: HDFS-9658.2.patch
Thanks Zhe. Update patch to increase {{maxPerLevel}}.
> Erasure Coding: allow to use multiple EC policies in striping related tests
> ---
>
> Key: HDFS-9658
> URL: https://issues.apache.org/jira/browse/HDFS-9658
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HDFS-9658.1.patch, HDFS-9658.2.patch
>
>
> Currently many of the EC-related tests assume we're using RS-6-3
> schema/policy. There're lots of hard coded fields as well as computations
> based on that. To support multiple EC policies, we need to remove these hard
> coded logic and make the tests more flexible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages multiple erasure coding policies
[
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15127885#comment-15127885
]
Rui Li commented on HDFS-7866:
--
Thanks [~zhz] for pointing me to the discussion and
{{BlockStoragePolicySuite}}. I'll update the patch once we reach an agreement.
As to policy ID mapping, do you mean we should pre-define the IDs for system EC
policies, like what we do in {{HdfsConstants}} for storage policies? If we want
to support custom policies later, we still need a way to generate unique IDs
right?
> Erasure coding: NameNode manages multiple erasure coding policies
> -
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch,
> HDFS-7866-v3.patch, HDFS-7866.4.patch, HDFS-7866.5.patch, HDFS-7866.6.patch,
> HDFS-7866.7.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in
> authorized and controlled approach. The provided facilities will be used to
> implement DFSAdmin commands so admin can list available EC schemas, then
> could choose some of them for target EC zones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9658) Erasure Coding: allow to use multiple EC policies in striping related tests
[
https://issues.apache.org/jira/browse/HDFS-9658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15127428#comment-15127428
]
Rui Li commented on HDFS-9658:
--
Hi [~zhz], thanks for the review!
I kept {{dnIndexSuite}} so that the new method is only called once
({{runTestWithMultipleFailure}} may be called multiple times in future, e.g.
when {{testMultipleDatanodeFailureRandomLength}} is enabled). What do you think?
I can increase {{maxPerLevel}} for sure.
> Erasure Coding: allow to use multiple EC policies in striping related tests
> ---
>
> Key: HDFS-9658
> URL: https://issues.apache.org/jira/browse/HDFS-9658
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HDFS-9658.1.patch
>
>
> Currently many of the EC-related tests assume we're using RS-6-3
> schema/policy. There're lots of hard coded fields as well as computations
> based on that. To support multiple EC policies, we need to remove these hard
> coded logic and make the tests more flexible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9658) Erasure Coding: allow to use multiple EC policies in striping related tests
[ https://issues.apache.org/jira/browse/HDFS-9658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15106105#comment-15106105 ] Rui Li commented on HDFS-9658: -- Test failures don't seem related and cannot be reproduced locally. > Erasure Coding: allow to use multiple EC policies in striping related tests > --- > > Key: HDFS-9658 > URL: https://issues.apache.org/jira/browse/HDFS-9658 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li > Attachments: HDFS-9658.1.patch > > > Currently many of the EC-related tests assume we're using RS-6-3 > schema/policy. There're lots of hard coded fields as well as computations > based on that. To support multiple EC policies, we need to remove these hard > coded logic and make the tests more flexible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages multiple erasure coding policies
[ https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105209#comment-15105209 ] Rui Li commented on HDFS-7866: -- Thanks Kai for the suggestions. I just filed HDFS-9658 to break this task into small pieces. Will create more as we progress. > Erasure coding: NameNode manages multiple erasure coding policies > - > > Key: HDFS-7866 > URL: https://issues.apache.org/jira/browse/HDFS-7866 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, > HDFS-7866-v3.patch, HDFS-7866.4.patch, HDFS-7866.5.patch, HDFS-7866.6.patch, > HDFS-7866.7.patch > > > This is to extend NameNode to load, list and sync predefine EC schemas in > authorized and controlled approach. The provided facilities will be used to > implement DFSAdmin commands so admin can list available EC schemas, then > could choose some of them for target EC zones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9658) Erasure Coding: make tests compatible with multiple EC policies [Part 1]
[ https://issues.apache.org/jira/browse/HDFS-9658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HDFS-9658: - Status: Patch Available (was: Open) > Erasure Coding: make tests compatible with multiple EC policies [Part 1] > > > Key: HDFS-9658 > URL: https://issues.apache.org/jira/browse/HDFS-9658 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li > Attachments: HDFS-9658.1.patch > > > Currently many of the EC-related tests assume we're using RS-6-3 > schema/policy. There're lots of hard coded fields as well as computations > based on that. To support multiple EC policies, we need to remove these hard > coded logic and make the tests more flexible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9658) Erasure Coding: make tests compatible with multiple EC policies [Part 1]
[ https://issues.apache.org/jira/browse/HDFS-9658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HDFS-9658: - Attachment: HDFS-9658.1.patch > Erasure Coding: make tests compatible with multiple EC policies [Part 1] > > > Key: HDFS-9658 > URL: https://issues.apache.org/jira/browse/HDFS-9658 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li > Attachments: HDFS-9658.1.patch > > > Currently many of the EC-related tests assume we're using RS-6-3 > schema/policy. There're lots of hard coded fields as well as computations > based on that. To support multiple EC policies, we need to remove these hard > coded logic and make the tests more flexible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9658) Erasure Coding: make tests compatible with multiple EC policies [Part 1]
Rui Li created HDFS-9658: Summary: Erasure Coding: make tests compatible with multiple EC policies [Part 1] Key: HDFS-9658 URL: https://issues.apache.org/jira/browse/HDFS-9658 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Rui Li Assignee: Rui Li Currently many of the EC-related tests assume we're using RS-6-3 schema/policy. There're lots of hard coded fields as well as computations based on that. To support multiple EC policies, we need to remove these hard coded logic and make the tests more flexible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9603) Erasure Coding: Use ErasureCoder to encode/decode a block group
[
https://issues.apache.org/jira/browse/HDFS-9603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HDFS-9603:
-
Attachment: HDFS-9603.3.patch
Fix checkstyle.
> Erasure Coding: Use ErasureCoder to encode/decode a block group
> ---
>
> Key: HDFS-9603
> URL: https://issues.apache.org/jira/browse/HDFS-9603
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HDFS-9603.1.patch, HDFS-9603.2.patch, HDFS-9603.3.patch
>
>
> According to design, {{ErasureCoder}} is responsible to encode/decode a block
> group. Currently however, we directly use {{RawErasureCoder}} to do the work,
> e.g. in {{DFSStripedOutputStream}}. This task attempts to encapsulate
> {{RawErasureCoder}} to comply with the design.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9603) Erasure Coding: Use ErasureCoder to encode/decode a block group
[
https://issues.apache.org/jira/browse/HDFS-9603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HDFS-9603:
-
Attachment: HDFS-9603.2.patch
Update patch to address Kai's comments.
> Erasure Coding: Use ErasureCoder to encode/decode a block group
> ---
>
> Key: HDFS-9603
> URL: https://issues.apache.org/jira/browse/HDFS-9603
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HDFS-9603.1.patch, HDFS-9603.2.patch
>
>
> According to design, {{ErasureCoder}} is responsible to encode/decode a block
> group. Currently however, we directly use {{RawErasureCoder}} to do the work,
> e.g. in {{DFSStripedOutputStream}}. This task attempts to encapsulate
> {{RawErasureCoder}} to comply with the design.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9603) Erasure Coding: Use ErasureCoder to encode/decode a block group
[
https://issues.apache.org/jira/browse/HDFS-9603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15082268#comment-15082268
]
Rui Li commented on HDFS-9603:
--
Thanks Kai for clarifying!
> Erasure Coding: Use ErasureCoder to encode/decode a block group
> ---
>
> Key: HDFS-9603
> URL: https://issues.apache.org/jira/browse/HDFS-9603
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HDFS-9603.1.patch
>
>
> According to design, {{ErasureCoder}} is responsible to encode/decode a block
> group. Currently however, we directly use {{RawErasureCoder}} to do the work,
> e.g. in {{DFSStripedOutputStream}}. This task attempts to encapsulate
> {{RawErasureCoder}} to comply with the design.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9603) Erasure Coding: Use ErasureCoder to encode/decode a block group
[
https://issues.apache.org/jira/browse/HDFS-9603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15082217#comment-15082217
]
Rui Li commented on HDFS-9603:
--
Hi Kai, I got one more question about {{ECBlock}}. I remember you said we're
supposed to get {{ECChunk}}s from {{ECBlock}}. If that's right, shall we make
{{ECBlock}} abstract and and API like {{getECChunks}}?
> Erasure Coding: Use ErasureCoder to encode/decode a block group
> ---
>
> Key: HDFS-9603
> URL: https://issues.apache.org/jira/browse/HDFS-9603
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HDFS-9603.1.patch
>
>
> According to design, {{ErasureCoder}} is responsible to encode/decode a block
> group. Currently however, we directly use {{RawErasureCoder}} to do the work,
> e.g. in {{DFSStripedOutputStream}}. This task attempts to encapsulate
> {{RawErasureCoder}} to comply with the design.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9603) Erasure Coding: Use ErasureCoder to encode/decode a block group
[
https://issues.apache.org/jira/browse/HDFS-9603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15082210#comment-15082210
]
Rui Li commented on HDFS-9603:
--
Thanks Kai for the review.
I used the concrete class because we already have methods like
{{CodecUtil::createRSRawEncoder}}. But I agree we can wait until HitchHicker
coder gets in. Then we can figure out how to create different coders according
to configurations.
The star import wasn't introduced in this patch. I can fix it anyway.
> Erasure Coding: Use ErasureCoder to encode/decode a block group
> ---
>
> Key: HDFS-9603
> URL: https://issues.apache.org/jira/browse/HDFS-9603
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HDFS-9603.1.patch
>
>
> According to design, {{ErasureCoder}} is responsible to encode/decode a block
> group. Currently however, we directly use {{RawErasureCoder}} to do the work,
> e.g. in {{DFSStripedOutputStream}}. This task attempts to encapsulate
> {{RawErasureCoder}} to comply with the design.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9603) Erasure Coding: Use ErasureCoder to encode/decode a block group
[
https://issues.apache.org/jira/browse/HDFS-9603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HDFS-9603:
-
Status: Patch Available (was: Open)
> Erasure Coding: Use ErasureCoder to encode/decode a block group
> ---
>
> Key: HDFS-9603
> URL: https://issues.apache.org/jira/browse/HDFS-9603
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HDFS-9603.1.patch
>
>
> According to design, {{ErasureCoder}} is responsible to encode/decode a block
> group. Currently however, we directly use {{RawErasureCoder}} to do the work,
> e.g. in {{DFSStripedOutputStream}}. This task attempts to encapsulate
> {{RawErasureCoder}} to comply with the design.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9603) Erasure Coding: Use ErasureCoder to encode/decode a block group
[
https://issues.apache.org/jira/browse/HDFS-9603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HDFS-9603:
-
Attachment: HDFS-9603.1.patch
Upload a patch for review. It only changes how we encode data.
[~drankye] please let me know if it's the right way to use {{ErasureCoder}}.
Thanks.
> Erasure Coding: Use ErasureCoder to encode/decode a block group
> ---
>
> Key: HDFS-9603
> URL: https://issues.apache.org/jira/browse/HDFS-9603
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HDFS-9603.1.patch
>
>
> According to design, {{ErasureCoder}} is responsible to encode/decode a block
> group. Currently however, we directly use {{RawErasureCoder}} to do the work,
> e.g. in {{DFSStripedOutputStream}}. This task attempts to encapsulate
> {{RawErasureCoder}} to comply with the design.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas
[
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15075676#comment-15075676
]
Rui Li commented on HDFS-7866:
--
Thanks [~drankye] for the review!
1. Sure, will do.
2. Since the system policies are static members, they'll have the same ID after
NN restarts. But I agree it's better not to let {{ErasureCodingPolicy}}
generate the ID itself. How about pass the ID to it as a constructor parameter,
and have the ID generator in {{ErasureCodingPolicyManager}}?
3. Currently striped files all have 0 as the replication factor. E.g. in the
INodeFile part of the patch:
{code}
- // Replication factor for striped files is zero
+ // For erasure coded files, replication is used to store ec policy id
+ if (isStriped) {
+Preconditions.checkArgument(replication >= 0);
+ }
+ h = REPLICATION.BITS.combine(replication, h);
if (isStriped) {
-h = REPLICATION.BITS.combine(0L, h);
h = IS_STRIPED.BITS.combine(1L, h);
} else {
-h = REPLICATION.BITS.combine(replication, h);
h = IS_STRIPED.BITS.combine(0L, h);
}
{code}
With the patch, striped file's replication factor is actually the policy ID.
The modification you mentioned in {{HdfsFileStatus}} is to avoid confusing
users by displaying policy ID as replication factor.
4. {{QUEUE_VERY_UNDER_REPLICATED}} means the current redundancy is 1/3 of
desired redundancy. So I changed the comment to be consistent with the code.
You can refer to discussions in HDFS-8461.
5. Which map do you think we should keep?
> Erasure coding: NameNode manages EC schemas
> ---
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch,
> HDFS-7866-v3.patch, HDFS-7866.4.patch, HDFS-7866.5.patch, HDFS-7866.6.patch,
> HDFS-7866.7.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in
> authorized and controlled approach. The provided facilities will be used to
> implement DFSAdmin commands so admin can list available EC schemas, then
> could choose some of them for target EC zones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9603) Erasure Coding: Use ErasureCoder to encode/decode a block group
[
https://issues.apache.org/jira/browse/HDFS-9603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15072522#comment-15072522
]
Rui Li commented on HDFS-9603:
--
cc [~drankye]
> Erasure Coding: Use ErasureCoder to encode/decode a block group
> ---
>
> Key: HDFS-9603
> URL: https://issues.apache.org/jira/browse/HDFS-9603
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
>
> According to design, {{ErasureCoder}} is responsible to encode/decode a block
> group. Currently however, we directly use {{RawErasureCoder}} to do the work,
> e.g. in {{DFSStripedOutputStream}}. This task attempts to encapsulate
> {{RawErasureCoder}} to comply with the design.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9603) Erasure Coding: Use ErasureCoder to encode/decode a block group
Rui Li created HDFS-9603:
Summary: Erasure Coding: Use ErasureCoder to encode/decode a block
group
Key: HDFS-9603
URL: https://issues.apache.org/jira/browse/HDFS-9603
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rui Li
Assignee: Rui Li
According to design, {{ErasureCoder}} is responsible to encode/decode a block
group. Currently however, we directly use {{RawErasureCoder}} to do the work,
e.g. in {{DFSStripedOutputStream}}. This task attempts to encapsulate
{{RawErasureCoder}} to comply with the design.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7866) Erasure coding: NameNode manages EC schemas
[
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HDFS-7866:
-
Attachment: HDFS-7866.7.patch
Rebase patch with latest trunk.
Update to {{TestUnderReplicatedBlockQueues}} is based on discussions in
HDFS-8461.
> Erasure coding: NameNode manages EC schemas
> ---
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch,
> HDFS-7866-v3.patch, HDFS-7866.4.patch, HDFS-7866.5.patch, HDFS-7866.6.patch,
> HDFS-7866.7.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in
> authorized and controlled approach. The provided facilities will be used to
> implement DFSAdmin commands so admin can list available EC schemas, then
> could choose some of them for target EC zones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7866) Erasure coding: NameNode manages EC schemas
[
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HDFS-7866:
-
Attachment: HDFS-7866.6.patch
The v6 patch adds {{TestErasureCodingPolicies::testMultiplePoliciesCoExist}} to
test using different EC policies within a cluster.
> Erasure coding: NameNode manages EC schemas
> ---
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch,
> HDFS-7866-v3.patch, HDFS-7866.4.patch, HDFS-7866.5.patch, HDFS-7866.6.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in
> authorized and controlled approach. The provided facilities will be used to
> implement DFSAdmin commands so admin can list available EC schemas, then
> could choose some of them for target EC zones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas
[
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15066031#comment-15066031
]
Rui Li commented on HDFS-7866:
--
Thanks Kai for your comments.
bp. However, please don't set it as default policy or change the default policy
at all.
Yeah I didn't mean to change the default policy. The change in the patch is
temporary to make the tests pick the new policy.
bp. Would you share the details about making use of the remain bits in the
replicaiton field?
Sure. The HeaderFormat is a 64-bit long and its layout is {{[4-bit
storagePolicyID][1-bit isStriped][11-bit replication][48-bit
preferredBlockSize]}}. The patch doesn't change the layout -- it just uses the
11-bit replication to store the policy ID. Therefore it can support up to
{{2^11-1=2047}} different policies. I think that'll be enough for all the
combinations of codecs, cell sizes and layouts.
bq. do we add/have tests to test multiple policies can work at the same time in
a cluster
Good point. I'll add a test to cover this.
> Erasure coding: NameNode manages EC schemas
> ---
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch,
> HDFS-7866-v3.patch, HDFS-7866.4.patch, HDFS-7866.5.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in
> authorized and controlled approach. The provided facilities will be used to
> implement DFSAdmin commands so admin can list available EC schemas, then
> could choose some of them for target EC zones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8461) Erasure coding: fix priority level of UnderReplicatedBlocks for striped block
[
https://issues.apache.org/jira/browse/HDFS-8461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15065194#comment-15065194
]
Rui Li commented on HDFS-8461:
--
Thanks [~zhz] for your input. In
{{TestUnderReplicatedBlockQueues::doTestStripedBlockPriorities}}, we expect the
block in {{QUEUE_VERY_UNDER_REPLICATED}} if {{curReplicas - dataBlkNum == 1}}.
So we should change the test to comply with the 1/3 logic?
> Erasure coding: fix priority level of UnderReplicatedBlocks for striped block
> -
>
> Key: HDFS-8461
> URL: https://issues.apache.org/jira/browse/HDFS-8461
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Fix For: HDFS-7285
>
> Attachments: HDFS-8461-HDFS-7285.001.patch,
> HDFS-8461-HDFS-7285.002.patch
>
>
> Issues 1: correctly mark corrupted blocks.
> Issues 2: distinguish highest risk priority and normal risk priority.
> {code:title=UnderReplicatedBlocks.java}
> private int getPriority(int curReplicas,
> ...
> } else if (curReplicas == 1) {
> //only on replica -risk of loss
> // highest priority
> return QUEUE_HIGHEST_PRIORITY;
> ...
> {code}
> For stripe blocks, we should return QUEUE_HIGHEST_PRIORITY when curReplicas
> == 6( Suppose 6+3 schema).
> That's important. Because
> {code:title=BlockManager.java}
> DatanodeDescriptor[] chooseSourceDatanodes(BlockInfo block,
> ...
> if(priority != UnderReplicatedBlocks.QUEUE_HIGHEST_PRIORITY
> && !node.isDecommissionInProgress()
> && node.getNumberOfBlocksToBeReplicated() >= maxReplicationStreams)
> {
> continue; // already reached replication limit
> }
> ...
> {code}
> It may return not enough source DNs ( maybe 5), and failed to recover.
> A busy node should not be skiped if a block has highest risk/priority. The
> issue is the striped block doesn't have priority.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas
[ https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063885#comment-15063885 ] Rui Li commented on HDFS-7866: -- The latest failures cannot be reproduced locally. > Erasure coding: NameNode manages EC schemas > --- > > Key: HDFS-7866 > URL: https://issues.apache.org/jira/browse/HDFS-7866 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, > HDFS-7866-v3.patch, HDFS-7866.4.patch, HDFS-7866.5.patch > > > This is to extend NameNode to load, list and sync predefine EC schemas in > authorized and controlled approach. The provided facilities will be used to > implement DFSAdmin commands so admin can list available EC schemas, then > could choose some of them for target EC zones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7866) Erasure coding: NameNode manages EC schemas
[ https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HDFS-7866: - Attachment: HDFS-7866.5.patch Update patch to fix more tests. > Erasure coding: NameNode manages EC schemas > --- > > Key: HDFS-7866 > URL: https://issues.apache.org/jira/browse/HDFS-7866 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, > HDFS-7866-v3.patch, HDFS-7866.4.patch, HDFS-7866.5.patch > > > This is to extend NameNode to load, list and sync predefine EC schemas in > authorized and controlled approach. The provided facilities will be used to > implement DFSAdmin commands so admin can list available EC schemas, then > could choose some of them for target EC zones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8461) Erasure coding: fix priority level of UnderReplicatedBlocks for striped block
[
https://issues.apache.org/jira/browse/HDFS-8461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063496#comment-15063496
]
Rui Li commented on HDFS-8461:
--
Hi, I got a question about the following code of the patch:
{code}
} else if ((curReplicas - dataBlkNum) * 3 < parityBlkNum + 1) {
// can only afford one replica loss
// this is considered very under-replicated
return QUEUE_VERY_UNDER_REPLICATED;
{code}
To make the code consistent with the comment, shall we change the condition to
{{if (curReplicas - dataBlkNum == 1)}} or {{if ((curReplicas - dataBlkNum) *
parityBlkNum < parityBlkNum + 1)}}?
The current code doesn't work properly when I tried to add multiple EC policies
in HDFS-7866, where parityBlkNum!=3.
> Erasure coding: fix priority level of UnderReplicatedBlocks for striped block
> -
>
> Key: HDFS-8461
> URL: https://issues.apache.org/jira/browse/HDFS-8461
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Fix For: HDFS-7285
>
> Attachments: HDFS-8461-HDFS-7285.001.patch,
> HDFS-8461-HDFS-7285.002.patch
>
>
> Issues 1: correctly mark corrupted blocks.
> Issues 2: distinguish highest risk priority and normal risk priority.
> {code:title=UnderReplicatedBlocks.java}
> private int getPriority(int curReplicas,
> ...
> } else if (curReplicas == 1) {
> //only on replica -risk of loss
> // highest priority
> return QUEUE_HIGHEST_PRIORITY;
> ...
> {code}
> For stripe blocks, we should return QUEUE_HIGHEST_PRIORITY when curReplicas
> == 6( Suppose 6+3 schema).
> That's important. Because
> {code:title=BlockManager.java}
> DatanodeDescriptor[] chooseSourceDatanodes(BlockInfo block,
> ...
> if(priority != UnderReplicatedBlocks.QUEUE_HIGHEST_PRIORITY
> && !node.isDecommissionInProgress()
> && node.getNumberOfBlocksToBeReplicated() >= maxReplicationStreams)
> {
> continue; // already reached replication limit
> }
> ...
> {code}
> It may return not enough source DNs ( maybe 5), and failed to recover.
> A busy node should not be skiped if a block has highest risk/priority. The
> issue is the striped block doesn't have priority.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7866) Erasure coding: NameNode manages EC schemas
[ https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HDFS-7866: - Status: Patch Available (was: In Progress) > Erasure coding: NameNode manages EC schemas > --- > > Key: HDFS-7866 > URL: https://issues.apache.org/jira/browse/HDFS-7866 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, > HDFS-7866-v3.patch, HDFS-7866.4.patch > > > This is to extend NameNode to load, list and sync predefine EC schemas in > authorized and controlled approach. The provided facilities will be used to > implement DFSAdmin commands so admin can list available EC schemas, then > could choose some of them for target EC zones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7866) Erasure coding: NameNode manages EC schemas
[
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HDFS-7866:
-
Attachment: HDFS-7866.4.patch
The v4 patch is mainly for review and testing. It basically includes:
1. Add a new system EC policy {{RS-3-2}} and set it as the default policy. The
3-2 schema is nothing specific and just for testing. And we can revert to our
current default policy (or make it configurable) when the patch is ready.
2. EC policy can be retrieved either by name or by ID. The ID is stored in
HEADER::replication in InodeFile, as suggested by the TODO comment.
3. Lots of modifications to the tests to make them work with multiple policies.
This may better fit as a follow-on task. But to verify the patch I have to make
the tests pass here.
> Erasure coding: NameNode manages EC schemas
> ---
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch,
> HDFS-7866-v3.patch, HDFS-7866.4.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in
> authorized and controlled approach. The provided facilities will be used to
> implement DFSAdmin commands so admin can list available EC schemas, then
> could choose some of them for target EC zones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas
[
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15048347#comment-15048347
]
Rui Li commented on HDFS-7866:
--
I tried to add another system policy (with different schema) and hit the
following problem.
In {{FSDirErasureCodingOp::getErasureCodingPolicyForPath}}, if the inode is a
file, we retrieve the policy by a numeric ID:
{{INodeFile::getErasureCodingPolicyID}}.
For directories however, the policy is retrieved by name:
{code}
final XAttrFeature xaf = inode.getXAttrFeature();
if (xaf != null) {
XAttr xattr = xaf.getXAttr(XATTR_ERASURECODING_POLICY);
if (xattr != null) {
ByteArrayInputStream bIn = new
ByteArrayInputStream(xattr.getValue());
DataInputStream dIn = new DataInputStream(bIn);
String ecPolicyName = WritableUtils.readString(dIn);
return fsd.getFSNamesystem().getErasureCodingPolicyManager().
getPolicy(ecPolicyName);
}
}
{code}
Therefore the newly added policy works for dirs but not files. I think we need
a unified way to store/get a policy. Any thoughts?
> Erasure coding: NameNode manages EC schemas
> ---
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch,
> HDFS-7866-v3.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in
> authorized and controlled approach. The provided facilities will be used to
> implement DFSAdmin commands so admin can list available EC schemas, then
> could choose some of them for target EC zones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas
[ https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15046636#comment-15046636 ] Rui Li commented on HDFS-7866: -- Hi [~drankye], it seems HDFS-9345 depends on this JIRA. I can help with it if you're busy with other tasks. Thanks. > Erasure coding: NameNode manages EC schemas > --- > > Key: HDFS-7866 > URL: https://issues.apache.org/jira/browse/HDFS-7866 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Kai Zheng > Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, > HDFS-7866-v3.patch > > > This is to extend NameNode to load, list and sync predefine EC schemas in > authorized and controlled approach. The provided facilities will be used to > implement DFSAdmin commands so admin can list available EC schemas, then > could choose some of them for target EC zones. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9496) Erasure coding: an erasure codec throughput benchmark tool
[ https://issues.apache.org/jira/browse/HDFS-9496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037062#comment-15037062 ] Rui Li commented on HDFS-9496: -- [~huizane] and [~zhz] - We already have HADOOP-11588 to implement a benchmark tool at the codec level. Please refer to that to see if it's the same as what's suggested here. The patch is kind of outdated though so I'll rebase it and target to trunk. > Erasure coding: an erasure codec throughput benchmark tool > -- > > Key: HDFS-9496 > URL: https://issues.apache.org/jira/browse/HDFS-9496 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding, test >Reporter: Hui Zheng > > We need a tool which can help us decide/benchmark an Erasure Codec and schema. > Considering HDFS-8968 has implemented an I/O throughput benchmark tool.Maybe > we could simply add encode/decode operation to it or implement another tool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8968) Erasure coding: a comprehensive I/O throughput benchmark tool
[ https://issues.apache.org/jira/browse/HDFS-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003886#comment-15003886 ] Rui Li commented on HDFS-8968: -- Thanks Zhe for the review and good suggestions. I just filed HDFS-9424 as the follow on task. Regarding the code optimizations: 2. I think the number of required DNs is related to the EC policy we use, right? Currently the tool just uses the default policy. We can make this configurable so that user can choose a policy to test. 3. Yeah I can clear the client's cache. However, I don't think this is necessary because ideally, no disk I/O is involved on the client side. So clearing client's cache won't make much difference (I suppose you're referring the cache for disk, let me know if otherwise). The tool only runs the benchmark once instead of multiple rounds. I think it's better and easier to leave it to the user to take care of the cache. > Erasure coding: a comprehensive I/O throughput benchmark tool > - > > Key: HDFS-8968 > URL: https://issues.apache.org/jira/browse/HDFS-8968 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding, test >Affects Versions: 3.0.0 >Reporter: Kai Zheng >Assignee: Rui Li > Fix For: 3.0.0 > > Attachments: HDFS-8968-HDFS-7285.1.patch, > HDFS-8968-HDFS-7285.2.patch, HDFS-8968.3.patch, HDFS-8968.4.patch, > HDFS-8968.5.patch > > > We need a new benchmark tool to measure the throughput of client writing and > reading considering cases or factors: > * 3-replica or striping; > * write or read, stateful read or positional read; > * which erasure coder; > * striping cell size; > * concurrent readers/writers using processes or threads. > The tool should be easy to use and better to avoid unnecessary local > environment impact, like local disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9424) Erasure Coding: enhance the ErasureCodeBenchmarkThroughput tool
[ https://issues.apache.org/jira/browse/HDFS-9424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HDFS-9424: - Priority: Minor (was: Major) > Erasure Coding: enhance the ErasureCodeBenchmarkThroughput tool > --- > > Key: HDFS-9424 > URL: https://issues.apache.org/jira/browse/HDFS-9424 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > > This JIRA is a follow-on of HDFS-8968. We should support more kinds of tests > as well as optimize the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9424) Erasure Coding: enhance the ErasureCodeBenchmarkThroughput tool
Rui Li created HDFS-9424: Summary: Erasure Coding: enhance the ErasureCodeBenchmarkThroughput tool Key: HDFS-9424 URL: https://issues.apache.org/jira/browse/HDFS-9424 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Rui Li Assignee: Rui Li This JIRA is a follow-on of HDFS-8968. We should support more kinds of tests as well as optimize the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8968) New benchmark throughput tool for striping erasure coding
[ https://issues.apache.org/jira/browse/HDFS-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HDFS-8968: - Attachment: HDFS-8968.5.patch Update patch to avoid generating random buffers each time before writing to a file, which can be a hot spot method and interfere the results. > New benchmark throughput tool for striping erasure coding > - > > Key: HDFS-8968 > URL: https://issues.apache.org/jira/browse/HDFS-8968 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > Attachments: HDFS-8968-HDFS-7285.1.patch, > HDFS-8968-HDFS-7285.2.patch, HDFS-8968.3.patch, HDFS-8968.4.patch, > HDFS-8968.5.patch > > > We need a new benchmark tool to measure the throughput of client writing and > reading considering cases or factors: > * 3-replica or striping; > * write or read, stateful read or positional read; > * which erasure coder; > * striping cell size; > * concurrent readers/writers using processes or threads. > The tool should be easy to use and better to avoid unnecessary local > environment impact, like local disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8968) New benchmark throughput tool for striping erasure coding
[ https://issues.apache.org/jira/browse/HDFS-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HDFS-8968: - Attachment: HDFS-8968.4.patch Address Rakesh's comments. A new test is added to test the new benchmark tool, in which the tool runs with a MiniDFSCluster. > New benchmark throughput tool for striping erasure coding > - > > Key: HDFS-8968 > URL: https://issues.apache.org/jira/browse/HDFS-8968 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > Attachments: HDFS-8968-HDFS-7285.1.patch, > HDFS-8968-HDFS-7285.2.patch, HDFS-8968.3.patch, HDFS-8968.4.patch > > > We need a new benchmark tool to measure the throughput of client writing and > reading considering cases or factors: > * 3-replica or striping; > * write or read, stateful read or positional read; > * which erasure coder; > * striping cell size; > * concurrent readers/writers using processes or threads. > The tool should be easy to use and better to avoid unnecessary local > environment impact, like local disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8968) New benchmark throughput tool for striping erasure coding
[ https://issues.apache.org/jira/browse/HDFS-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14992866#comment-14992866 ] Rui Li commented on HDFS-8968: -- I was planning to run ErasureCodeBenchmarkThroughput with MiniDFSCluster in order to test ErasureCodeBenchmarkThroughput. My understanding is that MiniDFSCluster can be used to test classes that require a cluster to run, just like the tool here. Is that right? > New benchmark throughput tool for striping erasure coding > - > > Key: HDFS-8968 > URL: https://issues.apache.org/jira/browse/HDFS-8968 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > Attachments: HDFS-8968-HDFS-7285.1.patch, > HDFS-8968-HDFS-7285.2.patch, HDFS-8968.3.patch > > > We need a new benchmark tool to measure the throughput of client writing and > reading considering cases or factors: > * 3-replica or striping; > * write or read, stateful read or positional read; > * which erasure coder; > * striping cell size; > * concurrent readers/writers using processes or threads. > The tool should be easy to use and better to avoid unnecessary local > environment impact, like local disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8968) New benchmark throughput tool for striping erasure coding
[
https://issues.apache.org/jira/browse/HDFS-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14991596#comment-14991596
]
Rui Li commented on HDFS-8968:
--
Hi [~rakeshr],
One more question. What do you mean by running a real cluster inside
{{ErasureCodeBenchmarkThroughput}}? I checked {{BenchmarkThroughput}} and seems
it only runs against local FS and MiniDFSCluster.
I mentioned I can run {{ErasureCodeBenchmarkThroughput}} with MiniDFSCluster in
a unit test. And that's only intended to test the new tool. To really benchmark
performance, user has to deploy a real cluster, right?
Thanks.
> New benchmark throughput tool for striping erasure coding
> -
>
> Key: HDFS-8968
> URL: https://issues.apache.org/jira/browse/HDFS-8968
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-8968-HDFS-7285.1.patch,
> HDFS-8968-HDFS-7285.2.patch, HDFS-8968.3.patch
>
>
> We need a new benchmark tool to measure the throughput of client writing and
> reading considering cases or factors:
> * 3-replica or striping;
> * write or read, stateful read or positional read;
> * which erasure coder;
> * striping cell size;
> * concurrent readers/writers using processes or threads.
> The tool should be easy to use and better to avoid unnecessary local
> environment impact, like local disk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8968) New benchmark throughput tool for striping erasure coding
[ https://issues.apache.org/jira/browse/HDFS-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14990964#comment-14990964 ] Rui Li commented on HDFS-8968: -- Thanks [~rakeshr] for the detailed and helpful comments. I'll update the patch accordingly. Beside I don't understand the findbugs and asflicense error in the test report. The findbugs link returns a 404 and I do have the license header in my patch. Any idea what goes wrong? Thanks! > New benchmark throughput tool for striping erasure coding > - > > Key: HDFS-8968 > URL: https://issues.apache.org/jira/browse/HDFS-8968 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > Attachments: HDFS-8968-HDFS-7285.1.patch, > HDFS-8968-HDFS-7285.2.patch, HDFS-8968.3.patch > > > We need a new benchmark tool to measure the throughput of client writing and > reading considering cases or factors: > * 3-replica or striping; > * write or read, stateful read or positional read; > * which erasure coder; > * striping cell size; > * concurrent readers/writers using processes or threads. > The tool should be easy to use and better to avoid unnecessary local > environment impact, like local disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8968) New benchmark throughput tool for striping erasure coding
[ https://issues.apache.org/jira/browse/HDFS-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HDFS-8968: - Attachment: HDFS-8968.3.patch Rebase my patch with latest trunk code. > New benchmark throughput tool for striping erasure coding > - > > Key: HDFS-8968 > URL: https://issues.apache.org/jira/browse/HDFS-8968 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > Attachments: HDFS-8968-HDFS-7285.1.patch, > HDFS-8968-HDFS-7285.2.patch, HDFS-8968.3.patch > > > We need a new benchmark tool to measure the throughput of client writing and > reading considering cases or factors: > * 3-replica or striping; > * write or read, stateful read or positional read; > * which erasure coder; > * striping cell size; > * concurrent readers/writers using processes or threads. > The tool should be easy to use and better to avoid unnecessary local > environment impact, like local disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9345) Erasure Coding: create dummy coder and schema
[ https://issues.apache.org/jira/browse/HDFS-9345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14986864#comment-14986864 ] Rui Li commented on HDFS-9345: -- [~drankye] told me that dummy schema depends on some other JIRAs. So I just created HADOOP-12544 to implement the dummy coder. I'll use this JIRA for the dummy schema when the depended tasks are done. > Erasure Coding: create dummy coder and schema > - > > Key: HDFS-9345 > URL: https://issues.apache.org/jira/browse/HDFS-9345 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li > > We can create dummy coder which does no computation and simply returns zero > bytes. Similarly, we can create a test-only schema with no parity blocks. > Such coder and schema can be used to isolate the performance issue to > HDFS-side logic instead of codec, which would be useful when tuning > performance of EC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9345) Erasure Coding: create dummy coder and schema
Rui Li created HDFS-9345: Summary: Erasure Coding: create dummy coder and schema Key: HDFS-9345 URL: https://issues.apache.org/jira/browse/HDFS-9345 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Rui Li Assignee: Rui Li We can create dummy coder which does no computation and simply returns zero bytes. Similarly, we can create a test-only schema with no parity blocks. Such coder and schema can be used to isolate the performance issue to HDFS-side logic instead of codec, which would be useful when tuning performance of EC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8920) Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt performance
[
https://issues.apache.org/jira/browse/HDFS-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14902070#comment-14902070
]
Rui Li commented on HDFS-8920:
--
Thanks guys for the review.
> Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt
> performance
> -
>
> Key: HDFS-8920
> URL: https://issues.apache.org/jira/browse/HDFS-8920
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
> Fix For: HDFS-7285
>
> Attachments: HDFS-8920-HDFS-7285.1.patch, HDFS-8920-HDFS-7285.2.patch
>
>
> When we test reading data with datanodes killed,
> {{DFSInputStream::getBestNodeDNAddrPair}} becomes a hot spot method and
> effectively blocks the client JVM. This log seems too verbose:
> {code}
> if (chosenNode == null) {
> DFSClient.LOG.warn("No live nodes contain block " + block.getBlock() +
> " after checking nodes = " + Arrays.toString(nodes) +
> ", ignoredNodes = " + ignoredNodes);
> return null;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8920) Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt performance
[
https://issues.apache.org/jira/browse/HDFS-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HDFS-8920:
-
Attachment: HDFS-8920-HDFS-7285.2.patch
Address Kai's comments offline.
> Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt
> performance
> -
>
> Key: HDFS-8920
> URL: https://issues.apache.org/jira/browse/HDFS-8920
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HDFS-8920-HDFS-7285.1.patch, HDFS-8920-HDFS-7285.2.patch
>
>
> When we test reading data with datanodes killed,
> {{DFSInputStream::getBestNodeDNAddrPair}} becomes a hot spot method and
> effectively blocks the client JVM. This log seems too verbose:
> {code}
> if (chosenNode == null) {
> DFSClient.LOG.warn("No live nodes contain block " + block.getBlock() +
> " after checking nodes = " + Arrays.toString(nodes) +
> ", ignoredNodes = " + ignoredNodes);
> return null;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8968) New benchmark throughput tool for striping erasure coding
[ https://issues.apache.org/jira/browse/HDFS-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14804972#comment-14804972 ] Rui Li commented on HDFS-8968: -- [~andrew.wang] - It's good idea to put it in hadoop-tools. Problem is that I don't see any appropriate existing module where the tool fits in (the tool benchmarks throughput of read/write files for both EC and replica mode). Does it make sense to create a new module, say hadoop-benchmark, under hadoop-tools? Other upcoming benchmark tools can also be put there, e.g. HADOOP-11588, which benchmarks performance of different EC codecs. As to unit test, maybe I can add a test where the tool runs against a MiniDFSCluster. Sounds good? > New benchmark throughput tool for striping erasure coding > - > > Key: HDFS-8968 > URL: https://issues.apache.org/jira/browse/HDFS-8968 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > Attachments: HDFS-8968-HDFS-7285.1.patch, HDFS-8968-HDFS-7285.2.patch > > > We need a new benchmark tool to measure the throughput of client writing and > reading considering cases or factors: > * 3-replica or striping; > * write or read, stateful read or positional read; > * which erasure coder; > * striping cell size; > * concurrent readers/writers using processes or threads. > The tool should be easy to use and better to avoid unnecessary local > environment impact, like local disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8920) Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt performance
[
https://issues.apache.org/jira/browse/HDFS-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14791532#comment-14791532
]
Rui Li commented on HDFS-8920:
--
The latest failures still can't be reproduced locally.
> Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt
> performance
> -
>
> Key: HDFS-8920
> URL: https://issues.apache.org/jira/browse/HDFS-8920
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HDFS-8920-HDFS-7285.1.patch
>
>
> When we test reading data with datanodes killed,
> {{DFSInputStream::getBestNodeDNAddrPair}} becomes a hot spot method and
> effectively blocks the client JVM. This log seems too verbose:
> {code}
> if (chosenNode == null) {
> DFSClient.LOG.warn("No live nodes contain block " + block.getBlock() +
> " after checking nodes = " + Arrays.toString(nodes) +
> ", ignoredNodes = " + ignoredNodes);
> return null;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8968) New benchmark throughput tool for striping erasure coding
[ https://issues.apache.org/jira/browse/HDFS-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14791444#comment-14791444 ] Rui Li commented on HDFS-8968: -- Hi [~zhz] and [~andrew.wang], the tool is intended to benchmark end to end throughput on a real cluster. So it doesn't run against a mini cluster and I'm not sure how to add UT for it. Any suggestions? > New benchmark throughput tool for striping erasure coding > - > > Key: HDFS-8968 > URL: https://issues.apache.org/jira/browse/HDFS-8968 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > Attachments: HDFS-8968-HDFS-7285.1.patch, HDFS-8968-HDFS-7285.2.patch > > > We need a new benchmark tool to measure the throughput of client writing and > reading considering cases or factors: > * 3-replica or striping; > * write or read, stateful read or positional read; > * which erasure coder; > * striping cell size; > * concurrent readers/writers using processes or threads. > The tool should be easy to use and better to avoid unnecessary local > environment impact, like local disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8920) Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt performance
[
https://issues.apache.org/jira/browse/HDFS-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746906#comment-14746906
]
Rui Li commented on HDFS-8920:
--
I tried several failures locally and cannot reproduce them. Also both
{{TestDFSInputStream}} and {{TestDFSStripedInputStream}} pass on my side. So I
suppose the failures here are not related.
Since the patch only changes how we print logs, I didn't add new tests for it.
I did manually perform some tests on a cluster, as I mentioned above.
> Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt
> performance
> -
>
> Key: HDFS-8920
> URL: https://issues.apache.org/jira/browse/HDFS-8920
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HDFS-8920-HDFS-7285.1.patch
>
>
> When we test reading data with datanodes killed,
> {{DFSInputStream::getBestNodeDNAddrPair}} becomes a hot spot method and
> effectively blocks the client JVM. This log seems too verbose:
> {code}
> if (chosenNode == null) {
> DFSClient.LOG.warn("No live nodes contain block " + block.getBlock() +
> " after checking nodes = " + Arrays.toString(nodes) +
> ", ignoredNodes = " + ignoredNodes);
> return null;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8968) New benchmark throughput tool for striping erasure coding
[
https://issues.apache.org/jira/browse/HDFS-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746771#comment-14746771
]
Rui Li commented on HDFS-8968:
--
Hi [~drankye], the tool is intended to test and compare throughput of EC and
replication mode. And it does require some EC APIs to run, e.g.
{{DFSClient::setErasureCodingPolicy}}. So I didn't give it a general name. Let
me know if you think otherwise.
> New benchmark throughput tool for striping erasure coding
> -
>
> Key: HDFS-8968
> URL: https://issues.apache.org/jira/browse/HDFS-8968
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-8968-HDFS-7285.1.patch, HDFS-8968-HDFS-7285.2.patch
>
>
> We need a new benchmark tool to measure the throughput of client writing and
> reading considering cases or factors:
> * 3-replica or striping;
> * write or read, stateful read or positional read;
> * which erasure coder;
> * striping cell size;
> * concurrent readers/writers using processes or threads.
> The tool should be easy to use and better to avoid unnecessary local
> environment impact, like local disk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8920) Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt performance
[
https://issues.apache.org/jira/browse/HDFS-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HDFS-8920:
-
Status: Patch Available (was: Open)
> Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt
> performance
> -
>
> Key: HDFS-8920
> URL: https://issues.apache.org/jira/browse/HDFS-8920
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HDFS-8920-HDFS-7285.1.patch
>
>
> When we test reading data with datanodes killed,
> {{DFSInputStream::getBestNodeDNAddrPair}} becomes a hot spot method and
> effectively blocks the client JVM. This log seems too verbose:
> {code}
> if (chosenNode == null) {
> DFSClient.LOG.warn("No live nodes contain block " + block.getBlock() +
> " after checking nodes = " + Arrays.toString(nodes) +
> ", ignoredNodes = " + ignoredNodes);
> return null;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8920) Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt performance
[
https://issues.apache.org/jira/browse/HDFS-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HDFS-8920:
-
Attachment: HDFS-8920-HDFS-7285.1.patch
Sorry for taking some time to get back to this. Based on my discussion with Kai
offline, we think it's better to do this in {{DFSStripedInputStream}},
therefore the non-EC path won't be affected.
A simple test shows that the patch reduces the log file size from 60MB to 49KB,
when 5 clients read 10GB file respectively with 2 DN killed.
> Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt
> performance
> -
>
> Key: HDFS-8920
> URL: https://issues.apache.org/jira/browse/HDFS-8920
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HDFS-8920-HDFS-7285.1.patch
>
>
> When we test reading data with datanodes killed,
> {{DFSInputStream::getBestNodeDNAddrPair}} becomes a hot spot method and
> effectively blocks the client JVM. This log seems too verbose:
> {code}
> if (chosenNode == null) {
> DFSClient.LOG.warn("No live nodes contain block " + block.getBlock() +
> " after checking nodes = " + Arrays.toString(nodes) +
> ", ignoredNodes = " + ignoredNodes);
> return null;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8968) New benchmark throughput tool for striping erasure coding
[ https://issues.apache.org/jira/browse/HDFS-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14742837#comment-14742837 ] Rui Li commented on HDFS-8968: -- Findbugs seems to have 4 new warnings but Findbugs Report is just empty. Any ideas? > New benchmark throughput tool for striping erasure coding > - > > Key: HDFS-8968 > URL: https://issues.apache.org/jira/browse/HDFS-8968 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > Attachments: HDFS-8968-HDFS-7285.1.patch, HDFS-8968-HDFS-7285.2.patch > > > We need a new benchmark tool to measure the throughput of client writing and > reading considering cases or factors: > * 3-replica or striping; > * write or read, stateful read or positional read; > * which erasure coder; > * striping cell size; > * concurrent readers/writers using processes or threads. > The tool should be easy to use and better to avoid unnecessary local > environment impact, like local disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8968) New benchmark throughput tool for striping erasure coding
[ https://issues.apache.org/jira/browse/HDFS-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HDFS-8968: - Attachment: HDFS-8968-HDFS-7285.2.patch Rebase my patch and address comments. Thanks [~drankye] for your inputs. > New benchmark throughput tool for striping erasure coding > - > > Key: HDFS-8968 > URL: https://issues.apache.org/jira/browse/HDFS-8968 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > Attachments: HDFS-8968-HDFS-7285.1.patch, HDFS-8968-HDFS-7285.2.patch > > > We need a new benchmark tool to measure the throughput of client writing and > reading considering cases or factors: > * 3-replica or striping; > * write or read, stateful read or positional read; > * which erasure coder; > * striping cell size; > * concurrent readers/writers using processes or threads. > The tool should be easy to use and better to avoid unnecessary local > environment impact, like local disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8968) New benchmark throughput tool for striping erasure coding
[ https://issues.apache.org/jira/browse/HDFS-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14734521#comment-14734521 ] Rui Li commented on HDFS-8968: -- Failed tests don't seem related. The patch here doesn't modify any existing code. > New benchmark throughput tool for striping erasure coding > - > > Key: HDFS-8968 > URL: https://issues.apache.org/jira/browse/HDFS-8968 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > Attachments: HDFS-8968-HDFS-7285.1.patch > > > We need a new benchmark tool to measure the throughput of client writing and > reading considering cases or factors: > * 3-replica or striping; > * write or read, stateful read or positional read; > * which erasure coder; > * striping cell size; > * concurrent readers/writers using processes or threads. > The tool should be easy to use and better to avoid unnecessary local > environment impact, like local disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8968) New benchmark throughput tool for striping erasure coding
[ https://issues.apache.org/jira/browse/HDFS-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HDFS-8968: - Status: Patch Available (was: In Progress) I didn't add new test as it's an end to end throughput benchmark tool. > New benchmark throughput tool for striping erasure coding > - > > Key: HDFS-8968 > URL: https://issues.apache.org/jira/browse/HDFS-8968 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > Attachments: HDFS-8968-HDFS-7285.1.patch > > > We need a new benchmark tool to measure the throughput of client writing and > reading considering cases or factors: > * 3-replica or striping; > * write or read, stateful read or positional read; > * which erasure coder; > * striping cell size; > * concurrent readers/writers using processes or threads. > The tool should be easy to use and better to avoid unnecessary local > environment impact, like local disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8968) New benchmark throughput tool for striping erasure coding
[ https://issues.apache.org/jira/browse/HDFS-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HDFS-8968: - Attachment: HDFS-8968-HDFS-7285.1.patch > New benchmark throughput tool for striping erasure coding > - > > Key: HDFS-8968 > URL: https://issues.apache.org/jira/browse/HDFS-8968 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > Attachments: HDFS-8968-HDFS-7285.1.patch > > > We need a new benchmark tool to measure the throughput of client writing and > reading considering cases or factors: > * 3-replica or striping; > * write or read, stateful read or positional read; > * which erasure coder; > * striping cell size; > * concurrent readers/writers using processes or threads. > The tool should be easy to use and better to avoid unnecessary local > environment impact, like local disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8833) Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones
[
https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14733224#comment-14733224
]
Rui Li commented on HDFS-8833:
--
Hey [~zhz], we'd like to try out your patch. But it seems v5 patch doesn't
apply to latest HDFS-7285 branch. Would you mind rebase your patch? Thanks.
> Erasure coding: store EC schema and cell size in INodeFile and eliminate
> notion of EC zones
> ---
>
> Key: HDFS-8833
> URL: https://issues.apache.org/jira/browse/HDFS-8833
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8833-HDFS-7285-merge.00.patch,
> HDFS-8833-HDFS-7285-merge.01.patch, HDFS-8833-HDFS-7285.02.patch,
> HDFS-8833-HDFS-7285.03.patch, HDFS-8833-HDFS-7285.04.patch,
> HDFS-8833-HDFS-7285.05.patch
>
>
> We have [discussed |
> https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14357754&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14357754]
> storing EC schema with files instead of EC zones and recently revisited the
> discussion under HDFS-8059.
> As a recap, the _zone_ concept has severe limitations including renaming and
> nested configuration. Those limitations are valid in encryption for security
> reasons and it doesn't make sense to carry them over in EC.
> This JIRA aims to store EC schema and cell size on {{INodeFile}} level. For
> simplicity, we should first implement it as an xattr and consider memory
> optimizations (such as moving it to file header) as a follow-on. We should
> also disable changing EC policy on a non-empty file / dir in the first phase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7285) Erasure Coding Support inside HDFS
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14723077#comment-14723077 ] Rui Li commented on HDFS-7285: -- Yep. We intend to create a throughput benchmark tool in HDFS-8968. HADOOP-11588 is something like a micro-benchmark, which benchmarks performance at an API level. > Erasure Coding Support inside HDFS > -- > > Key: HDFS-7285 > URL: https://issues.apache.org/jira/browse/HDFS-7285 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Weihua Jiang >Assignee: Zhe Zhang > Attachments: Compare-consolidated-20150824.diff, > Consolidated-20150707.patch, Consolidated-20150806.patch, > Consolidated-20150810.patch, ECAnalyzer.py, ECParser.py, > HDFS-7285-initial-PoC.patch, HDFS-7285-merge-consolidated-01.patch, > HDFS-7285-merge-consolidated-trunk-01.patch, > HDFS-7285-merge-consolidated.trunk.03.patch, > HDFS-7285-merge-consolidated.trunk.04.patch, > HDFS-EC-Merge-PoC-20150624.patch, HDFS-EC-merge-consolidated-01.patch, > HDFS-bistriped.patch, HDFSErasureCodingDesign-20141028.pdf, > HDFSErasureCodingDesign-20141217.pdf, HDFSErasureCodingDesign-20150204.pdf, > HDFSErasureCodingDesign-20150206.pdf, HDFSErasureCodingPhaseITestPlan.pdf, > HDFSErasureCodingSystemTestPlan-20150824.pdf, > HDFSErasureCodingSystemTestReport-20150826.pdf, fsimage-analysis-20150105.pdf > > > Erasure Coding (EC) can greatly reduce the storage overhead without sacrifice > of data reliability, comparing to the existing HDFS 3-replica approach. For > example, if we use a 10+4 Reed Solomon coding, we can allow loss of 4 blocks, > with storage overhead only being 40%. This makes EC a quite attractive > alternative for big data storage, particularly for cold data. > Facebook had a related open source project called HDFS-RAID. It used to be > one of the contribute packages in HDFS but had been removed since Hadoop 2.0 > for maintain reason. The drawbacks are: 1) it is on top of HDFS and depends > on MapReduce to do encoding and decoding tasks; 2) it can only be used for > cold files that are intended not to be appended anymore; 3) the pure Java EC > coding implementation is extremely slow in practical use. Due to these, it > might not be a good idea to just bring HDFS-RAID back. > We (Intel and Cloudera) are working on a design to build EC into HDFS that > gets rid of any external dependencies, makes it self-contained and > independently maintained. This design lays the EC feature on the storage type > support and considers compatible with existing HDFS features like caching, > snapshot, encryption, high availability and etc. This design will also > support different EC coding schemes, implementations and policies for > different deployment scenarios. By utilizing advanced libraries (e.g. Intel > ISA-L library), an implementation can greatly improve the performance of EC > encoding/decoding and makes the EC solution even more attractive. We will > post the design document soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7285) Erasure Coding Support inside HDFS
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14723022#comment-14723022 ] Rui Li commented on HDFS-7285: -- [~brahmareddy] - Yeah I'm working on the two issues to support parallel read/write benchmarking. > Erasure Coding Support inside HDFS > -- > > Key: HDFS-7285 > URL: https://issues.apache.org/jira/browse/HDFS-7285 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Weihua Jiang >Assignee: Zhe Zhang > Attachments: Compare-consolidated-20150824.diff, > Consolidated-20150707.patch, Consolidated-20150806.patch, > Consolidated-20150810.patch, ECAnalyzer.py, ECParser.py, > HDFS-7285-initial-PoC.patch, HDFS-7285-merge-consolidated-01.patch, > HDFS-7285-merge-consolidated-trunk-01.patch, > HDFS-7285-merge-consolidated.trunk.03.patch, > HDFS-7285-merge-consolidated.trunk.04.patch, > HDFS-EC-Merge-PoC-20150624.patch, HDFS-EC-merge-consolidated-01.patch, > HDFS-bistriped.patch, HDFSErasureCodingDesign-20141028.pdf, > HDFSErasureCodingDesign-20141217.pdf, HDFSErasureCodingDesign-20150204.pdf, > HDFSErasureCodingDesign-20150206.pdf, HDFSErasureCodingPhaseITestPlan.pdf, > HDFSErasureCodingSystemTestPlan-20150824.pdf, > HDFSErasureCodingSystemTestReport-20150826.pdf, fsimage-analysis-20150105.pdf > > > Erasure Coding (EC) can greatly reduce the storage overhead without sacrifice > of data reliability, comparing to the existing HDFS 3-replica approach. For > example, if we use a 10+4 Reed Solomon coding, we can allow loss of 4 blocks, > with storage overhead only being 40%. This makes EC a quite attractive > alternative for big data storage, particularly for cold data. > Facebook had a related open source project called HDFS-RAID. It used to be > one of the contribute packages in HDFS but had been removed since Hadoop 2.0 > for maintain reason. The drawbacks are: 1) it is on top of HDFS and depends > on MapReduce to do encoding and decoding tasks; 2) it can only be used for > cold files that are intended not to be appended anymore; 3) the pure Java EC > coding implementation is extremely slow in practical use. Due to these, it > might not be a good idea to just bring HDFS-RAID back. > We (Intel and Cloudera) are working on a design to build EC into HDFS that > gets rid of any external dependencies, makes it self-contained and > independently maintained. This design lays the EC feature on the storage type > support and considers compatible with existing HDFS features like caching, > snapshot, encryption, high availability and etc. This design will also > support different EC coding schemes, implementations and policies for > different deployment scenarios. By utilizing advanced libraries (e.g. Intel > ISA-L library), an implementation can greatly improve the performance of EC > encoding/decoding and makes the EC solution even more attractive. We will > post the design document soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HDFS-8968) New benchmark throughput tool for striping erasure coding
[ https://issues.apache.org/jira/browse/HDFS-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-8968 started by Rui Li. > New benchmark throughput tool for striping erasure coding > - > > Key: HDFS-8968 > URL: https://issues.apache.org/jira/browse/HDFS-8968 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kai Zheng >Assignee: Rui Li > > We need a new benchmark tool to measure the throughput of client writing and > reading considering cases or factors: > * 3-replica or striping; > * write or read, stateful read or positional read; > * which erasure coder; > * striping cell size; > * concurrent readers/writers using processes or threads. > The tool should be easy to use and better to avoid unnecessary local > environment impact, like local disk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8987) Erasure coding: MapReduce job failed when I set the / foler to the EC zone
[
https://issues.apache.org/jira/browse/HDFS-8987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HDFS-8987:
-
Description:
Test progress is as follows
* For a new cluster, I format the namenode and then start HDFS service.
* After HDFS service is started, there is no files in HDFS and set the /
folder to the EC zone. the EC zone is created successfully.
* Start the yarn and mr JobHistoryServer services. All the services start
successfully.
* Then run hadoop example pi program and it failed.
The following is the exception.
{noformat}
```
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.UnsupportedActionException):
Cannot set replication to a file with striped blocks
at
org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.unprotectedSetReplication(FSDirAttrOp.java:391)
at
org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setReplication(FSDirAttrOp.java:151)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setReplication(FSNamesystem.java:2231)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setReplication(NameNodeRpcServer.java:682)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setReplication(ClientNamenodeProtocolServerSideTranslatorPB.java:445)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2171)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2165)
```
{noformat}
was:
Test progress is as follows
* For a new cluster, I format the namenode and then start HDFS service.
* After HDFS service is started, there is no files in HDFS and set the /
folder to the EC zone. the EC zone is created successfully.
* Start the yarn and mr JobHistoryServer services. All the services start
successfully.
* Then run hadoop example pi program and it failed.
The following is the exception.
```
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.UnsupportedActionException):
Cannot set replication to a file with striped blocks
at
org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.unprotectedSetReplication(FSDirAttrOp.java:391)
at
org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setReplication(FSDirAttrOp.java:151)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setReplication(FSNamesystem.java:2231)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setReplication(NameNodeRpcServer.java:682)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setReplication(ClientNamenodeProtocolServerSideTranslatorPB.java:445)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2171)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2165)
```
> Erasure coding: MapReduce job failed when I set the / foler to the EC zone
> ---
>
> Key: HDFS-8987
> URL: https://issues.apache.org/jira/browse/HDFS-8987
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: HDFS
>Affects Versions: 3.0.0
>Reporter: Lifeng Wang
>
> Test progress is as follows
> * For a new cluster, I format the namenode and then start HDFS service.
> * After HDFS service is started, there is no files in HDFS and set the /
> folder to the EC zone. the EC zone is created successfully.
> * Start the yarn and mr JobHistoryServer services. All the services start
> successfully.
> * Then run hadoop example pi program and it failed.
> The following is the exception.
> {noformat}
> ```
> org
