[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[
https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240774#comment-15240774
]
Kai Zheng commented on HADOOP-12924:
Thanks Rui for the great update and nice tests! Two more comments:
* Maybe we can define RS_DEFAULT_CODEC_NAME and the like in an enum?
* Could we move the static schema definitions like RS_6_3_SCHEMA out of
{{HdfsConstants}} maybe to {{CodecUtil}}? HdfsConstants isn't a good place for
such.
> Add default coder key for creating raw coders
> -
>
> Key: HADOOP-12924
> URL: https://issues.apache.org/jira/browse/HADOOP-12924
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Minor
> Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-12924.1.patch, HADOOP-12924.2.patch,
> HADOOP-12924.3.patch, HADOOP-12924.4.patch
>
>
> As suggested
> [here|https://issues.apache.org/jira/browse/HADOOP-12826?focusedCommentId=15194402&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15194402].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[ https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240534#comment-15240534 ] Rui Li commented on HADOOP-12924: - All the failures either cannot be reproduced or fail on trunk as well. So I think they're not related to the patch. > Add default coder key for creating raw coders > - > > Key: HADOOP-12924 > URL: https://issues.apache.org/jira/browse/HADOOP-12924 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Labels: hdfs-ec-3.0-must-do > Attachments: HADOOP-12924.1.patch, HADOOP-12924.2.patch, > HADOOP-12924.3.patch, HADOOP-12924.4.patch > > > As suggested > [here|https://issues.apache.org/jira/browse/HADOOP-12826?focusedCommentId=15194402&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15194402]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[
https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15239686#comment-15239686
]
Hadoop QA commented on HADOOP-12924:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 37s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 30s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 20s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 5s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 13s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 12s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 34s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 31s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 41s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_77.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 53s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_77. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 70m 2s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_77. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 28s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 1s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_95. {color} |
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[
https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15238526#comment-15238526
]
Rui Li commented on HADOOP-12924:
-
Thanks [~drankye] for the review!
1. Actually I do get NPEs in tests if I don't check whether conf is null.
2. Yes will do.
3. I think you mean the changes in {{TestHHXORErasureCoder}} and
{{TestRSErasureCoder}}? They're just renames because we have renamed the raw
coder config keys. And it's not configuring the codec but raw coder. Anyway
like you said, if we don't set those configs, we'll be using the default ones.
Do you think we should remove them?
4. Could you elaborate on what check should be done?
> Add default coder key for creating raw coders
> -
>
> Key: HADOOP-12924
> URL: https://issues.apache.org/jira/browse/HADOOP-12924
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Minor
> Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-12924.1.patch, HADOOP-12924.2.patch,
> HADOOP-12924.3.patch
>
>
> As suggested
> [here|https://issues.apache.org/jira/browse/HADOOP-12826?focusedCommentId=15194402&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15194402].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[
https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15238366#comment-15238366
]
Kai Zheng commented on HADOOP-12924:
Thanks [~lirui] for the nice work.
1. In {{CodecUtil}}, I guess we can always assert conf != null to simplify the
codes.
2. Maybe we can add {{getCodec}} method to {{ErasureCodingPolicy}}, as minor.
3. I don't think we have to set the codec property to {{conf}} in HH and test
related codes, as they can use the default one.
4. Would be good to add a new test to check all of these {{codecName->coder}}
mappings.
> Add default coder key for creating raw coders
> -
>
> Key: HADOOP-12924
> URL: https://issues.apache.org/jira/browse/HADOOP-12924
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Minor
> Labels: hdfs-ec-3.0-must-do
> Attachments: HADOOP-12924.1.patch, HADOOP-12924.2.patch,
> HADOOP-12924.3.patch
>
>
> As suggested
> [here|https://issues.apache.org/jira/browse/HADOOP-12826?focusedCommentId=15194402&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15194402].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[
https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221398#comment-15221398
]
Hadoop QA commented on HADOOP-12924:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 54s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 39s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 22s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 9s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 25s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 15s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 47s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 48s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 18s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 59s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_77. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 55s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_77. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 58m 23s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_77. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 36s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 3s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_95. {color} |
| {co
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[ https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219811#comment-15219811 ] Rui Li commented on HADOOP-12924: - Test failures are not related. I don't quite understand the "incorrect indentation level" reported by checkstyle, though. > Add default coder key for creating raw coders > - > > Key: HADOOP-12924 > URL: https://issues.apache.org/jira/browse/HADOOP-12924 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HADOOP-12924.1.patch, HADOOP-12924.2.patch > > > As suggested > [here|https://issues.apache.org/jira/browse/HADOOP-12826?focusedCommentId=15194402&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15194402]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[
https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219723#comment-15219723
]
Hadoop QA commented on HADOOP-12924:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 8s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 59s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 40s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 41s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 21s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 8s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 26s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 14s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 52s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 42s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 42s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 4s
{color} | {color:red} root: patch generated 17 new + 158 unchanged - 0 fixed =
175 total (was 158) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 18s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 10s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 58s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_77. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 51s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_77. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 55m 25s
{color} | {color:green} hadoop-hdfs in the patch passed with JDK v1.8.0_77.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 6s {color} |
{color:red} hadoop-common in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s
{color} | {color:green} hadoop-hdfs-client in th
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[ https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215500#comment-15215500 ] Kai Zheng commented on HADOOP-12924: >>For the other codec, how about rs-default? Cool! Thanks for the nice name, Zhe. > Add default coder key for creating raw coders > - > > Key: HADOOP-12924 > URL: https://issues.apache.org/jira/browse/HADOOP-12924 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HADOOP-12924.1.patch > > > As suggested > [here|https://issues.apache.org/jira/browse/HADOOP-12826?focusedCommentId=15194402&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15194402]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[
https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215487#comment-15215487
]
Zhe Zhang commented on HADOOP-12924:
Thanks Rui and Kai! The two points Rui proposed above make sense to me.
I also like the {{rs-legacy}} name. For the other codec, how about
{{rs-default}}?
> Add default coder key for creating raw coders
> -
>
> Key: HADOOP-12924
> URL: https://issues.apache.org/jira/browse/HADOOP-12924
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Minor
> Attachments: HADOOP-12924.1.patch
>
>
> As suggested
> [here|https://issues.apache.org/jira/browse/HADOOP-12826?focusedCommentId=15194402&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15194402].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[
https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15213877#comment-15213877
]
Kai Zheng commented on HADOOP-12924:
Thanks [~lirui] for the nice summary! it's fine to me.
We may need to define the required codec name and configuration key here right
now to proceed with. How about:
* For the HDFS-RAID coder/codec: {{rs-legacy}};
* For the ISA-L compatible coder/codec: {{rs-isal}}.
* Configuration key:
org.apache.hadoop.erasurecode.codec.{{CODEC-NAME}}.rawcoder, this configures
the corresponding raw coder factory class.
Sounds good? Thanks.
> Add default coder key for creating raw coders
> -
>
> Key: HADOOP-12924
> URL: https://issues.apache.org/jira/browse/HADOOP-12924
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Minor
> Attachments: HADOOP-12924.1.patch
>
>
> As suggested
> [here|https://issues.apache.org/jira/browse/HADOOP-12826?focusedCommentId=15194402&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15194402].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[ https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15210167#comment-15210167 ] Rui Li commented on HADOOP-12924: - Based on my offline discussion with Kai, we have the following proposal: # We can consider HDFS-RAID and new Java coder as different codec, so that they'll map to different EC policies. Within each codec, we can have different implementations of raw coders which are interoperable. For HDFS-RAID codec, the legacy coder is the only implementation. For "other RS" codec (we can figure out a better name of course), we have the new Java coder and will probably have the ISA-L coder soon. # Each codec will have a dedicated raw coder configuration key, so that user can conveniently switch between different implementations. And switching raw coder doesn't require porting/re-encoding data, as long as the raw coders belong to the same codec. [~drankye] please add if I missed anything. [~zhz] what's your opinion about this? > Add default coder key for creating raw coders > - > > Key: HADOOP-12924 > URL: https://issues.apache.org/jira/browse/HADOOP-12924 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HADOOP-12924.1.patch > > > As suggested > [here|https://issues.apache.org/jira/browse/HADOOP-12826?focusedCommentId=15194402&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15194402]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[ https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15209989#comment-15209989 ] Kai Zheng commented on HADOOP-12924: bq. I think the bottomline is that the file metadata (stored in the header) should present HDFS-RAID and the new Java RS coder differently. Yeah, agree. bq. The concept of Reed-Solomon / RS is actually very broad. Strictly speaking XOR is also a special case of RS. I appreciate this understanding. It's true. Going this way would avoid awkward situations that missed data are not recoverable due to the change of underlying concrete coders in the umbrella of the same so-called Reed-Solomon. I have had an off-line discussion with [~lirui] about the required changes to fill this gap raised here. Hopefully we can move on! > Add default coder key for creating raw coders > - > > Key: HADOOP-12924 > URL: https://issues.apache.org/jira/browse/HADOOP-12924 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HADOOP-12924.1.patch > > > As suggested > [here|https://issues.apache.org/jira/browse/HADOOP-12826?focusedCommentId=15194402&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15194402]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[
https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15207952#comment-15207952
]
Zhe Zhang commented on HADOOP-12924:
Technically, any coder that yields new numerical results should be a new
policy. Therefore both the dummy coder and the legacy / HDFS-RAID RS coder
should have their own policies.
[~drankye] I think the bottomline is that the file metadata (stored in the
header) should present HDFS-RAID and the new Java RS coder differently.
Otherwise a reader won't be able to correctly decode the data. I think we
should just create a new policy for HDFS-RAID coder (and one for the dummy
coder). The concept of Reed-Solomon / {{RS}} is actually very broad. Strictly
speaking XOR is also a special case of RS. So I think our
{{ErasureCodingPolicy}} (and the name of each policy) should be explicit about
what type of RS is being used.
> Add default coder key for creating raw coders
> -
>
> Key: HADOOP-12924
> URL: https://issues.apache.org/jira/browse/HADOOP-12924
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Minor
> Attachments: HADOOP-12924.1.patch
>
>
> As suggested
> [here|https://issues.apache.org/jira/browse/HADOOP-12826?focusedCommentId=15194402&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15194402].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[ https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15207635#comment-15207635 ] Kai Zheng commented on HADOOP-12924: [~lirui] and [~zhz], how do you think doing the above change here to support multiple raw erasure coders or codecs, while addressing the question/concern of Zhe? This may further close some gaps left in the around. > Add default coder key for creating raw coders > - > > Key: HADOOP-12924 > URL: https://issues.apache.org/jira/browse/HADOOP-12924 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HADOOP-12924.1.patch > > > As suggested > [here|https://issues.apache.org/jira/browse/HADOOP-12826?focusedCommentId=15194402&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15194402]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[ https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205952#comment-15205952 ] Kai Zheng commented on HADOOP-12924: Ping [~andrew.wang], in case needed. > Add default coder key for creating raw coders > - > > Key: HADOOP-12924 > URL: https://issues.apache.org/jira/browse/HADOOP-12924 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HADOOP-12924.1.patch > > > As suggested > [here|https://issues.apache.org/jira/browse/HADOOP-12826?focusedCommentId=15194402&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15194402]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[
https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205947#comment-15205947
]
Kai Zheng commented on HADOOP-12924:
OK, it won't be bad to have a policy for the legacy coder, since we would have
one for the dummy coder. So, what's needed for now would be, a mapping between
{{codec-name}} and {{raw-coder-factory-class}}. In the original design by
HDFS-7337, it would be done by {{ErasureCodec}}, but for phase 1 and right now,
for simple we can have a simple enum or hard-coded mapping codes somewhere. For
the legacy and dummy policies, we just hard-code the raw-coder-factory class;
for the default RS policy, in addition to the hard-coded default raw coder, we
allow to configure the class just as current code does using the default coder
key. Sounds good?
> Add default coder key for creating raw coders
> -
>
> Key: HADOOP-12924
> URL: https://issues.apache.org/jira/browse/HADOOP-12924
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Minor
> Attachments: HADOOP-12924.1.patch
>
>
> As suggested
> [here|https://issues.apache.org/jira/browse/HADOOP-12826?focusedCommentId=15194402&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15194402].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[ https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205942#comment-15205942 ] Rui Li commented on HADOOP-12924: - Actually we also have dummy raw coders, which I think should work with any EC policies. Maybe we need a more thorough discussion about the relationship between coders and policies. > Add default coder key for creating raw coders > - > > Key: HADOOP-12924 > URL: https://issues.apache.org/jira/browse/HADOOP-12924 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HADOOP-12924.1.patch > > > As suggested > [here|https://issues.apache.org/jira/browse/HADOOP-12826?focusedCommentId=15194402&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15194402]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[ https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205856#comment-15205856 ] Zhe Zhang commented on HADOOP-12924: Thanks for the thoughts Kai. Simplifying logic is a valid concern. However, if we have the legacy HDFS-RAID coder in the Hadoop codebase, there's always a possibility for some user to use it to encode files. Then by looking at file metadata, there's no way to determine whether the encoding was done with legacy or new Java coder. I think we should either add legacy coder as a policy, or take it out from the codebase and make it an external took only for migrating legacy HDFS-RAID data into HDFS-EC. Actually, how could our current legacy coder be used to migrate legacy cluster data? IIUC HDFS-RAID was developed in Facebook's private branch based on upstream version 0.21.0. Is it even possible to run Hadoop 3.0 software on blocks created by 0.21.0? I think an easier way to migrate is to use the original HDFS-RAID code to read data out and write into HDFS-EC. > Add default coder key for creating raw coders > - > > Key: HADOOP-12924 > URL: https://issues.apache.org/jira/browse/HADOOP-12924 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HADOOP-12924.1.patch > > > As suggested > [here|https://issues.apache.org/jira/browse/HADOOP-12826?focusedCommentId=15194402&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15194402]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[ https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198533#comment-15198533 ] Kai Zheng commented on HADOOP-12924: Thanks for the discussions, Zhe and Rui. bq. Kai mentioned that new RS Java coder and legacy RS coder from HDFS-RAID are not interoperable. ... they should be treated as 2 different policies rather than 2 different implementations? Yes right, but I'm not sure we would have the additional policy for the legacy coder. I believe the old coder is only useful for migrating old data from old systems (it's possible because the coder resides in hadoop common side and not tightly coupled with any HDFS version), but the migrating tool may not likely go into the current HDFS-EC implementation. So I suggest we still use the current configuration key for the RS coder, to simplify the current HDFS usage and also the HH coder implementation. In the feature user guide and doc, we should have one section about erasure coders and how to configure/use them. It should explicitly mention the legacy coder isn't compatible with the main RS coders (new Java coder, and ISA-L based native coder). In fact, anytime users change the configured codec or coder, they should be aware of the properties of these coders to understand which one is the right one to be used. > Add default coder key for creating raw coders > - > > Key: HADOOP-12924 > URL: https://issues.apache.org/jira/browse/HADOOP-12924 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HADOOP-12924.1.patch > > > As suggested > [here|https://issues.apache.org/jira/browse/HADOOP-12826?focusedCommentId=15194402&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15194402]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[ https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15196905#comment-15196905 ] Rui Li commented on HADOOP-12924: - Thanks Zhe for the review and comments. Yes I think the concern is valid. If the raw coders are not compatible with each other, we may have to encode it into the policy. And don't allow users to configure it separately. But I'm not sure if that will make it difficult for users to try new coders like ISA-L. Of course let's wait for Kai's inputs. > Add default coder key for creating raw coders > - > > Key: HADOOP-12924 > URL: https://issues.apache.org/jira/browse/HADOOP-12924 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HADOOP-12924.1.patch > > > As suggested > [here|https://issues.apache.org/jira/browse/HADOOP-12826?focusedCommentId=15194402&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15194402]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[
https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15196885#comment-15196885
]
Hadoop QA commented on HADOOP-12924:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 27s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
51s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 53s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 41s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 39s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 34s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 21s
{color} | {color:red} hadoop-common-project/hadoop-common: patch generated 2
new + 77 unchanged - 0 fixed = 79 total (was 77) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 24s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_74. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 10s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 10s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_74 Failed junit tests | hadoop.net.TestDNS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12793682/HADOOP-12924.1.patch |
| JIRA Issue | HADOOP-12924 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 9360da597ae3 3.13.0-36-lowlatency #63-Ubunt
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[
https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15196858#comment-15196858
]
Zhe Zhang commented on HADOOP-12924:
Thanks Rui. Patch itself LGTM.
However I realized some issues in the current code. Pinging [~drankye] for
discussion.
Kai mentioned that new RS Java coder and legacy RS coder from HDFS-RAID are not
interoperable. However, they map to the same EC policy at inode level. E.g. if
a client writes a file using HDFS-RAID coder, we will record EC policy RS-6-3.
Then if another client tries to decode the file with new coder it will fail.
These 2 clients could have different {{IO_ERASURECODE_CODEC_RS_RAWCODER_KEY}}
configured. IIUC, they should be treated as 2 different policies rather than 2
different implementations?
> Add default coder key for creating raw coders
> -
>
> Key: HADOOP-12924
> URL: https://issues.apache.org/jira/browse/HADOOP-12924
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Minor
> Attachments: HADOOP-12924.1.patch
>
>
> As suggested
> [here|https://issues.apache.org/jira/browse/HADOOP-12826?focusedCommentId=15194402&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15194402].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
