[jira] [Commented] (HDFS-13774) EC: "hdfs ec -getPolicy" is not retrieving policy details when the special REPLICATION policy set on the directory
[ https://issues.apache.org/jira/browse/HDFS-13774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599083#comment-16599083 ] Ayush Saxena commented on HDFS-13774: - Thanx [~xiaochen] for review!!! Have updated the patch!!! :) > EC: "hdfs ec -getPolicy" is not retrieving policy details when the special > REPLICATION policy set on the directory > -- > > Key: HDFS-13774 > URL: https://issues.apache.org/jira/browse/HDFS-13774 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0 > Environment: 3 Node Linux Cluster >Reporter: Souryakanta Dwivedy >Assignee: Ayush Saxena >Priority: Minor > Attachments: GetPolicy_EC.png, HDFS-13774-01.patch, > HDFS-13774-02.patch > > > Erasure coding: "hdfs ec -getPolicy"" is not retrieving policy details when > the special REPLICATION policy set on the directory > Steps :- > - Create a directory "testEC" > - Get the EC policy for the directory [Received message as : "The erasure > coding policy of /testEC is unspecified" ] > - Enable any Erasure coding policy like "XOR-2-1-1024k" > - Set the EC Policy on the Directory > - Get the EC policy for the directory [Received message as : "XOR-2-1-1024k" ] > - Now again set the EC Policy on the directory as "replicate" special > REPLICATION policy > - Get the EC policy for the directory [Received message as : "The erasure > coding policy of /testEC is unspecified" ] > The policy is being set for the Directory ,but while retrieving policy > details its throwing error as > policy for the directory is unspecified which is wrong behavior -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13774) EC: "hdfs ec -getPolicy" is not retrieving policy details when the special REPLICATION policy set on the directory
[ https://issues.apache.org/jira/browse/HDFS-13774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599080#comment-16599080 ] genericqa commented on HDFS-13774: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 32s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 31m 17s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 38s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 30s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 47m 30s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 | | JIRA Issue | HDFS-13774 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12937935/HDFS-13774-02.patch | | Optional Tests | dupname asflicense mvnsite | | uname | Linux 4c3d96842857 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 8aa6c4f | | maven | version: Apache Maven 3.3.9 | | Max. process+thread count | 334 (vs. ulimit of 1) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/24925/console | | Powered by | Apache Yetus 0.9.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > EC: "hdfs ec -getPolicy" is not retrieving policy details when the special > REPLICATION policy set on the directory > -- > > Key: HDFS-13774 > URL: https://issues.apache.org/jira/browse/HDFS-13774 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0 > Environment: 3 Node Linux Cluster >Reporter: Souryakanta Dwivedy >Assignee: Ayush Saxena >Priority: Minor > Attachments: GetPolicy_EC.png, HDFS-13774-01.patch, > HDFS-13774-02.patch > > > Erasure coding: "hdfs ec -getPolicy"" is not retrieving policy details when > the special REPLICATION policy set on the directory > Steps :- > - Create a directory "testEC" > - Get the EC policy for the directory [Received message as : "The erasure > coding policy of /testEC is unspecified" ] > - Enable any Erasure coding policy like "XOR-2-1-1024k" > - Set the EC Policy on the Directory > - Get the EC policy for the directory [Received message as : "XOR-2-1-1024k" ] > - Now again set the EC Policy on the directory as "replicate" special > REPLICATION policy > - Get the EC policy for the directory [Received message as : "The erasure > coding policy of /testEC is unspecified" ] > The policy is being set for the Directory ,but while retrieving policy > details its throwing error as > policy for the directory is unspecified which is wrong behavior -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13774) EC: "hdfs ec -getPolicy" is not retrieving policy details when the special REPLICATION policy set on the directory
[ https://issues.apache.org/jira/browse/HDFS-13774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597069#comment-16597069 ] Xiao Chen commented on HDFS-13774: -- bq. For other built-in EC policies, they are disabled by default. This isn't from your change, but we should fix. Out of all the EC policies, RS(6,3) is enabled by default. +1 pending, thanks for the work. > EC: "hdfs ec -getPolicy" is not retrieving policy details when the special > REPLICATION policy set on the directory > -- > > Key: HDFS-13774 > URL: https://issues.apache.org/jira/browse/HDFS-13774 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0 > Environment: 3 Node Linux Cluster >Reporter: Souryakanta Dwivedy >Assignee: Ayush Saxena >Priority: Minor > Attachments: GetPolicy_EC.png, HDFS-13774-01.patch > > > Erasure coding: "hdfs ec -getPolicy"" is not retrieving policy details when > the special REPLICATION policy set on the directory > Steps :- > - Create a directory "testEC" > - Get the EC policy for the directory [Received message as : "The erasure > coding policy of /testEC is unspecified" ] > - Enable any Erasure coding policy like "XOR-2-1-1024k" > - Set the EC Policy on the Directory > - Get the EC policy for the directory [Received message as : "XOR-2-1-1024k" ] > - Now again set the EC Policy on the directory as "replicate" special > REPLICATION policy > - Get the EC policy for the directory [Received message as : "The erasure > coding policy of /testEC is unspecified" ] > The policy is being set for the Directory ,but while retrieving policy > details its throwing error as > policy for the directory is unspecified which is wrong behavior -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13774) EC: "hdfs ec -getPolicy" is not retrieving policy details when the special REPLICATION policy set on the directory
[ https://issues.apache.org/jira/browse/HDFS-13774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596050#comment-16596050 ] Ayush Saxena commented on HDFS-13774: - [~vinayrpet] can you pls review!!! :) > EC: "hdfs ec -getPolicy" is not retrieving policy details when the special > REPLICATION policy set on the directory > -- > > Key: HDFS-13774 > URL: https://issues.apache.org/jira/browse/HDFS-13774 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0 > Environment: 3 Node Linux Cluster >Reporter: Souryakanta Dwivedy >Assignee: Ayush Saxena >Priority: Minor > Attachments: GetPolicy_EC.png, HDFS-13774-01.patch > > > Erasure coding: "hdfs ec -getPolicy"" is not retrieving policy details when > the special REPLICATION policy set on the directory > Steps :- > - Create a directory "testEC" > - Get the EC policy for the directory [Received message as : "The erasure > coding policy of /testEC is unspecified" ] > - Enable any Erasure coding policy like "XOR-2-1-1024k" > - Set the EC Policy on the Directory > - Get the EC policy for the directory [Received message as : "XOR-2-1-1024k" ] > - Now again set the EC Policy on the directory as "replicate" special > REPLICATION policy > - Get the EC policy for the directory [Received message as : "The erasure > coding policy of /testEC is unspecified" ] > The policy is being set for the Directory ,but while retrieving policy > details its throwing error as > policy for the directory is unspecified which is wrong behavior -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13774) EC: "hdfs ec -getPolicy" is not retrieving policy details when the special REPLICATION policy set on the directory
[ https://issues.apache.org/jira/browse/HDFS-13774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593190#comment-16593190 ] genericqa commented on HDFS-13774: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 36s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 30m 46s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 1s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 45m 57s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 | | JIRA Issue | HDFS-13774 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12936939/HDFS-13774-01.patch | | Optional Tests | dupname asflicense mvnsite | | uname | Linux 0427f49de499 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27 10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / a4121c7 | | maven | version: Apache Maven 3.3.9 | | Max. process+thread count | 301 (vs. ulimit of 1) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/24888/console | | Powered by | Apache Yetus 0.9.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > EC: "hdfs ec -getPolicy" is not retrieving policy details when the special > REPLICATION policy set on the directory > -- > > Key: HDFS-13774 > URL: https://issues.apache.org/jira/browse/HDFS-13774 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0 > Environment: 3 Node Linux Cluster >Reporter: Souryakanta Dwivedy >Assignee: Ayush Saxena >Priority: Minor > Attachments: GetPolicy_EC.png, HDFS-13774-01.patch > > > Erasure coding: "hdfs ec -getPolicy"" is not retrieving policy details when > the special REPLICATION policy set on the directory > Steps :- > - Create a directory "testEC" > - Get the EC policy for the directory [Received message as : "The erasure > coding policy of /testEC is unspecified" ] > - Enable any Erasure coding policy like "XOR-2-1-1024k" > - Set the EC Policy on the Directory > - Get the EC policy for the directory [Received message as : "XOR-2-1-1024k" ] > - Now again set the EC Policy on the directory as "replicate" special > REPLICATION policy > - Get the EC policy for the directory [Received message as : "The erasure > coding policy of /testEC is unspecified" ] > The policy is being set for the Directory ,but while retrieving policy > details its throwing error as > policy for the directory is unspecified which is wrong behavior -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13774) EC: "hdfs ec -getPolicy" is not retrieving policy details when the special REPLICATION policy set on the directory
[ https://issues.apache.org/jira/browse/HDFS-13774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591091#comment-16591091 ] Ayush Saxena commented on HDFS-13774: - Thanks [~xiaochen] for the suggestion. Have updated the documentation on this as suggested by you.So as to reduce any confusion that replication is not any kind of EC policy to users. Pls Review!!! > EC: "hdfs ec -getPolicy" is not retrieving policy details when the special > REPLICATION policy set on the directory > -- > > Key: HDFS-13774 > URL: https://issues.apache.org/jira/browse/HDFS-13774 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0 > Environment: 3 Node Linux Cluster >Reporter: Souryakanta Dwivedy >Assignee: Ayush Saxena >Priority: Minor > Attachments: GetPolicy_EC.png, HDFS-13774-01.patch > > > Erasure coding: "hdfs ec -getPolicy"" is not retrieving policy details when > the special REPLICATION policy set on the directory > Steps :- > - Create a directory "testEC" > - Get the EC policy for the directory [Received message as : "The erasure > coding policy of /testEC is unspecified" ] > - Enable any Erasure coding policy like "XOR-2-1-1024k" > - Set the EC Policy on the Directory > - Get the EC policy for the directory [Received message as : "XOR-2-1-1024k" ] > - Now again set the EC Policy on the directory as "replicate" special > REPLICATION policy > - Get the EC policy for the directory [Received message as : "The erasure > coding policy of /testEC is unspecified" ] > The policy is being set for the Directory ,but while retrieving policy > details its throwing error as > policy for the directory is unspecified which is wrong behavior -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13774) EC: "hdfs ec -getPolicy" is not retrieving policy details when the special REPLICATION policy set on the directory
[ https://issues.apache.org/jira/browse/HDFS-13774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583028#comment-16583028 ] Xiao Chen commented on HDFS-13774: -- Thanks for reporting the issue and the discussion [~SouryakantaDwivedy] and [~ayushtkn]. I agree this would be confusing to users - we can update the description and documentation around this. Unfortunately, for compatibility reasons, we cannot change the getPolicy output (e.g. is unspecified). The message is deliberately printed from [here|https://github.com/apache/hadoop/blob/branch-3.0.0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/ECAdmin.java#L238-L243] > EC: "hdfs ec -getPolicy" is not retrieving policy details when the special > REPLICATION policy set on the directory > -- > > Key: HDFS-13774 > URL: https://issues.apache.org/jira/browse/HDFS-13774 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0 > Environment: 3 Node Linux Cluster >Reporter: Souryakanta Dwivedy >Assignee: Ayush Saxena >Priority: Minor > Attachments: GetPolicy_EC.png > > > Erasure coding: "hdfs ec -getPolicy"" is not retrieving policy details when > the special REPLICATION policy set on the directory > Steps :- > - Create a directory "testEC" > - Get the EC policy for the directory [Received message as : "The erasure > coding policy of /testEC is unspecified" ] > - Enable any Erasure coding policy like "XOR-2-1-1024k" > - Set the EC Policy on the Directory > - Get the EC policy for the directory [Received message as : "XOR-2-1-1024k" ] > - Now again set the EC Policy on the directory as "replicate" special > REPLICATION policy > - Get the EC policy for the directory [Received message as : "The erasure > coding policy of /testEC is unspecified" ] > The policy is being set for the Directory ,but while retrieving policy > details its throwing error as > policy for the directory is unspecified which is wrong behavior -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13774) EC: "hdfs ec -getPolicy" is not retrieving policy details when the special REPLICATION policy set on the directory
[ https://issues.apache.org/jira/browse/HDFS-13774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582024#comment-16582024 ] Souryakanta Dwivedy commented on HDFS-13774: But as per the " -setPolicy" description for "HDFS Erasure Coding" it is specified in as {{"-replicate}} apply the special {{REPLICATION}} policy on the directory, force the directory to adopt 3x replication scheme." which is quite confusing as what is special REPLICATION policy. It needs to be modified as if we set policy as -replicate then the directory take the default Replication policy instead of any defined EC policy. > EC: "hdfs ec -getPolicy" is not retrieving policy details when the special > REPLICATION policy set on the directory > -- > > Key: HDFS-13774 > URL: https://issues.apache.org/jira/browse/HDFS-13774 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0 > Environment: 3 Node Linux Cluster >Reporter: Souryakanta Dwivedy >Assignee: Ayush Saxena >Priority: Minor > Attachments: GetPolicy_EC.png > > > Erasure coding: "hdfs ec -getPolicy"" is not retrieving policy details when > the special REPLICATION policy set on the directory > Steps :- > - Create a directory "testEC" > - Get the EC policy for the directory [Received message as : "The erasure > coding policy of /testEC is unspecified" ] > - Enable any Erasure coding policy like "XOR-2-1-1024k" > - Set the EC Policy on the Directory > - Get the EC policy for the directory [Received message as : "XOR-2-1-1024k" ] > - Now again set the EC Policy on the directory as "replicate" special > REPLICATION policy > - Get the EC policy for the directory [Received message as : "The erasure > coding policy of /testEC is unspecified" ] > The policy is being set for the Directory ,but while retrieving policy > details its throwing error as > policy for the directory is unspecified which is wrong behavior -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13774) EC: "hdfs ec -getPolicy" is not retrieving policy details when the special REPLICATION policy set on the directory
[ https://issues.apache.org/jira/browse/HDFS-13774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579586#comment-16579586 ] Ayush Saxena commented on HDFS-13774: - Thanks [~SouryakantaDwivedy] for putting up the issue. The reason for not returning the name of policy is that "Replication" is not an EC policy. It just acts like a flag to make the directory take the default Replication policy instead of an EC policy defined on its parent. Even if you use the ec -listPolicies command you won't see replication there. > EC: "hdfs ec -getPolicy" is not retrieving policy details when the special > REPLICATION policy set on the directory > -- > > Key: HDFS-13774 > URL: https://issues.apache.org/jira/browse/HDFS-13774 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0 > Environment: 3 Node Linux Cluster >Reporter: Souryakanta Dwivedy >Assignee: Ayush Saxena >Priority: Minor > Attachments: GetPolicy_EC.png > > > Erasure coding: "hdfs ec -getPolicy"" is not retrieving policy details when > the special REPLICATION policy set on the directory > Steps :- > - Create a directory "testEC" > - Get the EC policy for the directory [Received message as : "The erasure > coding policy of /testEC is unspecified" ] > - Enable any Erasure coding policy like "XOR-2-1-1024k" > - Set the EC Policy on the Directory > - Get the EC policy for the directory [Received message as : "XOR-2-1-1024k" ] > - Now again set the EC Policy on the directory as "replicate" special > REPLICATION policy > - Get the EC policy for the directory [Received message as : "The erasure > coding policy of /testEC is unspecified" ] > The policy is being set for the Directory ,but while retrieving policy > details its throwing error as > policy for the directory is unspecified which is wrong behavior -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org