[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17379042#comment-17379042 ] liuyongpan commented on HDFS-14195: --- Currently, we can get the storage policy for directories or files using {code:java} hdfs oiv-sp-p Delimited{code} but there is a problem: For example, in the path {color:#FF}/A/B/C{color}, the storage policy of B is ARCHIVE, while the storage policy of C is unspecified. According to the current processing logic, the storage policy of C is 0, instead of getting the storage policy for the parent directory. Similar to the code processing logic in {code:java} hdfs storagePolicies -getStoragePolicy -path{code} {code:java} org.apache.hadoop.hdfs.server.namenode.INodeDirectory @Override public byte getStoragePolicyID() { byte id = getLocalStoragePolicyID(); if (id != BLOCK_STORAGE_POLICY_ID_UNSPECIFIED) { return id; } // if it is unspecified, check its parent return getParent() != null ? getParent().getStoragePolicyID() : BLOCK_STORAGE_POLICY_ID_UNSPECIFIED; } {code} > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Fix For: 3.3.0 > > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch, > HDFS-14195.009.patch, HDFS-14195.010.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904364#comment-16904364 ] Adam Antal commented on HDFS-14195: --- Thanks for the patch [~suxingfate], and [~jojochuang] for the commit! > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Fix For: 3.3.0 > > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch, > HDFS-14195.009.patch, HDFS-14195.010.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904234#comment-16904234 ] Hudson commented on HDFS-14195: --- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17079 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17079/]) HDFS-14195. OIV: print out storage policy id in oiv Delimited output. (weichiu: rev 865021b8c96ae96940ca060faae87452b433d970) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageTextWriter.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageDelimitedTextWriter.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testStoragePolicy.csv * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageViewerPB.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewerForStoragePolicy.java > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Fix For: 3.3.0 > > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch, > HDFS-14195.009.patch, HDFS-14195.010.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904223#comment-16904223 ] Wei-Chiu Chuang commented on HDFS-14195: +1 with the following small change just to make the text more clear: {code:java} + " -sp print storage policy, used by delimiter only.\n"{code} to {code:java} + " -sp print storage policy, used by delimited processor only.\n"{code} I'll commit the 010 patch with this change, and upload a 011 patch for future reference. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch, > HDFS-14195.009.patch, HDFS-14195.010.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903062#comment-16903062 ] Hadoop QA commented on HDFS-14195: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 46s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 4 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 50s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 58s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}108m 47s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 35s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}176m 26s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.datanode.TestLargeBlockReport | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 | | JIRA Issue | HDFS-14195 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12976978/HDFS-14195.010.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 8ed32fa53b1b 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 397a563 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_222 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/27443/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/27443/testReport/ | | Max. process+thread count | 3137 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/27443/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16902599#comment-16902599 ] Wang, Xinglong commented on HDFS-14195: --- HDFS-14195.010.patch to address checkstyle in https://builds.apache.org/job/PreCommit-HDFS-Build/27439/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch, > HDFS-14195.009.patch, HDFS-14195.010.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16902202#comment-16902202 ] Hadoop QA commented on HDFS-14195: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 38s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 4 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 2s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 24s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 35s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 136 unchanged - 0 fixed = 138 total (was 136) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 15s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 3s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 37s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}131m 35s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.diskbalancer.TestDiskBalancer | | | hadoop.hdfs.server.datanode.TestLargeBlockReport | | | hadoop.hdfs.server.balancer.TestBalancerRPCDelay | | | hadoop.hdfs.server.balancer.TestBalancer | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e | | JIRA Issue | HDFS-14195 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12976945/HDFS-14195.009.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 0d5cd04aa881 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 70f4674 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_212 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/27439/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/27439/artifact/out/patch-unit-hadoop-hdfs-projec
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16902090#comment-16902090 ] Wang, Xinglong commented on HDFS-14195: --- Thank you [~jojochuang] and [~adam.antal] for the comments. I think the comments should be addressed now. Submitting new patch. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch, > HDFS-14195.009.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900337#comment-16900337 ] Wei-Chiu Chuang commented on HDFS-14195: Thanks [~suxingfate] really appreciate your work! Other than the checkstyle warnings, a few nits: {code} + "-sp print storage policy.\n" {code} We should also state this is used by delimited output only. Storage policy is always exported for XML output. We should also update the doc. Can be a separate jira. {code} BufferedReader reader = new BufferedReader(new FileReader(file)); {code} the reader is not closed properly. This'll result in leaked open file descriptors. {code} FSDataOutputStream o = hdfs.create(file); o.write(123); o.close(); {code} You should ideally use try .. with () to ensure resource is not leaked upon failure. It is less critical for test code, but nice to have. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899917#comment-16899917 ] Adam Antal commented on HDFS-14195: --- Thanks for the patch, [~suxingfate]. It is structurally correct, nice job! One last item is some checkstyle issues: - Some long >80 lines and unused imports, please check [logs|https://builds.apache.org/job/PreCommit-HDFS-Build/27382/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt] on jenkins. - There are also a few missing whitespaces before and after the condition here: {noformat} if(writer.printStoragePolicy){ writer.append(buffer, storagePolicy); } {noformat} and here in {{PBImageDelimitedTextWriter.java}}: {noformat} if(printStoragePolicy){ append(buffer, "StoragePolicyId"); } {noformat} and in {{PBImageTextWriter.java}}: {noformat} for(XAttr xattr : xattrs){ if (BlockStoragePolicySuite.isStoragePolicyXAttr(xattr)) { return xattr.getValue()[0]; } } {noformat} Otherwise it's a +1 (non-binding) from me. Could you take a look as well [~jojochuang] to assure we didn't miss anything? > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899421#comment-16899421 ] Wang, Xinglong commented on HDFS-14195: --- [~jojochuang] [~adam.antal] The failed tests are not related. Could you help to merge the patch? > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899304#comment-16899304 ] Wei-Chiu Chuang commented on HDFS-14195: [~suxingfate] the latest precommit was run against the latest patch 008. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899298#comment-16899298 ] Wang, Xinglong commented on HDFS-14195: --- Failed TCs are not related with this patch. And How to trigger the test again on the same patch. I tried cancel patch and submit patch without uploading new patch file and seems can't trigger CI. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899057#comment-16899057 ] Hadoop QA commented on HDFS-14195: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 43s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 4 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 2s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 10s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 39s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 16 new + 136 unchanged - 0 fixed = 152 total (was 136) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 18s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 14s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 31s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}154m 2s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.tools.TestDFSZKFailoverController | | | hadoop.hdfs.server.datanode.TestLargeBlockReport | | | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery | | | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA | \\ \\ || Subsystem || Report/Notes || | Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:bdbca0e53b4 | | JIRA Issue | HDFS-14195 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12976540/HDFS-14195.008.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 107ba6c3d005 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 54ac801 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_212 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/27382/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/27382/artif
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898820#comment-16898820 ] Hadoop QA commented on HDFS-14195: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color} | {color:red} HDFS-14195 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | HDFS-14195 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12976519/HDFS-14195.007.patch | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/27381/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898744#comment-16898744 ] Wang, Xinglong commented on HDFS-14195: --- Sure, I will do the rebase. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898727#comment-16898727 ] Adam Antal commented on HDFS-14195: --- Hi [~suxingfate], Could you rebase the patch on HDFS-14203? Alternatively if you don't have the bandwidth I can take this over. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16864689#comment-16864689 ] Wei-Chiu Chuang commented on HDFS-14195: HDFS-14203 was committed and so this patch will need to be rebased. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851589#comment-16851589 ] Adam Antal commented on HDFS-14195: --- Maybe it makes sense to push HDFS-14203 in, and rebase this patch on top of that - which is probably makes it easier, and also adding the extra option to make it backward compatible. Do you agree [~jojochuang], [~suxingfate]? > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851310#comment-16851310 ] Wei-Chiu Chuang commented on HDFS-14195: +1 I think the patch looks good to me. I think it would be even better if it outputs the canonical name of the policy rather than a number (say ALL_SSD instead of 12) but it's easy to look up in the Hadoop doc so I'm okay with that. [~adam.antal] let's use HDFS-14203 to add the switch to enable storage policy id. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16834780#comment-16834780 ] Adam Antal commented on HDFS-14195: --- Thanks for sharing your concerns [~jojochuang]: I agree that it may cause backward compatibility. For future expandability it makes sense to configure this in a more flexible way, but for this issue I'd probably just stick to adding an option to the cli, something like {{--display-storage-policy}}. I also filed a jira earlier, where I suggested a refactor in that area: HDFS-14203. The addition of CLI options of displaying extra fields can be also handled there. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16834062#comment-16834062 ] Hadoop QA commented on HDFS-14195: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 26s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 6s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 16s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 57s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 29s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}105m 34s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 36s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}163m 44s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.namenode.ha.TestBootstrapAliasmap | | | hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality | | | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy | | | hadoop.hdfs.server.balancer.TestBalancer | | | hadoop.hdfs.web.TestWebHdfsFileSystemContract | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e | | JIRA Issue | HDFS-14195 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12967943/HDFS-14195.006.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 7e2a7adcdc8f 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 12b7059 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/26752/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/26752/testReport/ | | Max. process+thread count | 2841
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16834031#comment-16834031 ] Wei-Chiu Chuang commented on HDFS-14195: Thanks [~suxingfate] for reporting the issue and submitting the patch. I think the patch looks good overall. The only thing I am concerned is that the patch adds a new field in the delimited output, and there are production tools that parse the delimited oiv output. Additionally, in the future, it is likely we will need to support more fields and more xattr types. I would suggest to either add a command line option to turn this on only when the option is specified, or update the doc stating the delimited output may not be backward compatible, and provide the guideline for tools to adapt. For example, perhaps the tool should instead look at the header to figure out which column is the column for the data it looks for. Thoughts? > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16833899#comment-16833899 ] Wang, Xinglong commented on HDFS-14195: --- Updated the following part. * import style {code:java} java.io.* {code} * try () try ( ByteArrayInputStream input = new ByteArrayInputStream(output.toByteArray()); > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16833791#comment-16833791 ] Adam Antal commented on HDFS-14195: --- Hi, Thanks for the patch [~suxingfate]. Yes, thanks for the items having been addressed. Some cosmetics there: * {{java.io.*}} wildcard import is still there. It's matter of taste, but I prefer explicitly stating the used imports. * Also could you remove the extra line there: {code:java} try ( ByteArrayInputStream input = new ByteArrayInputStream(output.toByteArray()); {code} I'm +1 (non-binding) on committing this, thanks again for the work. [~jojochuang], could you please take a look and commit this, if you agree. Also [~smeng], if you have time for the review I'd be glad, since you have great knowledge around the OIV tool. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829924#comment-16829924 ] Wang, Xinglong commented on HDFS-14195: --- [~adam.antal] Your comments have been addressed. Could you please take a look?Thanks. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829628#comment-16829628 ] Hadoop QA commented on HDFS-14195: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 9s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 10s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 50s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 11s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 30s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}139m 11s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e | | JIRA Issue | HDFS-14195 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12967390/HDFS-14195.005.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux e02834a5374a 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb 13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / dbce729 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/26726/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/26726/testReport/ | | Max. process+thread count | 3442 (vs. ulimit of 1) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/26726/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829274#comment-16829274 ] Hadoop QA commented on HDFS-14195: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 13s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 38s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 13 new + 137 unchanged - 0 fixed = 150 total (was 137) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 27s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}101m 25s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 45s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}157m 58s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes | | | hadoop.hdfs.TestReconstructStripedFile | | | hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics | | | hadoop.hdfs.TestSafeModeWithStripedFileWithRandomECPolicy | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e | | JIRA Issue | HDFS-14195 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12967369/HDFS-14195.003.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 23c316d6f0a5 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / b434f55 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/26724/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/26724/artifact/
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16742009#comment-16742009 ] Adam Antal commented on HDFS-14195: --- Thanks for the patch [~suxingfate], good job. I have some general comment regarding your patch: Although as I see it too, the imports are not in the correct order, but I think it is more important to keep the git history clean, and not reorganizing the import. Also be aware not to use * imports (in {{TestOfflineImageViewerForStoragePolicy.java:36}} import java.io.*; ). Making a new test file {{TestOfflineImageViewerForStoragePolicy.java}} to test only the StoragePolicy field may be a bit too much, but since we do need to modify the fsimage and can't force it into the {{createOriginalFSImage()}} function of the {{TestOfflineImageViewer.java}}, I'm okay with it. There are a bunch of checkstyle issues, please take care of them. You can find the list here: [https://builds.apache.org/job/PreCommit-HDFS-Build/25971/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt] For too long lines you can break up strings. I had a suggestion that we can add a Builder pattern to the construction of these entries in {{PBImageDelimitedTextWriter.java}}. It's quite unrelated, I have filed a separate issue for that (HDFS-14203). Though comparing the file line by line can also point to the place of the problem, but maybe you can compare the whole file as well (storing the expected result in a csv and reading it in the testcase). If the compar fails, the diff is also logged out anyways. It can be something like in {{TestOfflineImageViewer$testCorruptionDetectionSingleFileCorruption}}. I observed some typos: {code:java} final static HashMap writtenStoragePolicys = Maps.newHashMap(); {code} where should be policies, and also: {code:java} Path dir = new Path("/dirWithStoragePolicyUnspecifed"); {code} and later in some variable names like {{subDirWithStoragePolicyUnspecifed}}: unspecified is missing an "i". Also these names can be abbreviated a bit, to variable names of length 15-20 maximum The FSImage generation is basically the same as in {{TestOfflineImageViewer$createOriginalFSImage}}, but the comment "We only want to generate the fsimage file once and use it for multiple tests." is not true here, since we only have one testcase. As I said earlier, we still do need to construct it somewhere, so it is ok, but the comment should be modify to not mislead the person reading the comment. DFSTestUtil has some StoragePolicy related function, that may or may not be useful (just pointing to it, I did not read dig into it). Also the DFSTestUtil.createFile might come handy for you. I wonder what other tests do to generate a file with some given StoragePolicy. As I saw DFSTestUtil does not include this kind of function, but you can wrap it into the testfile, if it makes it more compact: function which creates a file and gives a StoragePolicy by a parameter. I think the line {code:java} final String DELIMITER = "\t"; {code} can be move to class-level static variable (also the notice checkstyle warning). After this you can simply leave the commented part out. As I see this line {code:java} System.out.println(line); {code} is not needed in {{TestOfflineImageViewerForStoragePolicy$testPBDelimitedWriterForStoragePolicy:183}}. In {{TestOfflineImageViewer}} it is also that way, which was my bad. The addition of the field and the principle of the test is good though, so I can't add anything to it. After you take care of these items, the patch will be fine. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16741131#comment-16741131 ] Wang, Xinglong commented on HDFS-14195: --- Unrelated test failure and all tests passed in a local run. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16741129#comment-16741129 ] Hadoop QA commented on HDFS-14195: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 19s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 3s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 9s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 47s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 37 new + 41 unchanged - 4 fixed = 78 total (was 45) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 6s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 34s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 35s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}154m 48s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestRollingUpgrade | | | hadoop.hdfs.server.balancer.TestBalancer | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14195 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12954669/HDFS-14195.002.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 90b48ffdbb04 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 35fa3bd | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/25971/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/25971/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/25971/testReport/ | | Max. process+t
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740991#comment-16740991 ] Wang, Xinglong commented on HDFS-14195: --- Hi [~adam.antal]. I am not able to assign the ticket to myself. Could you please help to assign it to me? * reordered storage policy position in getEntry() * fixed 2 related test failure * Added TestOfflineImageViewerForStoragePolicy.java to test this field. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740217#comment-16740217 ] Adam Antal commented on HDFS-14195: --- Hi [~suxingfate]. Thanks for the issue and the patch. Don't forget to assign this issue to yourself. The patch is good, some minor things that I noticed: - In the function {{getEntry()}} of symlink type in {{PBImageDelimitedTextWriter.java}} the fields are not in the same order as in the file's and folder's case (storage policy, ns quota, ds quota). - Don't forget to address the test failures related to the OIV tool. The tests are now should expect the extra field. Also a new test case with some given storage policy, which is not the default (BLOCK_STORAGE_POLICY_ID_UNSPECIFIED) is recommended to test this field addition. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16739404#comment-16739404 ] Hadoop QA commented on HDFS-14195: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 38s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 31m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 48s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 39s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 20s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 16 unchanged - 0 fixed = 19 total (was 16) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 56s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}148m 43s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 35s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}236m 17s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl | | | hadoop.hdfs.TestReadStripedFileWithMissingBlocks | | | hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer | | | hadoop.hdfs.server.namenode.TestNameNodeMXBean | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure | | | hadoop.hdfs.qjournal.client.TestQuorumJournalManager | | | hadoop.hdfs.server.balancer.TestBalancerRPCDelay | | | hadoop.hdfs.server.datanode.TestDirectoryScanner | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | HDFS-14195 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12954406/HDFS-14195.001.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux ec279cff74bc 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/pe