[jira] [Commented] (HDFS-15412) Add options to set different block scan period for diffrent StorageType
[ https://issues.apache.org/jira/browse/HDFS-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17256862#comment-17256862 ] Ayush Saxena commented on HDFS-15412: - Well changes are looking fine, I would need to go through them once more though. {code:java} 317if (this.scanPeriodMs == 0) { 318 this.scanPeriodMs = conf.scanPeriodMs; 319} {code} can we make the check as <=, in order to handle mis configuration, and may Secondly : {code:java} 313this.scanPeriodMs = TimeUnit.MILLISECONDS.convert( 314StorageType.getConfLong(datanode.getConf(), 315ref.getVolume().getStorageType(), "scan.period.hours", 0), 316TimeUnit.HOURS); {code} Why are we passing 0 here as default, Why not {{conf.scanPeriodMs}}? > Add options to set different block scan period for diffrent StorageType > --- > > Key: HDFS-15412 > URL: https://issues.apache.org/jira/browse/HDFS-15412 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15412.001.patch, HDFS-15412.002.patch, > HDFS-15412.003.patch > > > For some cold data, sometime, we don't want to scan cold data as often as > hot data. Add options that we can set the scan period time according to > StorageType. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15756) [RBF] Cannot get updated delegation token from zookeeper
[ https://issues.apache.org/jira/browse/HDFS-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17256860#comment-17256860 ] Yuxuan Wang commented on HDFS-15756: I modify the code let router throw StandbyException, so the client can retry and failover to other routers. {code:title=RouterSecurityManager.java|java} renewDelegationToken(...){ ... catch (SecretManager.InvalidToken e) { throw new StandbyException(e.getMessage()); } } {code} You can have a try. > [RBF] Cannot get updated delegation token from zookeeper > > > Key: HDFS-15756 > URL: https://issues.apache.org/jira/browse/HDFS-15756 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.0.0 >Reporter: hbprotoss >Priority: Major > > Affected version: all version with rbf > When RBF work with spark 2.4 client mode, there will be a chance that token > is missing across different nodes in RBF cluster. The root cause is that > spark renew the token(via resource manager) immediately after got one, as > zookeeper don't have a strong consistency guarantee after an update in > cluster, zookeeper client may read a stale value in some followers not synced > with other nodes. > > We apply a patch in spark, but it is still the problem of RBF. Is it possible > for RBF to replace the delegation token store using some other > datasource(redis for example)? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15756) [RBF] Cannot get updated delegation token from zookeeper
[ https://issues.apache.org/jira/browse/HDFS-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HDFS-15756: Summary: [RBF] Cannot get updated delegation token from zookeeper (was: [RBF]Cannot get updated delegation token from zookeeper) > [RBF] Cannot get updated delegation token from zookeeper > > > Key: HDFS-15756 > URL: https://issues.apache.org/jira/browse/HDFS-15756 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.0.0 >Reporter: hbprotoss >Priority: Major > > Affected version: all version with rbf > When RBF work with spark 2.4 client mode, there will be a chance that token > is missing across different nodes in RBF cluster. The root cause is that > spark renew the token(via resource manager) immediately after got one, as > zookeeper don't have a strong consistency guarantee after an update in > cluster, zookeeper client may read a stale value in some followers not synced > with other nodes. > > We apply a patch in spark, but it is still the problem of RBF. Is it possible > for RBF to replace the delegation token store using some other > datasource(redis for example)? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15748) RBF: Move the router related part from hadoop-federation-balance module to hadoop-hdfs-rbf.
[ https://issues.apache.org/jira/browse/HDFS-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17256859#comment-17256859 ] Ayush Saxena commented on HDFS-15748: - {quote}I add a check in preCheck phase. If the dfs.permissions.enabled=false then throw an IOException {quote} This would not help, this should be true in the Namenode conf, no use of validating in the client conf, In UT it might work but not in actual prod. {quote}About the super user I don't know how to restrict him. Can we depend on admin's experience? {quote} I don't think there is a way to restrict a super-user, The best we can do is as of now mark a TODO and document it that in case of non RBF usage super-user shouldn't bother the directory. {code:java} +srcFs.setPermission(src, FsPermission.createImmutable((short) 0)); {code} Will this not disable read as well? Secondly while restoring we should ensure the actual permissions which were set are restored. TBH. This disableWrite() I don't think would be a very strong in disabling write, May be if possible, can we do something with snapshots here? Take a snapshot after disabling and before enabling confirm nothing changed, if changed we redo this logic or fail or something like that? > RBF: Move the router related part from hadoop-federation-balance module to > hadoop-hdfs-rbf. > --- > > Key: HDFS-15748 > URL: https://issues.apache.org/jira/browse/HDFS-15748 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15748.001.patch, HDFS-15748.002.patch, > HDFS-15748.003.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17256851#comment-17256851 ] Hadoop QA commented on HDFS-15755: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 36s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 58s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 7s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 6s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 20s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 35s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 4s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 21m 23s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 9s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 12s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m 15s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 28s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 37s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 6s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m 26s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 19m 26s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 20s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m 20s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 35s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/390/artifact/out/diff-checkstyle-root.txt{color} | {color:orange} root: The patch generated 1 new + 18 unchanged - 0 fixed = 19 total (was 18) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 58s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:g
[jira] [Updated] (HDFS-15756) [RBF]Cannot get updated delegation token from zookeeper
[ https://issues.apache.org/jira/browse/HDFS-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hbprotoss updated HDFS-15756: - Description: Affected version: all version with rbf When RBF work with spark 2.4 client mode, there will be a chance that token is missing across different nodes in RBF cluster. The root cause is that spark renew the token(via resource manager) immediately after got one, as zookeeper don't have a strong consistency guarantee after an update in cluster, zookeeper client may read a stale value in some followers not synced with other nodes. We apply a patch in spark, but it is still the problem of RBF. Is it possible for RBF to replace the delegation token store using some other datasource(redis for example)? was: When RBF work with spark 2.4 client mode, there will be a chance that token is missing across different nodes in RBF cluster. The root cause is that spark renew the token(via resource manager) immediately after got one, as zookeeper don't have a strong consistency guarantee after an update in cluster, zookeeper client may read a stale value in some followers not synced with other nodes. We apply a patch in spark, but it is still the problem of RBF. Is it possible for RBF to replace the delegation token store using some other datasource(redis for example)? > [RBF]Cannot get updated delegation token from zookeeper > --- > > Key: HDFS-15756 > URL: https://issues.apache.org/jira/browse/HDFS-15756 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.0.0 >Reporter: hbprotoss >Priority: Major > > Affected version: all version with rbf > When RBF work with spark 2.4 client mode, there will be a chance that token > is missing across different nodes in RBF cluster. The root cause is that > spark renew the token(via resource manager) immediately after got one, as > zookeeper don't have a strong consistency guarantee after an update in > cluster, zookeeper client may read a stale value in some followers not synced > with other nodes. > > We apply a patch in spark, but it is still the problem of RBF. Is it possible > for RBF to replace the delegation token store using some other > datasource(redis for example)? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15756) [RBF]Cannot get updated delegation token from zookeeper
hbprotoss created HDFS-15756: Summary: [RBF]Cannot get updated delegation token from zookeeper Key: HDFS-15756 URL: https://issues.apache.org/jira/browse/HDFS-15756 Project: Hadoop HDFS Issue Type: Bug Components: rbf Affects Versions: 3.0.0 Reporter: hbprotoss When RBF work with spark 2.4 client mode, there will be a chance that token is missing across different nodes in RBF cluster. The root cause is that spark renew the token(via resource manager) immediately after got one, as zookeeper don't have a strong consistency guarantee after an update in cluster, zookeeper client may read a stale value in some followers not synced with other nodes. We apply a patch in spark, but it is still the problem of RBF. Is it possible for RBF to replace the delegation token store using some other datasource(redis for example)? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15755: Status: Open (was: Patch Available) > Add option to set retry times of checking failure volume and Import powermock > for mock static method > > > Key: HDFS-15755 > URL: https://issues.apache.org/jira/browse/HDFS-15755 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15755.001.patch, HDFS-15755.002.patch, > HDFS-15755.003.patch > > > For some mounted remote volume, small network shaking may causes the failure > of volume checking and the the volume will removed out from datanode. Add an > option to set retry times and interval according to StorageType. > For mocking static methods in unit test, import PowerMock to > hadoop-hdfs-project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15755: Attachment: HDFS-15755.003.patch Status: Patch Available (was: Open) > Add option to set retry times of checking failure volume and Import powermock > for mock static method > > > Key: HDFS-15755 > URL: https://issues.apache.org/jira/browse/HDFS-15755 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15755.001.patch, HDFS-15755.002.patch, > HDFS-15755.003.patch > > > For some mounted remote volume, small network shaking may causes the failure > of volume checking and the the volume will removed out from datanode. Add an > option to set retry times and interval according to StorageType. > For mocking static methods in unit test, import PowerMock to > hadoop-hdfs-project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15412) Add options to set different block scan period for diffrent StorageType
[ https://issues.apache.org/jira/browse/HDFS-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17256763#comment-17256763 ] Hadoop QA commented on HDFS-15412: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 2s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 45s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 25m 59s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/389/artifact/out/branch-mvninstall-root.txt{color} | {color:red} root in trunk failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 28s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/389/artifact/out/branch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color} | {color:red} root in trunk failed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 31s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/389/artifact/out/branch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt{color} | {color:red} root in trunk failed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 26s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/389/artifact/out/buildtool-branch-checkstyle-root.txt{color} | {color:orange} The patch fails to run checkstyle in root {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 28s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/389/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt{color} | {color:red} hadoop-common in trunk failed. {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 31s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/389/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt{color} | {color:red} hadoop-hdfs in trunk failed. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 2m 9s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 1s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 26s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/389/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt{color} | {color:red} hadoop-hdfs in trunk failed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 6m 55s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 20s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/389/artifact/out/branch-findbugs-hadoop-common-project_hadoop-common.txt{color} | {color:red} hadoop-common in trunk failed. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 27s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/389/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt{color} | {color:red} hadoop-hdfs in trunk failed. {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 29s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {colo
[jira] [Updated] (HDFS-15412) Add options to set different block scan period for diffrent StorageType
[ https://issues.apache.org/jira/browse/HDFS-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15412: Status: Open (was: Patch Available) > Add options to set different block scan period for diffrent StorageType > --- > > Key: HDFS-15412 > URL: https://issues.apache.org/jira/browse/HDFS-15412 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15412.001.patch, HDFS-15412.002.patch, > HDFS-15412.003.patch > > > For some cold data, sometime, we don't want to scan cold data as often as > hot data. Add options that we can set the scan period time according to > StorageType. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15412) Add options to set different block scan period for diffrent StorageType
[ https://issues.apache.org/jira/browse/HDFS-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15412: Attachment: HDFS-15412.003.patch Status: Patch Available (was: Open) > Add options to set different block scan period for diffrent StorageType > --- > > Key: HDFS-15412 > URL: https://issues.apache.org/jira/browse/HDFS-15412 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15412.001.patch, HDFS-15412.002.patch, > HDFS-15412.003.patch > > > For some cold data, sometime, we don't want to scan cold data as often as > hot data. Add options that we can set the scan period time according to > StorageType. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15412) Add options to set different block scan period for diffrent StorageType
[ https://issues.apache.org/jira/browse/HDFS-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15412: Attachment: (was: HDFS-15412.003.patch) > Add options to set different block scan period for diffrent StorageType > --- > > Key: HDFS-15412 > URL: https://issues.apache.org/jira/browse/HDFS-15412 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15412.001.patch, HDFS-15412.002.patch, > HDFS-15412.003.patch > > > For some cold data, sometime, we don't want to scan cold data as often as > hot data. Add options that we can set the scan period time according to > StorageType. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15751) Add documentation for msync() API to filesystem.md
[ https://issues.apache.org/jira/browse/HDFS-15751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17256656#comment-17256656 ] Chao Sun commented on HDFS-15751: - I agree with [~hexiaoqiao] and think we can mention that this is currently only implemented for HDFS and others will just throw {{UnsupportedOperationException}}, similar to what we're doing for {{concat}}, {{truncate}} etc. Otherwise I'm +1 on this. Thanks. > Add documentation for msync() API to filesystem.md > -- > > Key: HDFS-15751 > URL: https://issues.apache.org/jira/browse/HDFS-15751 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Reporter: Konstantin Shvachko >Assignee: Konstantin Shvachko >Priority: Major > Attachments: HDFS-15751-01.patch > > > HDFS-15567 introduced new {{FileSystem}} call {{msync()}}. Should add it to > the API definitions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15412) Add options to set different block scan period for diffrent StorageType
[ https://issues.apache.org/jira/browse/HDFS-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17256644#comment-17256644 ] Hadoop QA commented on HDFS-15412: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m 29s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 14m 0s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m 12s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m 47s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 38s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 47s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 6s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 23m 38s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 8s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 28s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m 25s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 46s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 24s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 8s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 23m 11s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 23m 11s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 1s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m 1s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 54s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 8s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 19m 2s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifac
[jira] [Work logged] (HDFS-15754) Create packet metrics for DataNode
[ https://issues.apache.org/jira/browse/HDFS-15754?focusedWorklogId=529581&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-529581 ] ASF GitHub Bot logged work on HDFS-15754: - Author: ASF GitHub Bot Created on: 30/Dec/20 17:57 Start Date: 30/Dec/20 17:57 Worklog Time Spent: 10m Work Description: goiri commented on a change in pull request #2578: URL: https://github.com/apache/hadoop/pull/2578#discussion_r550278063 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java ## @@ -161,6 +163,65 @@ public void testReceivePacketMetrics() throws Exception { } } + @Test + public void testReceivePacketSlowMetrics() throws Exception { +Configuration conf = new HdfsConfiguration(); +final int interval = 1; +conf.set(DFSConfigKeys.DFS_METRICS_PERCENTILES_INTERVALS_KEY, "" + interval); Review comment: setInt This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 529581) Time Spent: 1h (was: 50m) > Create packet metrics for DataNode > -- > > Key: HDFS-15754 > URL: https://issues.apache.org/jira/browse/HDFS-15754 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Minor > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > In BlockReceiver, right now when there is slowness in writeToMirror, > writeToDisk and writeToOsCache, it is dumped in the debug log. In practice we > have found these are quite useful signal to detect issues in DataNode, so it > will be great these metrics can be exposed by JMX. > Also we introduced totalPacket received to use a percentage as a signal to > detect the potentially underperforming datanode since datanodes across one > HDFS cluster may received different numbers of packets totally. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15754) Create packet metrics for DataNode
[ https://issues.apache.org/jira/browse/HDFS-15754?focusedWorklogId=529582&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-529582 ] ASF GitHub Bot logged work on HDFS-15754: - Author: ASF GitHub Bot Created on: 30/Dec/20 17:57 Start Date: 30/Dec/20 17:57 Worklog Time Spent: 10m Work Description: goiri commented on a change in pull request #2578: URL: https://github.com/apache/hadoop/pull/2578#discussion_r550278246 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMetrics.java ## @@ -161,6 +163,65 @@ public void testReceivePacketMetrics() throws Exception { } } + @Test + public void testReceivePacketSlowMetrics() throws Exception { +Configuration conf = new HdfsConfiguration(); +final int interval = 1; +conf.set(DFSConfigKeys.DFS_METRICS_PERCENTILES_INTERVALS_KEY, "" + interval); +MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf) +.numDataNodes(3).build(); +try { + cluster.waitActive(); + DistributedFileSystem fs = cluster.getFileSystem(); + final DataNodeFaultInjector injector = + Mockito.mock(DataNodeFaultInjector.class); + Mockito.doAnswer(new Answer() { +@Override +public Object answer(InvocationOnMock invocationOnMock) Review comment: Extract the sleeping answer and return for each? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 529582) Time Spent: 1h 10m (was: 1h) > Create packet metrics for DataNode > -- > > Key: HDFS-15754 > URL: https://issues.apache.org/jira/browse/HDFS-15754 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > In BlockReceiver, right now when there is slowness in writeToMirror, > writeToDisk and writeToOsCache, it is dumped in the debug log. In practice we > have found these are quite useful signal to detect issues in DataNode, so it > will be great these metrics can be exposed by JMX. > Also we introduced totalPacket received to use a percentage as a signal to > detect the potentially underperforming datanode since datanodes across one > HDFS cluster may received different numbers of packets totally. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15754) Create packet metrics for DataNode
[ https://issues.apache.org/jira/browse/HDFS-15754?focusedWorklogId=529580&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-529580 ] ASF GitHub Bot logged work on HDFS-15754: - Author: ASF GitHub Bot Created on: 30/Dec/20 17:55 Start Date: 30/Dec/20 17:55 Worklog Time Spent: 10m Work Description: goiri commented on a change in pull request #2578: URL: https://github.com/apache/hadoop/pull/2578#discussion_r550277730 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java ## @@ -603,12 +604,15 @@ private int receivePacket() throws IOException { mirrorAddr, duration); trackSendPacketToLastNodeInPipeline(duration); -if (duration > datanodeSlowLogThresholdMs && LOG.isWarnEnabled()) { - LOG.warn("Slow BlockReceiver write packet to mirror took " + duration - + "ms (threshold=" + datanodeSlowLogThresholdMs + "ms), " - + "downstream DNs=" + Arrays.toString(downstreamDNs) - + ", blockId=" + replicaInfo.getBlockId() - + ", seqno=" + seqno); +if (duration > datanodeSlowLogThresholdMs) { + datanode.metrics.incrPacketSlowWriteToMirror(); + if (LOG.isWarnEnabled()) { +LOG.warn("Slow BlockReceiver write packet to mirror took " + duration ++ "ms (threshold=" + datanodeSlowLogThresholdMs + "ms), " ++ "downstream DNs=" + Arrays.toString(downstreamDNs) ++ ", blockId=" + replicaInfo.getBlockId() Review comment: As we are at it, let's use the logger format. We still need the Arrays toString so we need the isWarnEnabled though. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 529580) Time Spent: 50m (was: 40m) > Create packet metrics for DataNode > -- > > Key: HDFS-15754 > URL: https://issues.apache.org/jira/browse/HDFS-15754 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Minor > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > In BlockReceiver, right now when there is slowness in writeToMirror, > writeToDisk and writeToOsCache, it is dumped in the debug log. In practice we > have found these are quite useful signal to detect issues in DataNode, so it > will be great these metrics can be exposed by JMX. > Also we introduced totalPacket received to use a percentage as a signal to > detect the potentially underperforming datanode since datanodes across one > HDFS cluster may received different numbers of packets totally. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15748) RBF: Move the router related part from hadoop-federation-balance module to hadoop-hdfs-rbf.
[ https://issues.apache.org/jira/browse/HDFS-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17256632#comment-17256632 ] Íñigo Goiri commented on HDFS-15748: As we are at it, let's fix the javadocs and go with [^HDFS-15748.003.patch]. > RBF: Move the router related part from hadoop-federation-balance module to > hadoop-hdfs-rbf. > --- > > Key: HDFS-15748 > URL: https://issues.apache.org/jira/browse/HDFS-15748 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15748.001.patch, HDFS-15748.002.patch, > HDFS-15748.003.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17256630#comment-17256630 ] Hadoop QA commented on HDFS-15755: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 44s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 1s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 2s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 24s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m 34s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m 17s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 34s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 25s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 21m 46s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 31s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m 21s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 27s{color} | {color:blue}{color} | {color:blue} branch/hadoop-project no findbugs output file (findbugsXml.xml) {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 38s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 19s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 13m 21s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/387/artifact/out/patch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color} | {color:red} root in the patch failed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 13m 21s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/387/artifact/out/patch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color} | {color:red} root in the patch failed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 10m 49s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/387/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt{color} | {color:red} root in the patch failed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 49s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/387/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt{color} | {color:red} root in the patch failed with JDK Private Build-1.8.0_27
[jira] [Commented] (HDFS-15748) RBF: Move the router related part from hadoop-federation-balance module to hadoop-hdfs-rbf.
[ https://issues.apache.org/jira/browse/HDFS-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17256603#comment-17256603 ] Hadoop QA commented on HDFS-15748: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 39s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:blue}0{color} | {color:blue} markdownlint {color} | {color:blue} 0m 0s{color} | {color:blue}{color} | {color:blue} markdownlint was not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 31s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 24s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m 11s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m 29s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 36s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 14s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 29s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 40s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m 59s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 58s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 37s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 59s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 44s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 20m 44s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/386/artifact/out/diff-compile-javac-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color} | {color:red} root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 generated 139 new + 1902 unchanged - 139 fixed = 2041 total (was 2041) {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m 49s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 18m 49s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/386/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt{color} | {color:red} root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18
[jira] [Commented] (HDFS-15412) Add options to set different block scan period for diffrent StorageType
[ https://issues.apache.org/jira/browse/HDFS-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17256541#comment-17256541 ] Yang Yun commented on HDFS-15412: - retrigger a build for some tests faled with OOM. > Add options to set different block scan period for diffrent StorageType > --- > > Key: HDFS-15412 > URL: https://issues.apache.org/jira/browse/HDFS-15412 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15412.001.patch, HDFS-15412.002.patch, > HDFS-15412.003.patch > > > For some cold data, sometime, we don't want to scan cold data as often as > hot data. Add options that we can set the scan period time according to > StorageType. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15412) Add options to set different block scan period for diffrent StorageType
[ https://issues.apache.org/jira/browse/HDFS-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15412: Attachment: HDFS-15412.003.patch Status: Patch Available (was: Open) > Add options to set different block scan period for diffrent StorageType > --- > > Key: HDFS-15412 > URL: https://issues.apache.org/jira/browse/HDFS-15412 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15412.001.patch, HDFS-15412.002.patch, > HDFS-15412.003.patch > > > For some cold data, sometime, we don't want to scan cold data as often as > hot data. Add options that we can set the scan period time according to > StorageType. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15412) Add options to set different block scan period for diffrent StorageType
[ https://issues.apache.org/jira/browse/HDFS-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15412: Attachment: (was: HDFS-15412.003.patch) > Add options to set different block scan period for diffrent StorageType > --- > > Key: HDFS-15412 > URL: https://issues.apache.org/jira/browse/HDFS-15412 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15412.001.patch, HDFS-15412.002.patch, > HDFS-15412.003.patch > > > For some cold data, sometime, we don't want to scan cold data as often as > hot data. Add options that we can set the scan period time according to > StorageType. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15412) Add options to set different block scan period for diffrent StorageType
[ https://issues.apache.org/jira/browse/HDFS-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15412: Status: Open (was: Patch Available) > Add options to set different block scan period for diffrent StorageType > --- > > Key: HDFS-15412 > URL: https://issues.apache.org/jira/browse/HDFS-15412 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15412.001.patch, HDFS-15412.002.patch, > HDFS-15412.003.patch > > > For some cold data, sometime, we don't want to scan cold data as often as > hot data. Add options that we can set the scan period time according to > StorageType. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17256540#comment-17256540 ] Yang Yun commented on HDFS-15755: - Remove a file that should not be modified and trigger a new build. > Add option to set retry times of checking failure volume and Import powermock > for mock static method > > > Key: HDFS-15755 > URL: https://issues.apache.org/jira/browse/HDFS-15755 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15755.001.patch, HDFS-15755.002.patch > > > For some mounted remote volume, small network shaking may causes the failure > of volume checking and the the volume will removed out from datanode. Add an > option to set retry times and interval according to StorageType. > For mocking static methods in unit test, import PowerMock to > hadoop-hdfs-project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15755: Attachment: HDFS-15755.002.patch Status: Patch Available (was: Open) > Add option to set retry times of checking failure volume and Import powermock > for mock static method > > > Key: HDFS-15755 > URL: https://issues.apache.org/jira/browse/HDFS-15755 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15755.001.patch, HDFS-15755.002.patch > > > For some mounted remote volume, small network shaking may causes the failure > of volume checking and the the volume will removed out from datanode. Add an > option to set retry times and interval according to StorageType. > For mocking static methods in unit test, import PowerMock to > hadoop-hdfs-project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15755: Status: Open (was: Patch Available) > Add option to set retry times of checking failure volume and Import powermock > for mock static method > > > Key: HDFS-15755 > URL: https://issues.apache.org/jira/browse/HDFS-15755 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15755.001.patch, HDFS-15755.002.patch > > > For some mounted remote volume, small network shaking may causes the failure > of volume checking and the the volume will removed out from datanode. Add an > option to set retry times and interval according to StorageType. > For mocking static methods in unit test, import PowerMock to > hadoop-hdfs-project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15755: Attachment: (was: HDFS-15755.002.patch) > Add option to set retry times of checking failure volume and Import powermock > for mock static method > > > Key: HDFS-15755 > URL: https://issues.apache.org/jira/browse/HDFS-15755 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15755.001.patch, HDFS-15755.002.patch > > > For some mounted remote volume, small network shaking may causes the failure > of volume checking and the the volume will removed out from datanode. Add an > option to set retry times and interval according to StorageType. > For mocking static methods in unit test, import PowerMock to > hadoop-hdfs-project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15748) RBF: Move the router related part from hadoop-federation-balance module to hadoop-hdfs-rbf.
[ https://issues.apache.org/jira/browse/HDFS-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinglun updated HDFS-15748: --- Attachment: HDFS-15748.003.patch > RBF: Move the router related part from hadoop-federation-balance module to > hadoop-hdfs-rbf. > --- > > Key: HDFS-15748 > URL: https://issues.apache.org/jira/browse/HDFS-15748 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15748.001.patch, HDFS-15748.002.patch, > HDFS-15748.003.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15748) RBF: Move the router related part from hadoop-federation-balance module to hadoop-hdfs-rbf.
[ https://issues.apache.org/jira/browse/HDFS-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17256523#comment-17256523 ] Jinglun commented on HDFS-15748: Hi [~elgoiri] [~ayushtkn], thanks your comments ! {quote}Setting permissions to 0, If the dfs.persmissions is set to false, or the call is from super user, this won't help us? {quote} I add a check in preCheck phase. If the dfs.permissions.enabled=false then throw an IOException. About the super user I don't know how to restrict him. Can we depend on admin's experience?:) {quote}Do let me know if you changed some logic anywhere. {quote} Logic is not changed. {quote} this would need a test for RBF-Balance {quote} I have tested it in my cluster. I also checked the CLASSPATH when executing command rbfbalance. Upload v03. Fix checkstyle, check dfs.permissions.enabled and fix HDFSFederationBalance.md. > RBF: Move the router related part from hadoop-federation-balance module to > hadoop-hdfs-rbf. > --- > > Key: HDFS-15748 > URL: https://issues.apache.org/jira/browse/HDFS-15748 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15748.001.patch, HDFS-15748.002.patch, > HDFS-15748.003.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17256490#comment-17256490 ] Hadoop QA commented on HDFS-15755: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 56s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 59s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m 17s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m 30s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 23m 7s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 18s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 19s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 23m 6s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 26s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 26s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m 52s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 29s{color} | {color:blue}{color} | {color:blue} branch/hadoop-project no findbugs output file (findbugsXml.xml) {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 12s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/385/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-documentstore.txt{color} | {color:red} hadoop-yarn-server-timelineservice-documentstore in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 15s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/385/artifact/out/patch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color} | {color:red} root in the patch failed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 15s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/385/artifact/out/patch-compile-root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04.txt{color} | {color:red} root in the patch failed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 15s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/385/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01.txt{color} | {color:red} root in the patch failed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 15s{color} | {c
[jira] [Commented] (HDFS-15745) Make DataNodePeerMetrics#LOW_THRESHOLD_MS and MIN_OUTLIER_DETECTION_NODES configurable
[ https://issues.apache.org/jira/browse/HDFS-15745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17256481#comment-17256481 ] Ayush Saxena commented on HDFS-15745: - Thanx [~huanghaibin] for the patch. Had a quick look, Minor comments * Making configurable is Ok, but we should keep the default the value as is. * You need to add the new conf in hdfs-defaults * Checkstyle needs to be fixed. > Make DataNodePeerMetrics#LOW_THRESHOLD_MS and MIN_OUTLIER_DETECTION_NODES > configurable > -- > > Key: HDFS-15745 > URL: https://issues.apache.org/jira/browse/HDFS-15745 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haibin Huang >Assignee: Haibin Huang >Priority: Major > Attachments: HDFS-15745-001.patch, image-2020-12-22-17-00-50-796.png > > > When i enable DataNodePeerMetrics to find slow slow peer in cluster, i found > there is a lot of slow peer but ReportingNodes's averageDelay is very low, > and these slow peer node are normal. I think the reason of why generating so > many slow peer is that the value of DataNodePeerMetrics#LOW_THRESHOLD_MS is > too small (only 5ms) and it is not configurable. The default value of slow io > warning log threshold is 300ms, i.e. > DFSConfigKeys.DFS_DATANODE_SLOW_IO_WARNING_THRESHOLD_DEFAULT = 300, so > DataNodePeerMetrics#LOW_THRESHOLD_MS should not be less than 300ms, otherwise > namenode will get a lot of invalid slow peer information. > !image-2020-12-22-17-00-50-796.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15748) RBF: Move the router related part from hadoop-federation-balance module to hadoop-hdfs-rbf.
[ https://issues.apache.org/jira/browse/HDFS-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17256478#comment-17256478 ] Ayush Saxena commented on HDFS-15748: - Thanx [~LiJinglun] for the changes In this new one, I am not sure if your {{disableWrite}} would work, Setting permissions to 0, If the dfs.persmissions is set to false, or the call is from super user, this won't help us? Would need some tweak there, Though in case of RBF that is still sorted through readonly mount entry stuff. Other changes were looking quite direct. Do let me know if you changed some logic anywhere, though I didn't find anything bothering. But this would need a test for RBF-Balance I am not sure what is the best approach, The first one or now the new one. [~elgoiri] any preference here? > RBF: Move the router related part from hadoop-federation-balance module to > hadoop-hdfs-rbf. > --- > > Key: HDFS-15748 > URL: https://issues.apache.org/jira/browse/HDFS-15748 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15748.001.patch, HDFS-15748.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17256397#comment-17256397 ] Yang Yun commented on HDFS-15755: - Hi [~ayushtkn] I update the description add code(HDFS-15755.002.patch) with new changes, * Add options to set retry time and interval for volume checking according to StorageType. * Move the version to hadoop-project/pom.xml. the default verson of mockito-core is 2.28.2, Powermock-2.0.4 is compatible with it. > Add option to set retry times of checking failure volume and Import powermock > for mock static method > > > Key: HDFS-15755 > URL: https://issues.apache.org/jira/browse/HDFS-15755 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15755.001.patch, HDFS-15755.002.patch > > > For some mounted remote volume, small network shaking may causes the failure > of volume checking and the the volume will removed out from datanode. Add an > option to set retry times and interval according to StorageType. > For mocking static methods in unit test, import PowerMock to > hadoop-hdfs-project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15755: Attachment: HDFS-15755.002.patch Status: Patch Available (was: Open) > Add option to set retry times of checking failure volume and Import powermock > for mock static method > > > Key: HDFS-15755 > URL: https://issues.apache.org/jira/browse/HDFS-15755 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15755.001.patch, HDFS-15755.002.patch > > > For some mounted remote volume, small network shaking may causes the failure > of volume checking and the the volume will removed out from datanode. Add an > option to set retry times and interval according to StorageType. > For mocking static methods in unit test, import PowerMock to > hadoop-hdfs-project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15755: Status: Open (was: Patch Available) > Add option to set retry times of checking failure volume and Import powermock > for mock static method > > > Key: HDFS-15755 > URL: https://issues.apache.org/jira/browse/HDFS-15755 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15755.001.patch > > > For some mounted remote volume, small network shaking may causes the failure > of volume checking and the the volume will removed out from datanode. Add an > option to set retry times and interval according to StorageType. > For mocking static methods in unit test, import PowerMock to > hadoop-hdfs-project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15755: Description: For some mounted remote volume, small network shaking may causes the failure of volume checking and the the volume will removed out from datanode. Add an option to set retry times and interval according to StorageType. For mocking static methods in unit test, import PowerMock to hadoop-hdfs-project. was: For some mounted remote volume, small network shaking may causes the f For mocking static methods in unit test, import PowerMock to hadoop-hdfs-project. The same version PowerMock has been imported in part of hadoop-yarn-project. > Add option to set retry times of checking failure volume and Import powermock > for mock static method > > > Key: HDFS-15755 > URL: https://issues.apache.org/jira/browse/HDFS-15755 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15755.001.patch > > > For some mounted remote volume, small network shaking may causes the failure > of volume checking and the the volume will removed out from datanode. Add an > option to set retry times and interval according to StorageType. > For mocking static methods in unit test, import PowerMock to > hadoop-hdfs-project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15755: Description: For some mounted remote volume, small network shaking may causes the f For mocking static methods in unit test, import PowerMock to hadoop-hdfs-project. The same version PowerMock has been imported in part of hadoop-yarn-project. was: For mocking static methods in unit test, import PowerMock to hadoop-hdfs-project. The same version PowerMock has been imported in part of hadoop-yarn-project. > Add option to set retry times of checking failure volume and Import powermock > for mock static method > > > Key: HDFS-15755 > URL: https://issues.apache.org/jira/browse/HDFS-15755 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15755.001.patch > > > For some mounted remote volume, small network shaking may causes the f > For mocking static methods in unit test, import PowerMock to > hadoop-hdfs-project. > The same version PowerMock has been imported in part of hadoop-yarn-project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15755: Summary: Add option to set retry times of checking failure volume and Import powermock for mock static method (was: Import powermock to test of hadoop-hdfs-project) > Add option to set retry times of checking failure volume and Import powermock > for mock static method > > > Key: HDFS-15755 > URL: https://issues.apache.org/jira/browse/HDFS-15755 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15755.001.patch > > > For mocking static methods in unit test, import PowerMock to > hadoop-hdfs-project. > The same version PowerMock has been imported in part of hadoop-yarn-project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org