subject:"\[jira\] \[Work logged\] \(HDFS\-16287\) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable"

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-11 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=680609&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-680609
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 12/Nov/21 02:55
Start Date: 12/Nov/21 02:55
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-966779378


   @ferhui @tomscut  Thanks your review and merge.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 680609)
Time Spent: 11.5h  (was: 11h 20m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 11.5h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-11 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=680588&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-680588
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 12/Nov/21 02:08
Start Date: 12/Nov/21 02:08
Worklog Time Spent: 10m 
  Work Description: ferhui commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-966759997


   @haiyang1987 Thanks for contribution. @tomscut Thanks for review!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 680588)
Time Spent: 11h 20m  (was: 11h 10m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 11h 20m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-11 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=680587&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-680587
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 12/Nov/21 02:07
Start Date: 12/Nov/21 02:07
Worklog Time Spent: 10m 
  Work Description: ferhui merged pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 680587)
Time Spent: 11h 10m  (was: 11h)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 11h 10m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-09 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=678944&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-678944
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 09/Nov/21 08:42
Start Date: 09/Nov/21 08:42
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r745395703



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   > Thanks @tomscut for your comment. 
[HDFS-15120](https://issues.apache.org/jira/browse/HDFS-15120) already support 
BlockPlacementPolicyDefault#dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled
 reconfigurable. But the current way I think is heavy. Do we need to support 
alone logic make excludeSlowNodesEnabled reconfigurable？
   
   Hi @tomscut take a deep look at code discovery HDFS-15120, if we don't 
update 'dfs.block.replicator.classname', 
'dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled' can't support 
reconfigurable. 
   so we need to support alone logic make excludeSlowNodesEnabled 
reconfigurable. 
   Consider create a new issue to solve this problem. 
   What do you think?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 678944)
Time Spent: 11h  (was: 10h 50m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 11h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-08 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=678424&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-678424
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 08/Nov/21 11:36
Start Date: 08/Nov/21 11:36
Worklog Time Spent: 10m 
  Work Description: ferhui commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r744641344



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -260,17 +257,14 @@
 final Timer timer = new Timer();
 this.slowPeerTracker = dataNodePeerStatsEnabled ?
 new SlowPeerTracker(conf, timer) : null;
-this.excludeSlowNodesEnabled = conf.getBoolean(

Review comment:
   It is used in BlockPlacementPolicyDefault




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 678424)
Time Spent: 10h 50m  (was: 10h 40m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10h 50m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-08 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=678408&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-678408
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 08/Nov/21 10:28
Start Date: 08/Nov/21 10:28
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-963013131


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 54s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 2 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  35m 46s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   1m 23s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |   1m 11s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  checkstyle  |   1m  2s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 21s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   1m  4s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   1m 24s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   3m 20s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  25m 19s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   1m 13s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 19s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javac  |   1m 19s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m  9s |  |  the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  javac  |   1m  9s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | +1 :green_heart: |  checkstyle  |   0m 53s |  |  the patch passed  |
   | +1 :green_heart: |  mvnsite  |   1m 14s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 49s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   1m 17s |  |  the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   3m 24s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  24m 54s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  | 330m 15s |  |  hadoop-hdfs in the patch 
passed.  |
   | +1 :green_heart: |  asflicense  |   0m 41s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 436m 59s |  |  |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3596/5/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/3596 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell |
   | uname | Linux 3388666e1f8e 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 87920fce0c2d28bf28fa2d0ff7589b142e0d3a31 |
   | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   | Multi-JDK versions | 
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 
/usr/lib/jvm/java-8-openjdk-amd64:Private 
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   |  Test Results | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3596/5/testReport/ |
   | Max. process+thread count | 2491 (vs. ulimit of 5500) |
   | modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
   | Console output | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3596/5/console |
   | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
   | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
   
   
   This

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-07 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=678334&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-678334
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 08/Nov/21 03:36
Start Date: 08/Nov/21 03:36
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r744385743



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeReconfigure.java
##
@@ -394,6 +395,23 @@ public void testEnableParallelLoadAfterReconfigured()
 assertEquals(true, FSImageFormatProtobuf.getEnableParallelLoad());
   }
 
+  @Test
+  public void testEnableSlowNodesParametersAfterReconfigured()
+  throws ReconfigurationException {
+final NameNode nameNode = cluster.getNameNode();
+final DatanodeManager datanodeManager = nameNode.namesystem
+.getBlockManager().getDatanodeManager();
+
+// By default, avoidSlowDataNodesForRead is false
+assertEquals(false, datanodeManager.getEnableAvoidSlowDataNodesForRead());
+
+nameNode.reconfigureProperty(
+DFS_NAMENODE_AVOID_SLOW_DATANODE_FOR_READ_KEY, Boolean.toString(true));
+
+// After reconfigured, avoidSlowDataNodesForRead is true

Review comment:
   Thanks @tomscut for your comment. PR update




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 678334)
Time Spent: 10.5h  (was: 10h 20m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10.5h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-05 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=678018&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-678018
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 06/Nov/21 03:51
Start Date: 06/Nov/21 03:51
Worklog Time Spent: 10m 
  Work Description: tomscut commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r744070877



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeReconfigure.java
##
@@ -394,6 +395,23 @@ public void testEnableParallelLoadAfterReconfigured()
 assertEquals(true, FSImageFormatProtobuf.getEnableParallelLoad());
   }
 
+  @Test
+  public void testEnableSlowNodesParametersAfterReconfigured()
+  throws ReconfigurationException {
+final NameNode nameNode = cluster.getNameNode();
+final DatanodeManager datanodeManager = nameNode.namesystem
+.getBlockManager().getDatanodeManager();
+
+// By default, avoidSlowDataNodesForRead is false

Review comment:
   According to the standard of previous review, a period should be added 
after the comment. 

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeReconfigure.java
##
@@ -394,6 +395,23 @@ public void testEnableParallelLoadAfterReconfigured()
 assertEquals(true, FSImageFormatProtobuf.getEnableParallelLoad());
   }
 
+  @Test
+  public void testEnableSlowNodesParametersAfterReconfigured()
+  throws ReconfigurationException {
+final NameNode nameNode = cluster.getNameNode();
+final DatanodeManager datanodeManager = nameNode.namesystem
+.getBlockManager().getDatanodeManager();
+
+// By default, avoidSlowDataNodesForRead is false
+assertEquals(false, datanodeManager.getEnableAvoidSlowDataNodesForRead());
+
+nameNode.reconfigureProperty(
+DFS_NAMENODE_AVOID_SLOW_DATANODE_FOR_READ_KEY, Boolean.toString(true));
+
+// After reconfigured, avoidSlowDataNodesForRead is true

Review comment:
Please add a period after the comment. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 678018)
Time Spent: 10h 20m  (was: 10h 10m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10h 20m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-05 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=677833&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-677833
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 05/Nov/21 20:42
Start Date: 05/Nov/21 20:42
Worklog Time Spent: 10m 
  Work Description: tomscut commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r743349892



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   > Thanks @tomscut for your comment. You have a point there.
   > 
   > But there is a problem，current parameter 'dataNodePeerStatsEnabled' and 
'excludeSlowNodesEnabled' decision SlowPeerCollector thread whether to start 
,But it didn't take into account avoid SlowDataNodesForRead logic.
   > 
   > And from the perspective of stability, the production environment of this 
function online can be consider divided into two stages： The first is to start 
SlowPeerCollector thread, we can check whether the reported datanode is a slow 
node based on monitoring indicators. Second, you can control whether to enable 
read/write avoid slow datanode according to dynamic parameters.
   > 
   > What do you think?
   
   Thanks @haiyang1987 for your explanation. It is OK from my side. Can we also 
make ```excludeSlowNodesEnabled``` reconfigurable?

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Thanks @haiyang1987 for your reminding, I didn't see the logic here 
before. I think we don't need to implement it separately if it's already 
supported.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 677833)
Time Spent: 10h 10m  (was: 10h)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10h 10m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-05 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=677612&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-677612
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 05/Nov/21 20:15
Start Date: 05/Nov/21 20:15
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r743380933



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Thanks @tomscut for your comment. 
   HDFS-15120 already support 
BlockPlacementPolicyDefault#dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled
 reconfigurable.  But the current way I think is heavy.  Do we need to support 
alone logic  make excludeSlowNodesEnabled reconfigurable？

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Thanks @tomscut for your comment.
   Is there anything else that needs to be improved in the current code?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 677612)
Time Spent: 10h  (was: 9h 50m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-05 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=677462&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-677462
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 05/Nov/21 19:56
Start Date: 05/Nov/21 19:56
Worklog Time Spent: 10m 
  Work Description: tomscut commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r743349892



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   > Thanks @tomscut for your comment. You have a point there.
   > 
   > But there is a problem，current parameter 'dataNodePeerStatsEnabled' and 
'excludeSlowNodesEnabled' decision SlowPeerCollector thread whether to start 
,But it didn't take into account avoid SlowDataNodesForRead logic.
   > 
   > And from the perspective of stability, the production environment of this 
function online can be consider divided into two stages： The first is to start 
SlowPeerCollector thread, we can check whether the reported datanode is a slow 
node based on monitoring indicators. Second, you can control whether to enable 
read/write avoid slow datanode according to dynamic parameters.
   > 
   > What do you think?
   
   Thanks @haiyang1987 for your explanation. It is OK from my side. Can we also 
make ```excludeSlowNodesEnabled``` reconfigurable?

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Thanks @haiyang1987 for your reminding, I didn't see the logic here 
before. I think we don't need to implement it separately if it's already 
supported.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 677462)
Time Spent: 9h 50m  (was: 9h 40m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 9h 50m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-05 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=677206&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-677206
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 05/Nov/21 19:22
Start Date: 05/Nov/21 19:22
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r743380933



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Thanks @tomscut for your comment. 
   HDFS-15120 already support 
BlockPlacementPolicyDefault#dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled
 reconfigurable.  But the current way I think is heavy.  Do we need to support 
alone logic  make excludeSlowNodesEnabled reconfigurable？

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Thanks @tomscut for your comment.
   Is there anything else that needs to be improved in the current code?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 677206)
Time Spent: 9h 40m  (was: 9.5h)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 9h 40m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-05 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=677014&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-677014
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 05/Nov/21 12:29
Start Date: 05/Nov/21 12:29
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r743620039



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Thanks @tomscut for your comment.
   Is there anything else that needs to be improved in the current code?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 677014)
Time Spent: 9.5h  (was: 9h 20m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 9.5h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-04 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=676889&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-676889
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 05/Nov/21 05:00
Start Date: 05/Nov/21 05:00
Worklog Time Spent: 10m 
  Work Description: tomscut commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r743391423



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Thanks @haiyang1987 for your reminding, I didn't see the logic here 
before. I think we don't need to implement it separately if it's already 
supported.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 676889)
Time Spent: 9h 20m  (was: 9h 10m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 9h 20m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-04 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=676876&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-676876
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 05/Nov/21 04:18
Start Date: 05/Nov/21 04:18
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r743380933



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Thanks @tomscut for your comment. 
   HDFS-15120 already support 
BlockPlacementPolicyDefault#dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled
 reconfigurable.  But the current way I think is heavy.  Do we need to support 
alone logic  make excludeSlowNodesEnabled reconfigurable？




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 676876)
Time Spent: 9h 10m  (was: 9h)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 9h 10m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-04 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=676848&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-676848
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 05/Nov/21 02:25
Start Date: 05/Nov/21 02:25
Worklog Time Spent: 10m 
  Work Description: tomscut commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r743349892



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   > Thanks @tomscut for your comment. You have a point there.
   > 
   > But there is a problem，current parameter 'dataNodePeerStatsEnabled' and 
'excludeSlowNodesEnabled' decision SlowPeerCollector thread whether to start 
,But it didn't take into account avoid SlowDataNodesForRead logic.
   > 
   > And from the perspective of stability, the production environment of this 
function online can be consider divided into two stages： The first is to start 
SlowPeerCollector thread, we can check whether the reported datanode is a slow 
node based on monitoring indicators. Second, you can control whether to enable 
read/write avoid slow datanode according to dynamic parameters.
   > 
   > What do you think?
   
   Thanks @haiyang1987 for your explanation. It is OK from my side. Can we also 
make ```excludeSlowNodesEnabled``` reconfigurable?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 676848)
Time Spent: 9h  (was: 8h 50m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 9h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-04 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=676366&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-676366
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 04/Nov/21 12:09
Start Date: 04/Nov/21 12:09
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r742775193



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Thanks @tomscut  for your comment. You have a point there.
   
   But there is a problem，current parameter 'dataNodePeerStatsEnabled' and 
'excludeSlowNodesEnabled' decision SlowPeerCollector thread whether to start 
,But it didn't take into account avoid SlowDataNodesForRead logic.
   
   And from the perspective of stability, the production environment of this 
function online can be consider divided into two stages：
   The first is to start SlowPeerCollector thread, we can check whether the 
reported datanode is a slow node based on monitoring indicators.
   Second, you can control whether to enable read/write avoid slow datanode 
according to dynamic parameters.
   
   What do you think?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 676366)
Time Spent: 8h 50m  (was: 8h 40m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 8h 50m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=676066&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-676066
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 04/Nov/21 01:55
Start Date: 04/Nov/21 01:55
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-958731868






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 676066)
Time Spent: 8h 40m  (was: 8.5h)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 8h 40m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=676035&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-676035
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 04/Nov/21 01:52
Start Date: 04/Nov/21 01:52
Worklog Time Spent: 10m 
  Work Description: tomscut commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r742015162



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Thanks @haiyang1987 for your comment. I think the logic right now is no 
problem. 
   
   I mean that when ```excludeSlowNodesEnabled``` is set to true we 
```startSlowPeerCollector```, and ```stopSlowPeerCollector``` when 
```excludeSlowNodesEnabled``` is set to false. There is no extra overhead. What 
do you think?

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Thanks @haiyang1987 for your comment. I think the logic right now is no 
problem. 
   
   I mean that only when ```excludeSlowNodesEnabled``` is set to true we 
```startSlowPeerCollector```, and ```stopSlowPeerCollector``` when 
```excludeSlowNodesEnabled``` is set to false. There is no extra overhead. What 
do you think?

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Thanks @haiyang1987 for your comment. I think the logic right now is no 
problem. 
   
   I mean that only when `excludeSlowNodesEnabled` is set to `true` we 
`startSlowPeerCollector`, and `stopSlowPeerCollector` when 
`excludeSlowNodesEnabled` is set to `false`. There is no extra overhead. What 
do you think?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 676035)
Time Spent: 8.5h  (was: 8h 20m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 8.5h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=676031&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-676031
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 04/Nov/21 01:52
Start Date: 04/Nov/21 01:52
Worklog Time Spent: 10m 
  Work Description: haiyang1987 opened a new pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596


   ### Description of PR
   
   Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
   Details: HDFS-16287
   
   ### For code changes:
   
   - [ ] Consider that make dfs.namenode.avoid.read.slow.datanode 
reconfigurable and rapid rollback in case this feature HDFS-16076  unexpected 
things happen in production environment
   - [ ] DatanodeManager#startSlowPeerCollector  by parameter 
'dfs.datanode.peer.stats.enabled' to control
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 676031)
Time Spent: 8h 20m  (was: 8h 10m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 8h 20m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=676020&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-676020
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 04/Nov/21 01:51
Start Date: 04/Nov/21 01:51
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r741696525



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -260,17 +257,14 @@
 final Timer timer = new Timer();
 this.slowPeerTracker = dataNodePeerStatsEnabled ?
 new SlowPeerTracker(conf, timer) : null;
-this.excludeSlowNodesEnabled = conf.getBoolean(
-DFS_NAMENODE_BLOCKPLACEMENTPOLICY_EXCLUDE_SLOW_NODES_ENABLED_KEY,
-DFS_NAMENODE_BLOCKPLACEMENTPOLICY_EXCLUDE_SLOW_NODES_ENABLED_DEFAULT);
 this.maxSlowPeerReportNodes = conf.getInt(
 DFSConfigKeys.DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY,
 DFSConfigKeys.DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_DEFAULT);
 this.slowPeerCollectionInterval = conf.getTimeDuration(
 DFSConfigKeys.DFS_NAMENODE_SLOWPEER_COLLECT_INTERVAL_KEY,
 DFSConfigKeys.DFS_NAMENODE_SLOWPEER_COLLECT_INTERVAL_DEFAULT,
 TimeUnit.MILLISECONDS);
-if (slowPeerTracker != null && excludeSlowNodesEnabled) {

Review comment:
   @tomscut Thank you for your review.
   1.Current parameter 'dataNodePeerStatsEnabled' and 'excludeSlowNodesEnabled' 
decision SlowPeerCollector thread whether to start ,But it didn't take into 
account avoid SlowDataNodesForRead logic 
   2.So think about two phases:
   a.The first is to start SlowPeerCollector thread
   b.Second, you can control whether to enable read/write avoid slow datanode 
according to dynamic parameters

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Consider slowNodesUuidSet is generated when the SlowPeerCollector thread 
is started，therefore  it is logical to judge Therefore, it is logical to judge 
whether the dnUuid exists in the slowNodesUuidSet?

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Consider slowNodesUuidSet is generated when the SlowPeerCollector thread 
is started，therefore  it is logical to judge whether the dnUuid exists in the 
slowNodesUuidSet?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 676020)
Time Spent: 8h 10m  (was: 8h)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 8h 10m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=675985&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-675985
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 04/Nov/21 01:47
Start Date: 04/Nov/21 01:47
Worklog Time Spent: 10m 
  Work Description: haiyang1987 closed pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 675985)
Time Spent: 8h  (was: 7h 50m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 8h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=675857&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-675857
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 04/Nov/21 01:35
Start Date: 04/Nov/21 01:35
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-959651079


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   1m 24s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 2 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  36m 58s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   1m 35s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |   1m 22s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  checkstyle  |   1m  4s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 33s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   1m  3s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   1m 28s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   3m 41s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  25m 39s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   1m 27s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 28s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javac  |   1m 28s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 18s |  |  the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  javac  |   1m 18s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | +1 :green_heart: |  checkstyle  |   0m 57s |  |  the patch passed  |
   | +1 :green_heart: |  mvnsite  |   1m 25s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 54s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   1m 24s |  |  the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   3m 44s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  25m 53s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | -1 :x: |  unit  | 374m 11s | 
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3596/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
 |  hadoop-hdfs in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 43s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 486m  5s |  |  |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.hdfs.TestHDFSFileSystemContract |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3596/4/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/3596 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell |
   | uname | Linux 1fb2ee0e949b 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 2ec82e1c420789afb326f4ebb451522a8a4e2358 |
   | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   | Multi-JDK versions | 
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 
/usr/lib/jvm/java-8-openjdk-amd64:Private 
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   |  Test Results | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3596/4/testReport/ |
   | Max. process+thread count | 2022 (vs. ulimit of 5500) |
   | modules | C: hadoop-hdfs-project/hadoop-hdf

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=675702&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-675702
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 04/Nov/21 01:19
Start Date: 04/Nov/21 01:19
Worklog Time Spent: 10m 
  Work Description: haiyang1987 removed a comment on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-958731868






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 675702)
Time Spent: 7h 40m  (was: 7.5h)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 7h 40m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=675657&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-675657
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 04/Nov/21 01:13
Start Date: 04/Nov/21 01:13
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-958731868






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 675657)
Time Spent: 7.5h  (was: 7h 20m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 7.5h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=675628&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-675628
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 04/Nov/21 01:10
Start Date: 04/Nov/21 01:10
Worklog Time Spent: 10m 
  Work Description: haiyang1987 opened a new pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596


   ### Description of PR
   
   Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
   Details: HDFS-16287
   
   ### For code changes:
   
   - [ ] Consider that make dfs.namenode.avoid.read.slow.datanode 
reconfigurable and rapid rollback in case this feature HDFS-16076  unexpected 
things happen in production environment
   - [ ] DatanodeManager#startSlowPeerCollector  by parameter 
'dfs.datanode.peer.stats.enabled' to control
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 675628)
Time Spent: 7h 20m  (was: 7h 10m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 7h 20m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=675615&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-675615
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 04/Nov/21 01:08
Start Date: 04/Nov/21 01:08
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r741696525



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -260,17 +257,14 @@
 final Timer timer = new Timer();
 this.slowPeerTracker = dataNodePeerStatsEnabled ?
 new SlowPeerTracker(conf, timer) : null;
-this.excludeSlowNodesEnabled = conf.getBoolean(
-DFS_NAMENODE_BLOCKPLACEMENTPOLICY_EXCLUDE_SLOW_NODES_ENABLED_KEY,
-DFS_NAMENODE_BLOCKPLACEMENTPOLICY_EXCLUDE_SLOW_NODES_ENABLED_DEFAULT);
 this.maxSlowPeerReportNodes = conf.getInt(
 DFSConfigKeys.DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY,
 DFSConfigKeys.DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_DEFAULT);
 this.slowPeerCollectionInterval = conf.getTimeDuration(
 DFSConfigKeys.DFS_NAMENODE_SLOWPEER_COLLECT_INTERVAL_KEY,
 DFSConfigKeys.DFS_NAMENODE_SLOWPEER_COLLECT_INTERVAL_DEFAULT,
 TimeUnit.MILLISECONDS);
-if (slowPeerTracker != null && excludeSlowNodesEnabled) {

Review comment:
   @tomscut Thank you for your review.
   1.Current parameter 'dataNodePeerStatsEnabled' and 'excludeSlowNodesEnabled' 
decision SlowPeerCollector thread whether to start ,But it didn't take into 
account avoid SlowDataNodesForRead logic 
   2.So think about two phases:
   a.The first is to start SlowPeerCollector thread
   b.Second, you can control whether to enable read/write avoid slow datanode 
according to dynamic parameters

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Consider slowNodesUuidSet is generated when the SlowPeerCollector thread 
is started，therefore  it is logical to judge Therefore, it is logical to judge 
whether the dnUuid exists in the slowNodesUuidSet?

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Consider slowNodesUuidSet is generated when the SlowPeerCollector thread 
is started，therefore  it is logical to judge whether the dnUuid exists in the 
slowNodesUuidSet?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 675615)
Time Spent: 7h 10m  (was: 7h)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 7h 10m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=675584&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-675584
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 04/Nov/21 01:04
Start Date: 04/Nov/21 01:04
Worklog Time Spent: 10m 
  Work Description: haiyang1987 closed pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 675584)
Time Spent: 7h  (was: 6h 50m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 7h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=675398&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-675398
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 04/Nov/21 00:47
Start Date: 04/Nov/21 00:47
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-959651079






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 675398)
Time Spent: 6h 50m  (was: 6h 40m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=675371&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-675371
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 04/Nov/21 00:44
Start Date: 04/Nov/21 00:44
Worklog Time Spent: 10m 
  Work Description: tomscut commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r742015162



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Thanks @haiyang1987 for your comment. I think the logic right now is no 
problem. 
   
   I mean that when ```excludeSlowNodesEnabled``` is set to true we 
```startSlowPeerCollector```, and ```stopSlowPeerCollector``` when 
```excludeSlowNodesEnabled``` is set to false. There is no extra overhead. What 
do you think?

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Thanks @haiyang1987 for your comment. I think the logic right now is no 
problem. 
   
   I mean that only when ```excludeSlowNodesEnabled``` is set to true we 
```startSlowPeerCollector```, and ```stopSlowPeerCollector``` when 
```excludeSlowNodesEnabled``` is set to false. There is no extra overhead. What 
do you think?

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Thanks @haiyang1987 for your comment. I think the logic right now is no 
problem. 
   
   I mean that only when `excludeSlowNodesEnabled` is set to `true` we 
`startSlowPeerCollector`, and `stopSlowPeerCollector` when 
`excludeSlowNodesEnabled` is set to `false`. There is no extra overhead. What 
do you think?

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Thanks @haiyang1987 for your comment. I think the logic right now is no 
problem. 
   
   I mean that when ```excludeSlowNodesEnabled``` is set to true we 
```startSlowPeerCollector```, and ```stopSlowPeerCollector``` when 
```excludeSlowNodesEnabled``` is set to false. There is no extra overhead. What 
do you think?

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Thanks @haiyang1987 for your comment. I think the logic right now is no 
problem. 
   
   I mean that only when ```excludeSlowNodesEnabled``` is set to true we 
```startSlowPeerCollector```, and ```stopSlowPeerCollector``` when 
```excludeSlowNodesEnabled``` is set to false. There is no extra overhead. What 
do you think?

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Thanks @haiyang1987 for your comment. I think the logic right now is no 
problem. 
   
   I mean that only when `excludeSlowNodesEnabled` is set to `true` we

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=675197&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-675197
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 04/Nov/21 00:23
Start Date: 04/Nov/21 00:23
Worklog Time Spent: 10m 
  Work Description: haiyang1987 removed a comment on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-958731868


   > @haiyang1987 Thanks for contribution, some comments: we can change the 
title here and jira, If 
dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled is not 
reconfigurable. And I will check whether 
dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled is unused.
   
   
   
   > @haiyang1987 Thanks for contribution, some comments: we can change the 
title here and jira, If 
dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled is not 
reconfigurable. And I will check whether 
dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled is unused.
   
   @ferhui Thank you for your review. 
   1.Already change the title here and jira 
   2. 'dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled' in 
BlockPlacementPolicyDefault use


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 675197)
Time Spent: 6.5h  (was: 6h 20m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 6.5h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=675149&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-675149
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 04/Nov/21 00:16
Start Date: 04/Nov/21 00:16
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-958731868






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 675149)
Time Spent: 6h 20m  (was: 6h 10m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 6h 20m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=675120&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-675120
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 04/Nov/21 00:14
Start Date: 04/Nov/21 00:14
Worklog Time Spent: 10m 
  Work Description: haiyang1987 opened a new pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596


   ### Description of PR
   
   Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
   Details: HDFS-16287
   
   ### For code changes:
   
   - [ ] Consider that make dfs.namenode.avoid.read.slow.datanode 
reconfigurable and rapid rollback in case this feature HDFS-16076  unexpected 
things happen in production environment
   - [ ] DatanodeManager#startSlowPeerCollector  by parameter 
'dfs.datanode.peer.stats.enabled' to control
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 675120)
Time Spent: 6h 10m  (was: 6h)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=675106&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-675106
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 04/Nov/21 00:12
Start Date: 04/Nov/21 00:12
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r741696525



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -260,17 +257,14 @@
 final Timer timer = new Timer();
 this.slowPeerTracker = dataNodePeerStatsEnabled ?
 new SlowPeerTracker(conf, timer) : null;
-this.excludeSlowNodesEnabled = conf.getBoolean(
-DFS_NAMENODE_BLOCKPLACEMENTPOLICY_EXCLUDE_SLOW_NODES_ENABLED_KEY,
-DFS_NAMENODE_BLOCKPLACEMENTPOLICY_EXCLUDE_SLOW_NODES_ENABLED_DEFAULT);
 this.maxSlowPeerReportNodes = conf.getInt(
 DFSConfigKeys.DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY,
 DFSConfigKeys.DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_DEFAULT);
 this.slowPeerCollectionInterval = conf.getTimeDuration(
 DFSConfigKeys.DFS_NAMENODE_SLOWPEER_COLLECT_INTERVAL_KEY,
 DFSConfigKeys.DFS_NAMENODE_SLOWPEER_COLLECT_INTERVAL_DEFAULT,
 TimeUnit.MILLISECONDS);
-if (slowPeerTracker != null && excludeSlowNodesEnabled) {

Review comment:
   @tomscut Thank you for your review.
   1.Current parameter 'dataNodePeerStatsEnabled' and 'excludeSlowNodesEnabled' 
decision SlowPeerCollector thread whether to start ,But it didn't take into 
account avoid SlowDataNodesForRead logic 
   2.So think about two phases:
   a.The first is to start SlowPeerCollector thread
   b.Second, you can control whether to enable read/write avoid slow datanode 
according to dynamic parameters

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Consider slowNodesUuidSet is generated when the SlowPeerCollector thread 
is started，therefore  it is logical to judge Therefore, it is logical to judge 
whether the dnUuid exists in the slowNodesUuidSet?

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Consider slowNodesUuidSet is generated when the SlowPeerCollector thread 
is started，therefore  it is logical to judge whether the dnUuid exists in the 
slowNodesUuidSet?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 675106)
Time Spent: 6h  (was: 5h 50m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 6h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=675071&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-675071
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 04/Nov/21 00:07
Start Date: 04/Nov/21 00:07
Worklog Time Spent: 10m 
  Work Description: haiyang1987 closed pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 675071)
Time Spent: 5h 50m  (was: 5h 40m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 5h 50m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=674736&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-674736
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 03/Nov/21 16:28
Start Date: 03/Nov/21 16:28
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-959651079


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   1m 24s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 2 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  36m 58s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   1m 35s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |   1m 22s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  checkstyle  |   1m  4s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 33s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   1m  3s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   1m 28s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   3m 41s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  25m 39s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   1m 27s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 28s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javac  |   1m 28s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 18s |  |  the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  javac  |   1m 18s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | +1 :green_heart: |  checkstyle  |   0m 57s |  |  the patch passed  |
   | +1 :green_heart: |  mvnsite  |   1m 25s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 54s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   1m 24s |  |  the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   3m 44s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  25m 53s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | -1 :x: |  unit  | 374m 11s | 
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3596/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
 |  hadoop-hdfs in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 43s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 486m  5s |  |  |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.hdfs.TestHDFSFileSystemContract |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3596/4/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/3596 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell |
   | uname | Linux 1fb2ee0e949b 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 2ec82e1c420789afb326f4ebb451522a8a4e2358 |
   | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   | Multi-JDK versions | 
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 
/usr/lib/jvm/java-8-openjdk-amd64:Private 
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   |  Test Results | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3596/4/testReport/ |
   | Max. process+thread count | 2022 (vs. ulimit of 5500) |
   | modules | C: hadoop-hdfs-project/hadoop-hdf

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=674666&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-674666
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 03/Nov/21 14:47
Start Date: 03/Nov/21 14:47
Worklog Time Spent: 10m 
  Work Description: tomscut commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r742015162



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Thanks @haiyang1987 for your comment. I think the logic right now is no 
problem. 
   
   I mean that only when `excludeSlowNodesEnabled` is set to `true` we 
`startSlowPeerCollector`, and `stopSlowPeerCollector` when 
`excludeSlowNodesEnabled` is set to `false`. There is no extra overhead. What 
do you think?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 674666)
Time Spent: 5.5h  (was: 5h 20m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=674665&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-674665
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 03/Nov/21 14:46
Start Date: 03/Nov/21 14:46
Worklog Time Spent: 10m 
  Work Description: tomscut commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r742015162



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Thanks @haiyang1987 for your comment. I think the logic right now is no 
problem. 
   
   I mean that only when ```excludeSlowNodesEnabled``` is set to true we 
```startSlowPeerCollector```, and ```stopSlowPeerCollector``` when 
```excludeSlowNodesEnabled``` is set to false. There is no extra overhead. What 
do you think?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 674665)
Time Spent: 5h 20m  (was: 5h 10m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=674664&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-674664
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 03/Nov/21 14:45
Start Date: 03/Nov/21 14:45
Worklog Time Spent: 10m 
  Work Description: tomscut commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r742015162



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Thanks @haiyang1987 for your comment. I think the logic right now is no 
problem. 
   
   I mean that when ```excludeSlowNodesEnabled``` is set to true we 
```startSlowPeerCollector```, and ```stopSlowPeerCollector``` when 
```excludeSlowNodesEnabled``` is set to false. There is no extra overhead. What 
do you think?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 674664)
Time Spent: 5h 10m  (was: 5h)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=674448&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-674448
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 03/Nov/21 08:40
Start Date: 03/Nov/21 08:40
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r741707936



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Consider slowNodesUuidSet is generated when the SlowPeerCollector thread 
is started，therefore  it is logical to judge whether the dnUuid exists in the 
slowNodesUuidSet?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 674448)
Time Spent: 5h  (was: 4h 50m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=674447&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-674447
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 03/Nov/21 08:40
Start Date: 03/Nov/21 08:40
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r741707936



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   Consider slowNodesUuidSet is generated when the SlowPeerCollector thread 
is started，therefore  it is logical to judge Therefore, it is logical to judge 
whether the dnUuid exists in the slowNodesUuidSet?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 674447)
Time Spent: 4h 50m  (was: 4h 40m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=674440&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-674440
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 03/Nov/21 08:22
Start Date: 03/Nov/21 08:22
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r741696525



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -260,17 +257,14 @@
 final Timer timer = new Timer();
 this.slowPeerTracker = dataNodePeerStatsEnabled ?
 new SlowPeerTracker(conf, timer) : null;
-this.excludeSlowNodesEnabled = conf.getBoolean(
-DFS_NAMENODE_BLOCKPLACEMENTPOLICY_EXCLUDE_SLOW_NODES_ENABLED_KEY,
-DFS_NAMENODE_BLOCKPLACEMENTPOLICY_EXCLUDE_SLOW_NODES_ENABLED_DEFAULT);
 this.maxSlowPeerReportNodes = conf.getInt(
 DFSConfigKeys.DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY,
 DFSConfigKeys.DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_DEFAULT);
 this.slowPeerCollectionInterval = conf.getTimeDuration(
 DFSConfigKeys.DFS_NAMENODE_SLOWPEER_COLLECT_INTERVAL_KEY,
 DFSConfigKeys.DFS_NAMENODE_SLOWPEER_COLLECT_INTERVAL_DEFAULT,
 TimeUnit.MILLISECONDS);
-if (slowPeerTracker != null && excludeSlowNodesEnabled) {

Review comment:
   @tomscut Thank you for your review.
   1.Current parameter 'dataNodePeerStatsEnabled' and 'excludeSlowNodesEnabled' 
decision SlowPeerCollector thread whether to start ,But it didn't take into 
account avoid SlowDataNodesForRead logic 
   2.So think about two phases:
   a.The first is to start SlowPeerCollector thread
   b.Second, you can control whether to enable read/write avoid slow datanode 
according to dynamic parameters




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 674440)
Time Spent: 4h 40m  (was: 4.5h)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=674439&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-674439
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 03/Nov/21 08:21
Start Date: 03/Nov/21 08:21
Worklog Time Spent: 10m 
  Work Description: haiyang1987 opened a new pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596


   ### Description of PR
   
   Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
   Details: HDFS-16287
   
   ### For code changes:
   
   - [ ] Consider that make dfs.namenode.avoid.read.slow.datanode 
reconfigurable and rapid rollback in case this feature HDFS-16076  unexpected 
things happen in production environment
   - [ ] DatanodeManager#startSlowPeerCollector  by parameter 
'dfs.datanode.peer.stats.enabled' to control
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 674439)
Time Spent: 4.5h  (was: 4h 20m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=674438&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-674438
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 03/Nov/21 08:20
Start Date: 03/Nov/21 08:20
Worklog Time Spent: 10m 
  Work Description: haiyang1987 removed a comment on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-958731868


   > @haiyang1987 Thanks for contribution, some comments: we can change the 
title here and jira, If 
dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled is not 
reconfigurable. And I will check whether 
dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled is unused.
   
   
   
   > @haiyang1987 Thanks for contribution, some comments: we can change the 
title here and jira, If 
dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled is not 
reconfigurable. And I will check whether 
dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled is unused.
   
   @ferhui Thank you for your review. 
   1.Already change the title here and jira 
   2. 'dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled' in 
BlockPlacementPolicyDefault use


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 674438)
Time Spent: 4h 20m  (was: 4h 10m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=674436&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-674436
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 03/Nov/21 08:20
Start Date: 03/Nov/21 08:20
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-958734227


   
   
   
   
   > @haiyang1987 Thanks for contribution, some comments: we can change the 
title here and jira, If 
dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled is not 
reconfigurable. And I will check whether 
dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled is unused.
   
   @ferhui Thank you for your review. 
   1.Already change the title here and jira 
   2. 'dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled' in 
BlockPlacementPolicyDefault use


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 674436)
Time Spent: 4h  (was: 3h 50m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=674437&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-674437
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 03/Nov/21 08:20
Start Date: 03/Nov/21 08:20
Worklog Time Spent: 10m 
  Work Description: haiyang1987 closed pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 674437)
Time Spent: 4h 10m  (was: 4h)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=674434&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-674434
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 03/Nov/21 08:16
Start Date: 03/Nov/21 08:16
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-958731868


   > @haiyang1987 Thanks for contribution, some comments: we can change the 
title here and jira, If 
dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled is not 
reconfigurable. And I will check whether 
dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled is unused.
   
   
   
   > @haiyang1987 Thanks for contribution, some comments: we can change the 
title here and jira, If 
dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled is not 
reconfigurable. And I will check whether 
dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled is unused.
   
   @ferhui Thank you for your review. 
   1.Already change the title here and jira 
   2. 'dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled' in 
BlockPlacementPolicyDefault use


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 674434)
Time Spent: 3h 50m  (was: 3h 40m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-02 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=674197&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-674197
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 02/Nov/21 21:50
Start Date: 02/Nov/21 21:50
Worklog Time Spent: 10m 
  Work Description: ferhui commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-957338822


   @haiyang1987 Thanks for contribution, some comments:
   we can change the title here and jira, If 
dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled is not 
reconfigurable.
   And I will check whether 
dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled is unused.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 674197)
Time Spent: 3h 40m  (was: 3.5h)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-02 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=674168&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-674168
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 02/Nov/21 21:47
Start Date: 02/Nov/21 21:47
Worklog Time Spent: 10m 
  Work Description: tomscut commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-957437161


   > @ferhui @tomscut I submitted some code. Can you help review. thank you 
very much.
   
   Thanks for reminding me. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 674168)
Time Spent: 3.5h  (was: 3h 20m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-02 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=674122&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-674122
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 02/Nov/21 21:43
Start Date: 02/Nov/21 21:43
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-957160686


   @ferhui @tomscut I submitted some code. Can you help review.
   thank you very much.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 674122)
Time Spent: 3h 20m  (was: 3h 10m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-02 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=674104&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-674104
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 02/Nov/21 21:41
Start Date: 02/Nov/21 21:41
Worklog Time Spent: 10m 
  Work Description: tomscut commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r741006484



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   We might need to check whether the slowPeerTracker is started or we 
might not get the slow peers.

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -260,17 +257,14 @@
 final Timer timer = new Timer();
 this.slowPeerTracker = dataNodePeerStatsEnabled ?
 new SlowPeerTracker(conf, timer) : null;
-this.excludeSlowNodesEnabled = conf.getBoolean(
-DFS_NAMENODE_BLOCKPLACEMENTPOLICY_EXCLUDE_SLOW_NODES_ENABLED_KEY,
-DFS_NAMENODE_BLOCKPLACEMENTPOLICY_EXCLUDE_SLOW_NODES_ENABLED_DEFAULT);
 this.maxSlowPeerReportNodes = conf.getInt(
 DFSConfigKeys.DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY,
 DFSConfigKeys.DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_DEFAULT);
 this.slowPeerCollectionInterval = conf.getTimeDuration(
 DFSConfigKeys.DFS_NAMENODE_SLOWPEER_COLLECT_INTERVAL_KEY,
 DFSConfigKeys.DFS_NAMENODE_SLOWPEER_COLLECT_INTERVAL_DEFAULT,
 TimeUnit.MILLISECONDS);
-if (slowPeerTracker != null && excludeSlowNodesEnabled) {

Review comment:
   If this change is made, the SlowPeerCollector thread will be started 
regardless of whether we enable this feature.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 674104)
Time Spent: 3h 10m  (was: 3h)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-02 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=673893&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-673893
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 02/Nov/21 21:17
Start Date: 02/Nov/21 21:17
Worklog Time Spent: 10m 
  Work Description: ferhui commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r740948534



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -260,17 +257,14 @@
 final Timer timer = new Timer();
 this.slowPeerTracker = dataNodePeerStatsEnabled ?
 new SlowPeerTracker(conf, timer) : null;
-this.excludeSlowNodesEnabled = conf.getBoolean(

Review comment:
   Is it unused? If so we can remove it from hdfs-default.xml.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 673893)
Time Spent: 3h  (was: 2h 50m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-02 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=673676&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-673676
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 02/Nov/21 18:24
Start Date: 02/Nov/21 18:24
Worklog Time Spent: 10m 
  Work Description: ferhui commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-957338822


   @haiyang1987 Thanks for contribution, some comments:
   we can change the title here and jira, If 
dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled is not 
reconfigurable.
   And I will check whether 
dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled is unused.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 673676)
Time Spent: 2h 50m  (was: 2h 40m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-02 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=673594&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-673594
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 02/Nov/21 18:15
Start Date: 02/Nov/21 18:15
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-957160686


   @ferhui @tomscut I submitted some code. Can you help review.
   thank you very much.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 673594)
Time Spent: 2h 40m  (was: 2.5h)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-02 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=673575&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-673575
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 02/Nov/21 18:13
Start Date: 02/Nov/21 18:13
Worklog Time Spent: 10m 
  Work Description: tomscut commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r741006484



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   We might need to check whether the slowPeerTracker is started or we 
might not get the slow peers.

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -260,17 +257,14 @@
 final Timer timer = new Timer();
 this.slowPeerTracker = dataNodePeerStatsEnabled ?
 new SlowPeerTracker(conf, timer) : null;
-this.excludeSlowNodesEnabled = conf.getBoolean(
-DFS_NAMENODE_BLOCKPLACEMENTPOLICY_EXCLUDE_SLOW_NODES_ENABLED_KEY,
-DFS_NAMENODE_BLOCKPLACEMENTPOLICY_EXCLUDE_SLOW_NODES_ENABLED_DEFAULT);
 this.maxSlowPeerReportNodes = conf.getInt(
 DFSConfigKeys.DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY,
 DFSConfigKeys.DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_DEFAULT);
 this.slowPeerCollectionInterval = conf.getTimeDuration(
 DFSConfigKeys.DFS_NAMENODE_SLOWPEER_COLLECT_INTERVAL_KEY,
 DFSConfigKeys.DFS_NAMENODE_SLOWPEER_COLLECT_INTERVAL_DEFAULT,
 TimeUnit.MILLISECONDS);
-if (slowPeerTracker != null && excludeSlowNodesEnabled) {

Review comment:
   If this change is made, the SlowPeerCollector thread will be started 
regardless of whether we enable this feature.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 673575)
Time Spent: 2.5h  (was: 2h 20m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-02 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=673449&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-673449
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 02/Nov/21 17:58
Start Date: 02/Nov/21 17:58
Worklog Time Spent: 10m 
  Work Description: tomscut commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-957437161


   > @ferhui @tomscut I submitted some code. Can you help review. thank you 
very much.
   
   Thanks for reminding me. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 673449)
Time Spent: 2h 20m  (was: 2h 10m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-02 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=673309&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-673309
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 02/Nov/21 17:41
Start Date: 02/Nov/21 17:41
Worklog Time Spent: 10m 
  Work Description: ferhui commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r740948534



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -260,17 +257,14 @@
 final Timer timer = new Timer();
 this.slowPeerTracker = dataNodePeerStatsEnabled ?
 new SlowPeerTracker(conf, timer) : null;
-this.excludeSlowNodesEnabled = conf.getBoolean(

Review comment:
   Is it unused? If so we can remove it from hdfs-default.xml.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 673309)
Time Spent: 2h 10m  (was: 2h)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-02 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=673130&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-673130
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 02/Nov/21 12:27
Start Date: 02/Nov/21 12:27
Worklog Time Spent: 10m 
  Work Description: tomscut commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r741006484



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -511,7 +505,16 @@ private boolean isInactive(DatanodeInfo datanode) {
   private boolean isSlowNode(String dnUuid) {
 return avoidSlowDataNodesForRead && slowNodesUuidSet.contains(dnUuid);
   }
-  
+
+  public void setAvoidSlowDataNodesForReadEnabled(boolean enable) {

Review comment:
   We might need to check whether the slowPeerTracker is started or we 
might not get the slow peers.

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -260,17 +257,14 @@
 final Timer timer = new Timer();
 this.slowPeerTracker = dataNodePeerStatsEnabled ?
 new SlowPeerTracker(conf, timer) : null;
-this.excludeSlowNodesEnabled = conf.getBoolean(
-DFS_NAMENODE_BLOCKPLACEMENTPOLICY_EXCLUDE_SLOW_NODES_ENABLED_KEY,
-DFS_NAMENODE_BLOCKPLACEMENTPOLICY_EXCLUDE_SLOW_NODES_ENABLED_DEFAULT);
 this.maxSlowPeerReportNodes = conf.getInt(
 DFSConfigKeys.DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY,
 DFSConfigKeys.DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_DEFAULT);
 this.slowPeerCollectionInterval = conf.getTimeDuration(
 DFSConfigKeys.DFS_NAMENODE_SLOWPEER_COLLECT_INTERVAL_KEY,
 DFSConfigKeys.DFS_NAMENODE_SLOWPEER_COLLECT_INTERVAL_DEFAULT,
 TimeUnit.MILLISECONDS);
-if (slowPeerTracker != null && excludeSlowNodesEnabled) {

Review comment:
   If this change is made, the SlowPeerCollector thread will be started 
regardless of whether we enable this feature.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 673130)
Time Spent: 2h  (was: 1h 50m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-02 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=673125&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-673125
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 02/Nov/21 12:22
Start Date: 02/Nov/21 12:22
Worklog Time Spent: 10m 
  Work Description: tomscut commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-957437161


   > @ferhui @tomscut I submitted some code. Can you help review. thank you 
very much.
   
   Thanks for reminding me. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 673125)
Time Spent: 1h 50m  (was: 1h 40m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-02 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=673090&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-673090
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 02/Nov/21 11:07
Start Date: 02/Nov/21 11:07
Worklog Time Spent: 10m 
  Work Description: ferhui commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-957338822


   @haiyang1987 Thanks for contribution, some comments:
   we can change the title here and jira, If 
dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled is not 
reconfigurable.
   And I will check whether 
dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled is unused.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 673090)
Time Spent: 1h 40m  (was: 1.5h)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-02 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=673088&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-673088
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 02/Nov/21 11:06
Start Date: 02/Nov/21 11:06
Worklog Time Spent: 10m 
  Work Description: ferhui commented on a change in pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#discussion_r740948534



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
##
@@ -260,17 +257,14 @@
 final Timer timer = new Timer();
 this.slowPeerTracker = dataNodePeerStatsEnabled ?
 new SlowPeerTracker(conf, timer) : null;
-this.excludeSlowNodesEnabled = conf.getBoolean(

Review comment:
   Is it unused? If so we can remove it from hdfs-default.xml.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 673088)
Time Spent: 1.5h  (was: 1h 20m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-11-02 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=673001&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-673001
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 02/Nov/21 07:23
Start Date: 02/Nov/21 07:23
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-957160686


   @ferhui @tomscut I submitted some code. Can you help review.
   thank you very much.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 673001)
Time Spent: 1h 20m  (was: 1h 10m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-10-30 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=672346&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672346
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 30/Oct/21 13:10
Start Date: 30/Oct/21 13:10
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-955207002


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   1m  4s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 2 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  36m 40s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   1m 37s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |   1m 24s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  checkstyle  |   1m  3s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 33s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   1m  2s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   1m 31s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   3m 37s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  25m 21s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   1m 24s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 27s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javac  |   1m 27s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 17s |  |  the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  javac  |   1m 17s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | +1 :green_heart: |  checkstyle  |   0m 56s |  |  the patch passed  |
   | +1 :green_heart: |  mvnsite  |   1m 27s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 53s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   1m 21s |  |  the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   3m 43s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  25m 37s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | -1 :x: |  unit  | 356m 31s | 
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3596/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
 |  hadoop-hdfs in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 45s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 467m  0s |  |  |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.hdfs.TestViewDistributedFileSystemContract |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3596/3/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/3596 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell |
   | uname | Linux c7a67aa2444b 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 2ec82e1c420789afb326f4ebb451522a8a4e2358 |
   | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   | Multi-JDK versions | 
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 
/usr/lib/jvm/java-8-openjdk-amd64:Private 
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   |  Test Results | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3596/3/testReport/ |
   | Max. process+thread count | 2010 (vs. ulimit of 5500) |
   | modules | C: hadoop-hdfs-project

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-10-29 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=672316&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672316
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 30/Oct/21 05:22
Start Date: 30/Oct/21 05:22
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-955150991


   Fix code style and remove useless code


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 672316)
Time Spent: 1h  (was: 50m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-10-29 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=672141&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672141
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 29/Oct/21 19:35
Start Date: 29/Oct/21 19:35
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-955001002


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   1m  6s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 2 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  35m 37s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   1m 35s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |   1m 23s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  checkstyle  |   1m  2s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 31s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   1m  2s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   1m 30s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   3m 38s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  25m 34s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   1m 23s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 31s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javac  |   1m 31s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 18s |  |  the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  javac  |   1m 18s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | -0 :warning: |  checkstyle  |   0m 55s | 
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3596/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
 |  hadoop-hdfs-project/hadoop-hdfs: The patch generated 5 new + 122 unchanged 
- 0 fixed = 127 total (was 122)  |
   | +1 :green_heart: |  mvnsite  |   1m 24s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 54s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   1m 26s |  |  the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   3m 41s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  25m 50s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | -1 :x: |  unit  | 385m 45s | 
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3596/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
 |  hadoop-hdfs in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   1m  4s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 495m 49s |  |  |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.hdfs.TestHDFSFileSystemContract |
   |   | hadoop.hdfs.TestViewDistributedFileSystemContract |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3596/2/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/3596 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell |
   | uname | Linux 297d371d729d 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 91aaaea72037e88b2a3af970ab81380a990307e8 |
   | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   | Multi-JDK versions | 
/usr/lib/jvm/java-11-openjdk-amd64

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

2021-10-29 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=671912&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-671912
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 29/Oct/21 11:28
Start Date: 29/Oct/21 11:28
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-954667726


   Update PR, Make the following changes:
   1. Consider for  make dfs.namenode.avoid.read.slow.datanode reconfigurable
   2. HDFS-15120 already support 
BlockPlacementPolicyDefault#dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled
 reconfigurable


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 671912)
Time Spent: 40m  (was: 0.5h)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

66 matches

Mail list logo