[jira] [Work logged] (HDFS-16277) Improve decision in AvailableSpaceBlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=668761=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-668761 ] ASF GitHub Bot logged work on HDFS-16277: - Author: ASF GitHub Bot Created on: 22/Oct/21 02:32 Start Date: 22/Oct/21 02:32 Worklog Time Spent: 10m Work Description: GuoPhilipse commented on pull request #3559: URL: https://github.com/apache/hadoop/pull/3559#issuecomment-949237284 Thanks again @ayushtkn @prasad-acit for your kindly review. :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 668761) Time Spent: 5h 10m (was: 5h) > Improve decision in AvailableSpaceBlockPlacementPolicy > -- > > Key: HDFS-16277 > URL: https://issues.apache.org/jira/browse/HDFS-16277 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Affects Versions: 3.3.1 >Reporter: guo >Assignee: guo >Priority: Major > Labels: pull-request-available > Time Spent: 5h 10m > Remaining Estimate: 0h > > Hi > In product environment,we may meet two or more datanode usage reaches nealy > 100%,for exampe 99.99%,98%,97%. > if we configure `AvailableSpaceBlockPlacementPolicy` , we also have the > chance to choose the 99.99%(assume it is the highest usage),for we treat the > two choosen datanode as the same usage if their storage usage are different > within 5%. > but this is not what we want, so i suggest we can improve the decision. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16277) Improve decision in AvailableSpaceBlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=668760=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-668760 ] ASF GitHub Bot logged work on HDFS-16277: - Author: ASF GitHub Bot Created on: 22/Oct/21 02:31 Start Date: 22/Oct/21 02:31 Worklog Time Spent: 10m Work Description: ayushtkn commented on pull request #3559: URL: https://github.com/apache/hadoop/pull/3559#issuecomment-949236776 Thanx @GuoPhilipse for the contribution, Have merged the PR. Yeps, the build is ok. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 668760) Time Spent: 5h (was: 4h 50m) > Improve decision in AvailableSpaceBlockPlacementPolicy > -- > > Key: HDFS-16277 > URL: https://issues.apache.org/jira/browse/HDFS-16277 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Affects Versions: 3.3.1 >Reporter: guo >Assignee: guo >Priority: Major > Labels: pull-request-available > Time Spent: 5h > Remaining Estimate: 0h > > Hi > In product environment,we may meet two or more datanode usage reaches nealy > 100%,for exampe 99.99%,98%,97%. > if we configure `AvailableSpaceBlockPlacementPolicy` , we also have the > chance to choose the 99.99%(assume it is the highest usage),for we treat the > two choosen datanode as the same usage if their storage usage are different > within 5%. > but this is not what we want, so i suggest we can improve the decision. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16277) Improve decision in AvailableSpaceBlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=668759=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-668759 ] ASF GitHub Bot logged work on HDFS-16277: - Author: ASF GitHub Bot Created on: 22/Oct/21 02:29 Start Date: 22/Oct/21 02:29 Worklog Time Spent: 10m Work Description: ayushtkn merged pull request #3559: URL: https://github.com/apache/hadoop/pull/3559 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 668759) Time Spent: 4h 50m (was: 4h 40m) > Improve decision in AvailableSpaceBlockPlacementPolicy > -- > > Key: HDFS-16277 > URL: https://issues.apache.org/jira/browse/HDFS-16277 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Affects Versions: 3.3.1 >Reporter: guo >Assignee: guo >Priority: Major > Labels: pull-request-available > Time Spent: 4h 50m > Remaining Estimate: 0h > > Hi > In product environment,we may meet two or more datanode usage reaches nealy > 100%,for exampe 99.99%,98%,97%. > if we configure `AvailableSpaceBlockPlacementPolicy` , we also have the > chance to choose the 99.99%(assume it is the highest usage),for we treat the > two choosen datanode as the same usage if their storage usage are different > within 5%. > but this is not what we want, so i suggest we can improve the decision. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=668710=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-668710 ] ASF GitHub Bot logged work on HDFS-16277: - Author: ASF GitHub Bot Created on: 22/Oct/21 00:02 Start Date: 22/Oct/21 00:02 Worklog Time Spent: 10m Work Description: GuoPhilipse commented on a change in pull request #3559: URL: https://github.com/apache/hadoop/pull/3559#discussion_r734123264 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceRackFaultTolerantBlockPlacementPolicy.java ## @@ -54,6 +57,10 @@ public void initialize(Configuration conf, FSClusterStats stats, DFS_NAMENODE_AVAILABLE_SPACE_RACK_FAULT_TOLERANT_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_PREFERENCE_FRACTION_KEY, DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_RACK_FAULT_TOLERANT_PLACEMENT_POLICY_BALANCED_SPACE_PREFERENCE_FRACTION_DEFAULT); +balancedSpaceTolerance = conf.getInt( + DFS_NAMENODE_AVAILABLE_SPACE_RACK_FAULT_TOLERANT_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_TOLERANCE_KEY, + DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_RACK_FAULT_TOLERANT_PLACEMENT_POLICY_BALANCED_SPACE_TOLERANCE_DEFAULT); Review comment: Thanks @prasad-acit for your advice ,it should be used for total new code , for now, if we cut ir for short, the variable name meaning may not that clear,maybe we can igonre it, and apply the short name as far as possible in furture. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 668710) Time Spent: 4h 40m (was: 4.5h) > improve decision in AvailableSpaceBlockPlacementPolicy > -- > > Key: HDFS-16277 > URL: https://issues.apache.org/jira/browse/HDFS-16277 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Affects Versions: 3.3.1 >Reporter: guo >Priority: Major > Labels: pull-request-available > Time Spent: 4h 40m > Remaining Estimate: 0h > > Hi > In product environment,we may meet two or more datanode usage reaches nealy > 100%,for exampe 99.99%,98%,97%. > if we configure `AvailableSpaceBlockPlacementPolicy` , we also have the > chance to choose the 99.99%(assume it is the highest usage),for we treat the > two choosen datanode as the same usage if their storage usage are different > within 5%. > but this is not what we want, so i suggest we can improve the decision. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=668705=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-668705 ] ASF GitHub Bot logged work on HDFS-16277: - Author: ASF GitHub Bot Created on: 21/Oct/21 23:54 Start Date: 21/Oct/21 23:54 Worklog Time Spent: 10m Work Description: GuoPhilipse commented on pull request #3559: URL: https://github.com/apache/hadoop/pull/3559#issuecomment-949084796 > @GuoPhilipse I have retriggred the build here: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/13/console More or less looks like an Infra issue If it works, cool, if not I will try to figure that out. It is on me. :-) Cool,it shows +1 overall, but still have failed check in report ,not sure if it is normal ? @ayushtkn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 668705) Time Spent: 4.5h (was: 4h 20m) > improve decision in AvailableSpaceBlockPlacementPolicy > -- > > Key: HDFS-16277 > URL: https://issues.apache.org/jira/browse/HDFS-16277 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Affects Versions: 3.3.1 >Reporter: guo >Priority: Major > Labels: pull-request-available > Time Spent: 4.5h > Remaining Estimate: 0h > > Hi > In product environment,we may meet two or more datanode usage reaches nealy > 100%,for exampe 99.99%,98%,97%. > if we configure `AvailableSpaceBlockPlacementPolicy` , we also have the > chance to choose the 99.99%(assume it is the highest usage),for we treat the > two choosen datanode as the same usage if their storage usage are different > within 5%. > but this is not what we want, so i suggest we can improve the decision. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=668620=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-668620
]
ASF GitHub Bot logged work on HDFS-16277:
-
Author: ASF GitHub Bot
Created on: 21/Oct/21 19:42
Start Date: 21/Oct/21 19:42
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3559:
URL: https://github.com/apache/hadoop/pull/3559#issuecomment-948946779
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 16s | | trunk passed |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 5s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 29s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 34s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 13s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 42s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 16s | | the patch passed |
| +1 :green_heart: | compile | 1m 25s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 1m 10s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 51s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/13/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 8 new + 230 unchanged
- 1 fixed = 238 total (was 231) |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 25s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 35s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 224m 15s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 322m 26s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/13/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3559 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml markdownlint |
| uname | Linux d084898ca220 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7d5be4e32e772a732e222d058552b79f42bfafbe |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=668489=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-668489 ] ASF GitHub Bot logged work on HDFS-16277: - Author: ASF GitHub Bot Created on: 21/Oct/21 15:19 Start Date: 21/Oct/21 15:19 Worklog Time Spent: 10m Work Description: GuoPhilipse commented on pull request #3559: URL: https://github.com/apache/hadoop/pull/3559#issuecomment-948721675 > @GuoPhilipse I have retriggred the build here: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/13/console More or less looks like an Infra issue If it works, cool, if not I will try to figure that out. It is on me. :-) Realy appreciate it! @ayushtkn :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 668489) Time Spent: 4h 10m (was: 4h) > improve decision in AvailableSpaceBlockPlacementPolicy > -- > > Key: HDFS-16277 > URL: https://issues.apache.org/jira/browse/HDFS-16277 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Affects Versions: 3.3.1 >Reporter: guo >Priority: Major > Labels: pull-request-available > Time Spent: 4h 10m > Remaining Estimate: 0h > > Hi > In product environment,we may meet two or more datanode usage reaches nealy > 100%,for exampe 99.99%,98%,97%. > if we configure `AvailableSpaceBlockPlacementPolicy` , we also have the > chance to choose the 99.99%(assume it is the highest usage),for we treat the > two choosen datanode as the same usage if their storage usage are different > within 5%. > but this is not what we want, so i suggest we can improve the decision. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=668460=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-668460 ] ASF GitHub Bot logged work on HDFS-16277: - Author: ASF GitHub Bot Created on: 21/Oct/21 14:24 Start Date: 21/Oct/21 14:24 Worklog Time Spent: 10m Work Description: ayushtkn commented on pull request #3559: URL: https://github.com/apache/hadoop/pull/3559#issuecomment-948670394 @GuoPhilipse I have retriggred the build here: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/13/console More or less looks like an Infra issue If it works, cool, if not I will try to figure that out. It is on me. :-) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 668460) Time Spent: 4h (was: 3h 50m) > improve decision in AvailableSpaceBlockPlacementPolicy > -- > > Key: HDFS-16277 > URL: https://issues.apache.org/jira/browse/HDFS-16277 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Affects Versions: 3.3.1 >Reporter: guo >Priority: Major > Labels: pull-request-available > Time Spent: 4h > Remaining Estimate: 0h > > Hi > In product environment,we may meet two or more datanode usage reaches nealy > 100%,for exampe 99.99%,98%,97%. > if we configure `AvailableSpaceBlockPlacementPolicy` , we also have the > chance to choose the 99.99%(assume it is the highest usage),for we treat the > two choosen datanode as the same usage if their storage usage are different > within 5%. > but this is not what we want, so i suggest we can improve the decision. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=668441=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-668441 ] ASF GitHub Bot logged work on HDFS-16277: - Author: ASF GitHub Bot Created on: 21/Oct/21 13:59 Start Date: 21/Oct/21 13:59 Worklog Time Spent: 10m Work Description: prasad-acit commented on a change in pull request #3559: URL: https://github.com/apache/hadoop/pull/3559#discussion_r733708491 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceRackFaultTolerantBlockPlacementPolicy.java ## @@ -54,6 +57,10 @@ public void initialize(Configuration conf, FSClusterStats stats, DFS_NAMENODE_AVAILABLE_SPACE_RACK_FAULT_TOLERANT_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_PREFERENCE_FRACTION_KEY, DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_RACK_FAULT_TOLERANT_PLACEMENT_POLICY_BALANCED_SPACE_PREFERENCE_FRACTION_DEFAULT); +balancedSpaceTolerance = conf.getInt( + DFS_NAMENODE_AVAILABLE_SPACE_RACK_FAULT_TOLERANT_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_TOLERANCE_KEY, + DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_RACK_FAULT_TOLERANT_PLACEMENT_POLICY_BALANCED_SPACE_TOLERANCE_DEFAULT); Review comment: Constants exceed the length lead to length issue, can be shrinked. BLOCK_PLACEMENT_POLICY => BPP can be used. Other places already large, atleast new code we can target. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 668441) Time Spent: 3h 50m (was: 3h 40m) > improve decision in AvailableSpaceBlockPlacementPolicy > -- > > Key: HDFS-16277 > URL: https://issues.apache.org/jira/browse/HDFS-16277 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Affects Versions: 3.3.1 >Reporter: guo >Priority: Major > Labels: pull-request-available > Time Spent: 3h 50m > Remaining Estimate: 0h > > Hi > In product environment,we may meet two or more datanode usage reaches nealy > 100%,for exampe 99.99%,98%,97%. > if we configure `AvailableSpaceBlockPlacementPolicy` , we also have the > chance to choose the 99.99%(assume it is the highest usage),for we treat the > two choosen datanode as the same usage if their storage usage are different > within 5%. > but this is not what we want, so i suggest we can improve the decision. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=668438=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-668438
]
ASF GitHub Bot logged work on HDFS-16277:
-
Author: ASF GitHub Bot
Created on: 21/Oct/21 13:48
Start Date: 21/Oct/21 13:48
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3559:
URL: https://github.com/apache/hadoop/pull/3559#issuecomment-948636284
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 0m 25s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/12/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 0m 25s |
[/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/12/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in trunk failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 0m 25s |
[/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/12/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-hdfs in trunk failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -0 :warning: | checkstyle | 0m 23s |
[/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/12/artifact/out/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| The patch fails to run checkstyle in hadoop-hdfs |
| -1 :x: | mvnsite | 0m 24s |
[/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/12/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in trunk failed. |
| -1 :x: | javadoc | 0m 25s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/12/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in trunk failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javadoc | 0m 25s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/12/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-hdfs in trunk failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | spotbugs | 0m 24s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/12/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in trunk failed. |
| +1 :green_heart: | shadedclient | 2m 52s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 25s |
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/12/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| -1 :x: | compile | 0m 25s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/12/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 0m 25s |
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=668426=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-668426 ] ASF GitHub Bot logged work on HDFS-16277: - Author: ASF GitHub Bot Created on: 21/Oct/21 13:33 Start Date: 21/Oct/21 13:33 Worklog Time Spent: 10m Work Description: GuoPhilipse commented on pull request #3559: URL: https://github.com/apache/hadoop/pull/3559#issuecomment-948622637 It seems github issue ? the mvn error message was the following `ERROR: Failed to write github status. Token expired or missing repo:status write?` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 668426) Time Spent: 3.5h (was: 3h 20m) > improve decision in AvailableSpaceBlockPlacementPolicy > -- > > Key: HDFS-16277 > URL: https://issues.apache.org/jira/browse/HDFS-16277 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Affects Versions: 3.3.1 >Reporter: guo >Priority: Major > Labels: pull-request-available > Time Spent: 3.5h > Remaining Estimate: 0h > > Hi > In product environment,we may meet two or more datanode usage reaches nealy > 100%,for exampe 99.99%,98%,97%. > if we configure `AvailableSpaceBlockPlacementPolicy` , we also have the > chance to choose the 99.99%(assume it is the highest usage),for we treat the > two choosen datanode as the same usage if their storage usage are different > within 5%. > but this is not what we want, so i suggest we can improve the decision. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=668422=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-668422
]
ASF GitHub Bot logged work on HDFS-16277:
-
Author: ASF GitHub Bot
Created on: 21/Oct/21 13:31
Start Date: 21/Oct/21 13:31
Worklog Time Spent: 10m
Work Description: GuoPhilipse commented on a change in pull request #3559:
URL: https://github.com/apache/hadoop/pull/3559#discussion_r733679533
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java
##
@@ -77,6 +86,16 @@ public void initialize(Configuration conf, FSClusterStats
stats,
+ " is less than 0.5 so datanodes with more used percent will"
+ " receive more block allocations.");
}
+
+if (balancedSpaceTolerance > 20 || balancedSpaceTolerance < 0) {
+ LOG.warn("The value of "
+ +
DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_TOLERANCE_KEY
Review comment:
thanks @prasad-acit ,seems more clear for the warning message
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 668422)
Time Spent: 3h 20m (was: 3h 10m)
> improve decision in AvailableSpaceBlockPlacementPolicy
> --
>
> Key: HDFS-16277
> URL: https://issues.apache.org/jira/browse/HDFS-16277
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: block placement
>Affects Versions: 3.3.1
>Reporter: guo
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Hi
> In product environment,we may meet two or more datanode usage reaches nealy
> 100%,for exampe 99.99%,98%,97%.
> if we configure `AvailableSpaceBlockPlacementPolicy` , we also have the
> chance to choose the 99.99%(assume it is the highest usage),for we treat the
> two choosen datanode as the same usage if their storage usage are different
> within 5%.
> but this is not what we want, so i suggest we can improve the decision.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=668409=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-668409
]
ASF GitHub Bot logged work on HDFS-16277:
-
Author: ASF GitHub Bot
Created on: 21/Oct/21 13:25
Start Date: 21/Oct/21 13:25
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3559:
URL: https://github.com/apache/hadoop/pull/3559#issuecomment-948615491
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 0m 25s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/11/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 0m 25s |
[/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/11/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in trunk failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 0m 25s |
[/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/11/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-hdfs in trunk failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -0 :warning: | checkstyle | 0m 22s |
[/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/11/artifact/out/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| The patch fails to run checkstyle in hadoop-hdfs |
| -1 :x: | mvnsite | 0m 24s |
[/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/11/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in trunk failed. |
| -1 :x: | javadoc | 0m 24s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/11/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in trunk failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javadoc | 0m 24s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/11/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-hdfs in trunk failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | spotbugs | 0m 24s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/11/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in trunk failed. |
| +1 :green_heart: | shadedclient | 2m 49s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 24s |
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/11/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| -1 :x: | compile | 0m 25s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/11/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 0m 25s |
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=668401=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-668401
]
ASF GitHub Bot logged work on HDFS-16277:
-
Author: ASF GitHub Bot
Created on: 21/Oct/21 13:15
Start Date: 21/Oct/21 13:15
Worklog Time Spent: 10m
Work Description: prasad-acit commented on a change in pull request #3559:
URL: https://github.com/apache/hadoop/pull/3559#discussion_r733664093
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java
##
@@ -77,6 +86,16 @@ public void initialize(Configuration conf, FSClusterStats
stats,
+ " is less than 0.5 so datanodes with more used percent will"
+ " receive more block allocations.");
}
+
+if (balancedSpaceTolerance > 20 || balancedSpaceTolerance < 0) {
+ LOG.warn("The value of "
+ +
DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_TOLERANCE_KEY
Review comment:
It would be great, if you can print the use configured value here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 668401)
Time Spent: 3h (was: 2h 50m)
> improve decision in AvailableSpaceBlockPlacementPolicy
> --
>
> Key: HDFS-16277
> URL: https://issues.apache.org/jira/browse/HDFS-16277
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: block placement
>Affects Versions: 3.3.1
>Reporter: guo
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Hi
> In product environment,we may meet two or more datanode usage reaches nealy
> 100%,for exampe 99.99%,98%,97%.
> if we configure `AvailableSpaceBlockPlacementPolicy` , we also have the
> chance to choose the 99.99%(assume it is the highest usage),for we treat the
> two choosen datanode as the same usage if their storage usage are different
> within 5%.
> but this is not what we want, so i suggest we can improve the decision.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=668380=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-668380
]
ASF GitHub Bot logged work on HDFS-16277:
-
Author: ASF GitHub Bot
Created on: 21/Oct/21 12:38
Start Date: 21/Oct/21 12:38
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3559:
URL: https://github.com/apache/hadoop/pull/3559#issuecomment-948573705
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 0m 22s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/10/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 0m 22s |
[/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/10/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in trunk failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 0m 23s |
[/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/10/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-hdfs in trunk failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -0 :warning: | checkstyle | 0m 20s |
[/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/10/artifact/out/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| The patch fails to run checkstyle in hadoop-hdfs |
| -1 :x: | mvnsite | 0m 22s |
[/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/10/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in trunk failed. |
| -1 :x: | javadoc | 0m 22s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/10/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in trunk failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javadoc | 0m 22s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/10/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-hdfs in trunk failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | spotbugs | 0m 22s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/10/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in trunk failed. |
| +1 :green_heart: | shadedclient | 2m 36s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 22s |
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/10/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| -1 :x: | compile | 0m 22s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/10/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 0m 22s |
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=668341=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-668341
]
ASF GitHub Bot logged work on HDFS-16277:
-
Author: ASF GitHub Bot
Created on: 21/Oct/21 12:09
Start Date: 21/Oct/21 12:09
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3559:
URL: https://github.com/apache/hadoop/pull/3559#issuecomment-948549468
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 1m 27s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/9/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 0m 22s |
[/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/9/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in trunk failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 0m 23s |
[/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/9/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-hdfs in trunk failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -0 :warning: | checkstyle | 0m 21s |
[/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/9/artifact/out/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| The patch fails to run checkstyle in hadoop-hdfs |
| -1 :x: | mvnsite | 0m 23s |
[/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/9/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in trunk failed. |
| -1 :x: | javadoc | 0m 23s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/9/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in trunk failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javadoc | 0m 23s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/9/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-hdfs in trunk failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | spotbugs | 0m 22s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/9/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in trunk failed. |
| +1 :green_heart: | shadedclient | 2m 39s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 23s |
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/9/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| -1 :x: | compile | 0m 22s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/9/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 0m 22s |
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=668331=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-668331 ] ASF GitHub Bot logged work on HDFS-16277: - Author: ASF GitHub Bot Created on: 21/Oct/21 11:56 Start Date: 21/Oct/21 11:56 Worklog Time Spent: 10m Work Description: GuoPhilipse commented on pull request #3559: URL: https://github.com/apache/hadoop/pull/3559#issuecomment-948538645 > Jenkins have reported some checkstyle warnings https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/8/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt > > They seems to be for line length, Can you check once, I guess the ones reported from test can be fixed. Couple of them are because the variable name is big, I think we can ignore them, as there is no good way to get rid of them. > > Apart from that. Changes LGTM Thanks @ayushtkn ,checkstyle has been fixed.except for the long variable name ones. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 668331) Time Spent: 2.5h (was: 2h 20m) > improve decision in AvailableSpaceBlockPlacementPolicy > -- > > Key: HDFS-16277 > URL: https://issues.apache.org/jira/browse/HDFS-16277 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Affects Versions: 3.3.1 >Reporter: guo >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > Hi > In product environment,we may meet two or more datanode usage reaches nealy > 100%,for exampe 99.99%,98%,97%. > if we configure `AvailableSpaceBlockPlacementPolicy` , we also have the > chance to choose the 99.99%(assume it is the highest usage),for we treat the > two choosen datanode as the same usage if their storage usage are different > within 5%. > but this is not what we want, so i suggest we can improve the decision. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=668280=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-668280 ] ASF GitHub Bot logged work on HDFS-16277: - Author: ASF GitHub Bot Created on: 21/Oct/21 10:49 Start Date: 21/Oct/21 10:49 Worklog Time Spent: 10m Work Description: GuoPhilipse commented on pull request #3559: URL: https://github.com/apache/hadoop/pull/3559#issuecomment-948489020 @ayushtkn Thanks your timely review, comments have been improved and tests are ok. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 668280) Time Spent: 2h 20m (was: 2h 10m) > improve decision in AvailableSpaceBlockPlacementPolicy > -- > > Key: HDFS-16277 > URL: https://issues.apache.org/jira/browse/HDFS-16277 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Affects Versions: 3.3.1 >Reporter: guo >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > Hi > In product environment,we may meet two or more datanode usage reaches nealy > 100%,for exampe 99.99%,98%,97%. > if we configure `AvailableSpaceBlockPlacementPolicy` , we also have the > chance to choose the 99.99%(assume it is the highest usage),for we treat the > two choosen datanode as the same usage if their storage usage are different > within 5%. > but this is not what we want, so i suggest we can improve the decision. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=668256=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-668256
]
ASF GitHub Bot logged work on HDFS-16277:
-
Author: ASF GitHub Bot
Created on: 21/Oct/21 10:04
Start Date: 21/Oct/21 10:04
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3559:
URL: https://github.com/apache/hadoop/pull/3559#issuecomment-948455272
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 42s | | trunk passed |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 28s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 5s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 39s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 52s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/8/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 16 new + 230 unchanged
- 1 fixed = 246 total (was 231) |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 20s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 23s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 227m 53s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 323m 37s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3559 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml markdownlint |
| uname | Linux cabb3364f533 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19
23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 91b8d430f8e7e85062300f4c8e3019b46477ac2c |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=668075=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-668075
]
ASF GitHub Bot logged work on HDFS-16277:
-
Author: ASF GitHub Bot
Created on: 21/Oct/21 03:57
Start Date: 21/Oct/21 03:57
Worklog Time Spent: 10m
Work Description: ayushtkn commented on a change in pull request #3559:
URL: https://github.com/apache/hadoop/pull/3559#discussion_r733299381
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java
##
@@ -37,6 +39,7 @@
import org.apache.hadoop.net.NetworkTopology;
import org.apache.hadoop.net.Node;
+
Review comment:
nit:
Avoid this
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestAvailableSpaceRackFaultTolerantBPP.java
##
@@ -66,6 +68,9 @@ public static void setupCluster() throws Exception {
conf.setFloat(
DFSConfigKeys.DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_PREFERENCE_FRACTION_KEY,
0.6f);
+conf.setInt(
+
DFSConfigKeys.DFS_NAMENODE_AVAILABLE_SPACE_RACK_FAULT_TOLERANT_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_TOLERANCE_KEY,
+5);
Review comment:
It is by default 5. You need not to set it explicitly.
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestAvailableSpaceRackFaultTolerantBPP.java
##
@@ -206,6 +210,48 @@ public void testMaxRackAllocation() {
assertEquals(REPLICA, racks.size());
}
+ @Test
+ public void testChooseSimilarDataNode() {
+DatanodeDescriptor[] tolerateDataNodes;
+DatanodeStorageInfo[] tolerateStorages;
+AvailableSpaceRackFaultTolerantBlockPlacementPolicy
toleratePlacementPolicy;
Review comment:
Why are you declaring it separately. Could have done this below:
```
AvailableSpaceRackFaultTolerantBlockPlacementPolicy toleratePlacementPolicy
=
(AvailableSpaceRackFaultTolerantBlockPlacementPolicy)bm.getBlockPlacementPolicy();
```
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestAvailableSpaceBlockPlacementPolicy.java
##
@@ -61,6 +62,9 @@ public static void setupCluster() throws Exception {
conf.setFloat(
DFSConfigKeys.DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_PREFERENCE_FRACTION_KEY,
0.6f);
+conf.setInt(
+
DFSConfigKeys.DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_TOLERANCE_KEY,
+5);
Review comment:
It is by default 5. You need not to set it explicitly.
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceRackFaultTolerantBlockPlacementPolicy.java
##
@@ -70,6 +77,19 @@ public void initialize(Configuration conf, FSClusterStats
stats,
+ " is less than 0.5 so datanodes with more used percent will"
+ " receive more block allocations.");
}
+
+
+if (balancedSpaceTolerance >= 20 || balancedSpaceTolerance <= 0) {
+ LOG.warn("The value of "
+ +
DFS_NAMENODE_AVAILABLE_SPACE_RACK_FAULT_TOLERANT_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_TOLERANCE_KEY
+ + " is invalid, Default value " +
+
DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_RACK_FAULT_TOLERANT_PLACEMENT_POLICY_BALANCED_SPACE_TOLERANCE_DEFAULT
+ + " will be used instead. Increases tolerance of"
+ + " placing blocks on Datanodes with similar disk space used ");
Review comment:
No need of Increases to...
You can just stay `till will be used`
Remove the line after that
##
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
##
@@ -5067,6 +5067,18 @@
+
+
dfs.namenode.available-space-block-placement-policy.balanced-space-tolerance
+5
+
+ Only used when the dfs.block.replicator.classname is set to
+
org.apache.hadoop.hdfs.server.blockmanagement.AvailableSpaceBlockPlacementPolicy.
+ Special value between 0 and 20 , noninclusive. if the value is set
beyond the scope,
Review comment:
Change it to include 0 & 20
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java
##
@@ -77,6 +87,17 @@ public void initialize(Configuration conf, FSClusterStats
stats,
+ " is less than 0.5 so datanodes with more used percent will"
+ " receive more block allocations.");
}
+
+if (balancedSpaceTolerance >= 20 || balancedSpaceTolerance <= 0) {
Review comment:
I think 0 we can allow, May be just remove the = sign
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=668062=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-668062 ] ASF GitHub Bot logged work on HDFS-16277: - Author: ASF GitHub Bot Created on: 21/Oct/21 02:28 Start Date: 21/Oct/21 02:28 Worklog Time Spent: 10m Work Description: GuoPhilipse commented on pull request #3559: URL: https://github.com/apache/hadoop/pull/3559#issuecomment-948198750 @ayushtkn Thanks for your detail comments, they have been fixed and test cases passed as well. any new ideas ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 668062) Time Spent: 1h 50m (was: 1h 40m) > improve decision in AvailableSpaceBlockPlacementPolicy > -- > > Key: HDFS-16277 > URL: https://issues.apache.org/jira/browse/HDFS-16277 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Affects Versions: 3.3.1 >Reporter: guo >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > Hi > In product environment,we may meet two or more datanode usage reaches nealy > 100%,for exampe 99.99%,98%,97%. > if we configure `AvailableSpaceBlockPlacementPolicy` , we also have the > chance to choose the 99.99%(assume it is the highest usage),for we treat the > two choosen datanode as the same usage if their storage usage are different > within 5%. > but this is not what we want, so i suggest we can improve the decision. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=668032=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-668032
]
ASF GitHub Bot logged work on HDFS-16277:
-
Author: ASF GitHub Bot
Created on: 20/Oct/21 22:36
Start Date: 20/Oct/21 22:36
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3559:
URL: https://github.com/apache/hadoop/pull/3559#issuecomment-948088341
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 45s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 13s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 23s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 14s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 54s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/6/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 21 new + 230 unchanged
- 1 fixed = 251 total (was 231) |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 23s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 321m 19s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 426m 12s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3559 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml markdownlint |
| uname | Linux 0f728d034d69 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3af5420d63a9686a863f12176dbebff165d90c4f |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=667976=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-667976
]
ASF GitHub Bot logged work on HDFS-16277:
-
Author: ASF GitHub Bot
Created on: 20/Oct/21 20:51
Start Date: 20/Oct/21 20:51
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3559:
URL: https://github.com/apache/hadoop/pull/3559#issuecomment-948026808
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 41s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 21s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 6s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 6s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 53s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/7/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 21 new + 230 unchanged
- 1 fixed = 251 total (was 231) |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 18s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 224m 44s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 320m 9s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3559 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml markdownlint |
| uname | Linux 53862f04afc5 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19
23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3af5420d63a9686a863f12176dbebff165d90c4f |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=667806=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-667806
]
ASF GitHub Bot logged work on HDFS-16277:
-
Author: ASF GitHub Bot
Created on: 20/Oct/21 14:21
Start Date: 20/Oct/21 14:21
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3559:
URL: https://github.com/apache/hadoop/pull/3559#issuecomment-947716734
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 39s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 28s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 38s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 54s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/5/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 16 new + 230 unchanged
- 1 fixed = 246 total (was 231) |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 248m 43s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 344m 23s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestWriteReadStripedFile |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3559 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml markdownlint |
| uname | Linux 4dfed475fef5 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19
23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=667494=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-667494
]
ASF GitHub Bot logged work on HDFS-16277:
-
Author: ASF GitHub Bot
Created on: 20/Oct/21 07:55
Start Date: 20/Oct/21 07:55
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3559:
URL: https://github.com/apache/hadoop/pull/3559#issuecomment-947419371
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 14s | | trunk passed |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 18s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 3s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 6s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 10s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 10s | | the patch passed |
| -1 :x: | blanks | 0m 1s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/4/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 0m 54s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/4/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 47 new + 230 unchanged
- 1 fixed = 277 total (was 231) |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 57s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 229m 54s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 325m 1s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3559 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml markdownlint |
| uname | Linux c5fc82fb6975 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19
23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e91f9fd64fbf3e66986ea3836d61bddf752cbb06 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=667411=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-667411
]
ASF GitHub Bot logged work on HDFS-16277:
-
Author: ASF GitHub Bot
Created on: 20/Oct/21 02:06
Start Date: 20/Oct/21 02:06
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3559:
URL: https://github.com/apache/hadoop/pull/3559#issuecomment-947251645
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 15m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 9s | | trunk passed |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 20s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 29s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 43s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 1m 24s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/3/artifact/out/blanks-eol.txt)
| The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 0m 51s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/3/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 47 new + 230 unchanged
- 1 fixed = 277 total (was 231) |
| +1 :green_heart: | mvnsite | 1m 18s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 17s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 7s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 362m 5s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 485m 44s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3559 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml markdownlint |
| uname | Linux fb8930207713 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29
06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6ec0b22306fb2a8bcdf5d75ca086243c38c2261f |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=667185=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-667185 ] ASF GitHub Bot logged work on HDFS-16277: - Author: ASF GitHub Bot Created on: 19/Oct/21 18:03 Start Date: 19/Oct/21 18:03 Worklog Time Spent: 10m Work Description: GuoPhilipse commented on a change in pull request #3559: URL: https://github.com/apache/hadoop/pull/3559#discussion_r732119467 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java ## @@ -62,6 +63,11 @@ public void initialize(Configuration conf, FSClusterStats stats, + DFSConfigKeys.DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_PREFERENCE_FRACTION_KEY + " = " + balancedPreferencePercent); +balancedTolerate = +conf.getInt( + DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_TOLERATE_KEY, + DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_TOLERATE_DEFAULT); + Review comment: Thanks @ayushtkn，just have updated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 667185) Time Spent: 50m (was: 40m) > improve decision in AvailableSpaceBlockPlacementPolicy > -- > > Key: HDFS-16277 > URL: https://issues.apache.org/jira/browse/HDFS-16277 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Affects Versions: 3.3.1 >Reporter: guo >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Hi > In product environment,we may meet two or more datanode usage reaches nealy > 100%,for exampe 99.99%,98%,97%. > if we configure `AvailableSpaceBlockPlacementPolicy` , we also have the > chance to choose the 99.99%(assume it is the highest usage),for we treat the > two choosen datanode as the same usage if their storage usage are different > within 5%. > but this is not what we want, so i suggest we can improve the decision. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=667017=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-667017 ] ASF GitHub Bot logged work on HDFS-16277: - Author: ASF GitHub Bot Created on: 19/Oct/21 14:09 Start Date: 19/Oct/21 14:09 Worklog Time Spent: 10m Work Description: ayushtkn commented on a change in pull request #3559: URL: https://github.com/apache/hadoop/pull/3559#discussion_r731901083 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java ## @@ -1195,13 +1195,24 @@ "dfs.namenode.available-space-block-placement-policy.balanced-space-preference-fraction"; public static final float DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_PREFERENCE_FRACTION_DEFAULT = 0.6f; + public static final String DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_TOLERATE_KEY = + "dfs.namenode.available-space-block-placement-policy.balanced-space-tolerate"; + public static final int DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_TOLERATE_DEFAULT = + 5; public static final String DFS_NAMENODE_AVAILABLE_SPACE_RACK_FAULT_TOLERANT_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_PREFERENCE_FRACTION_KEY = "dfs.namenode.available-space-rack-fault-tolerant-block-placement-policy" + ".balanced-space-preference-fraction"; public static final float DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_RACK_FAULT_TOLERANT_PLACEMENT_POLICY_BALANCED_SPACE_PREFERENCE_FRACTION_DEFAULT = 0.6f; + public static final String + DFS_NAMENODE_AVAILABLE_SPACE_RACK_FAULT_TOLERANT_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_TOLERATE_KEY = + "dfs.namenode.available-space-rack-fault-tolerant-block-placement-policy" + + ".balanced-space-tolerate"; Review comment: Can you change tolerate to tolerance everywhere in the config names ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java ## @@ -37,6 +34,8 @@ import org.apache.hadoop.net.NetworkTopology; import org.apache.hadoop.net.Node; +import static org.apache.hadoop.hdfs.DFSConfigKeys.*; + Review comment: Don't club imports. Keep all imports separate and the existing ones untouched ## File path: hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsBlockPlacementPolicies.md ## @@ -119,6 +119,24 @@ The AvailableSpaceBlockPlacementPolicy is a space balanced block placement polic + +dfs.namenode.available-space-block-placement-policy.balanced-space-tolerate +5 + +Special value between 0 and 100, noninclusive. Increases tolerance of +placing blocks on Datanodes with similar disk space used. + + + + + dfs.namenode.available-space-block-placement-policy.datanode-usgae-tolerace-fraction + 0.6 + +Special value between 1 and 100, noninclusive. Increases chance of +placing blocks on Datanodes with less disk space used. + + Review comment: Didn't catch what is this? Some copy-paste error? ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java ## @@ -62,6 +63,11 @@ public void initialize(Configuration conf, FSClusterStats stats, + DFSConfigKeys.DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_PREFERENCE_FRACTION_KEY + " = " + balancedPreferencePercent); +balancedTolerate = +conf.getInt( + DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_TOLERATE_KEY, + DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_TOLERATE_DEFAULT); + Review comment: Rename this variable, current name doesn't make much sense. I think we should have some validation on this value, and an upper limit as well, May be at max 20,(document that as well) If the value is not in the valid range(0-20) add a warn log and use the default value. Same for the other class as well -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 667017) Time Spent: 40m (was: 0.5h) > improve decision in AvailableSpaceBlockPlacementPolicy > -- > > Key: HDFS-16277 > URL: https://issues.apache.org/jira/browse/HDFS-16277 >
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=666060=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-666060
]
ASF GitHub Bot logged work on HDFS-16277:
-
Author: ASF GitHub Bot
Created on: 17/Oct/21 17:06
Start Date: 17/Oct/21 17:06
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3559:
URL: https://github.com/apache/hadoop/pull/3559#issuecomment-945160089
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 25s | | trunk passed |
| +1 :green_heart: | compile | 1m 20s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 28s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 22s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 5s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 5s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 51s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 9 new + 229 unchanged
- 0 fixed = 238 total (was 229) |
| +1 :green_heart: | mvnsite | 1m 12s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 8s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 51s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 226m 50s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 320m 59s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3559 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml markdownlint |
| uname | Linux b9896daf0a1c 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19
23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0ed2dd8ca759253b2660284696ae140ed1aa94f1 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=666011=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-666011
]
ASF GitHub Bot logged work on HDFS-16277:
-
Author: ASF GitHub Bot
Created on: 17/Oct/21 05:42
Start Date: 17/Oct/21 05:42
Worklog Time Spent: 10m
Work Description: yakirgb commented on pull request #3559:
URL: https://github.com/apache/hadoop/pull/3559#issuecomment-945053647
Just an idea, i think also
`AvailableSpaceRackFaultTolerantBlockPlacementPolicy` with same issue and we
can apply same fix:
[AvailableSpaceRackFaultTolerantBlockPlacementPolicy.java](https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceRackFaultTolerantBlockPlacementPolicy.java#L121)
```
/**
* Compare the two data nodes.
*/
protected int compareDataNode(final DatanodeDescriptor a,
final DatanodeDescriptor b) {
if (a.equals(b)
|| Math.abs(a.getDfsUsedPercent() - b.getDfsUsedPercent()) < 5) {
return 0;
}
return a.getDfsUsedPercent() < b.getDfsUsedPercent() ? -1 : 1;
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 666011)
Time Spent: 20m (was: 10m)
> improve decision in AvailableSpaceBlockPlacementPolicy
> --
>
> Key: HDFS-16277
> URL: https://issues.apache.org/jira/browse/HDFS-16277
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: block placement
>Affects Versions: 3.3.1
>Reporter: guo
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Hi
> In product environment,we may meet two or more datanode usage reaches nealy
> 100%,for exampe 99.99%,98%,97%.
> if we configure `AvailableSpaceBlockPlacementPolicy` , we also have the
> chance to choose the 99.99%(assume it is the highest usage),for we treat the
> two choosen datanode as the same usage if their storage usage are different
> within 5%.
> but this is not what we want, so i suggest we can improve the decision.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16277) improve decision in AvailableSpaceBlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=665842=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-665842 ] ASF GitHub Bot logged work on HDFS-16277: - Author: ASF GitHub Bot Created on: 17/Oct/21 00:38 Start Date: 17/Oct/21 00:38 Worklog Time Spent: 10m Work Description: GuoPhilipse commented on pull request #3559: URL: https://github.com/apache/hadoop/pull/3559#issuecomment-945021829 @ayushtkn Thanks for you kindly review, will improve it later -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 665842) Remaining Estimate: 0h Time Spent: 10m > improve decision in AvailableSpaceBlockPlacementPolicy > -- > > Key: HDFS-16277 > URL: https://issues.apache.org/jira/browse/HDFS-16277 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Affects Versions: 3.3.1 >Reporter: guo >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Hi > In product environment,we may meet two or more datanode usage reaches nealy > 100%,for exampe 99.99%,98%,97%. > if we configure `AvailableSpaceBlockPlacementPolicy` , we also have the > chance to choose the 99.99%(assume it is the highest usage),for we treat the > two choosen datanode as the same usage if their storage usage are different > within 5%. > but this is not what we want, so i suggest we can improve the decision. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
