[GitHub] [hadoop] yakirgb commented on pull request #3559: HDFS-16277 improve decision in AvailableSpaceBlockPlacementPolicy
yakirgb commented on pull request #3559:
URL: https://github.com/apache/hadoop/pull/3559#issuecomment-945053647
Just an idea, i think also
`AvailableSpaceRackFaultTolerantBlockPlacementPolicy` with same issue and we
can apply same fix:
[AvailableSpaceRackFaultTolerantBlockPlacementPolicy.java](https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceRackFaultTolerantBlockPlacementPolicy.java#L121)
```
/**
* Compare the two data nodes.
*/
protected int compareDataNode(final DatanodeDescriptor a,
final DatanodeDescriptor b) {
if (a.equals(b)
|| Math.abs(a.getDfsUsedPercent() - b.getDfsUsedPercent()) < 5) {
return 0;
}
return a.getDfsUsedPercent() < b.getDfsUsedPercent() ? -1 : 1;
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17559) S3Guard import can OOM on large imports
[ https://issues.apache.org/jira/browse/HADOOP-17559?focusedWorklogId=666002=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-666002 ] ASF GitHub Bot logged work on HADOOP-17559: --- Author: ASF GitHub Bot Created on: 17/Oct/21 00:52 Start Date: 17/Oct/21 00:52 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2734: URL: https://github.com/apache/hadoop/pull/2734#issuecomment-880474863 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 666002) Time Spent: 1h 50m (was: 1h 40m) > S3Guard import can OOM on large imports > --- > > Key: HADOOP-17559 > URL: https://issues.apache.org/jira/browse/HADOOP-17559 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > I know I'm closing ~all S3Guard issues as wontfix, but this is pressing so > I'm going to do it anyway > S3guard import of directory tree containing many, many files will OOM. > Looking at the code this is going to be because > * import tool builds a map of all dirs imported, which as the comments note > "superfluous for DDB". - *cut* > * DDB AncestorState tracks files as well as dirs, purely as a safety check to > make sure current op doesn't somehow write a file entry above a dir entry in > the same operation > We've been running S3Guard for a long time, and condition #2 has never arisen. > Propose: don't store filenames there, so memory consumption goes from O(files > + dirs) to O(dirs) > Code straightforward, can't think of any tests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2734: HADOOP-17559. S3guard import OOM.
hadoop-yetus removed a comment on pull request #2734: URL: https://github.com/apache/hadoop/pull/2734#issuecomment-880474863 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16277) Expose getTokenKind method in FileSystem
[
https://issues.apache.org/jira/browse/HADOOP-16277?focusedWorklogId=665985=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-665985
]
ASF GitHub Bot logged work on HADOOP-16277:
---
Author: ASF GitHub Bot
Created on: 17/Oct/21 00:51
Start Date: 17/Oct/21 00:51
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3559:
URL: https://github.com/apache/hadoop/pull/3559#issuecomment-945003974
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 17s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 38s | | trunk passed |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 26s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 16s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 51s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 227m 49s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 333m 59s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3559 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 0ed47a568db3 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19
23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e71e324613b7e0268b927ae05b1df4882ebafd7d |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/1/testReport/ |
| Max. process+thread count | 3379 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/1/console |
| versions |
[GitHub] [hadoop] hadoop-yetus commented on pull request #3559: HADOOP-16277 improve decision in AvailableSpaceBlockPlacementPolicy
hadoop-yetus commented on pull request #3559:
URL: https://github.com/apache/hadoop/pull/3559#issuecomment-945003974
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 17s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 38s | | trunk passed |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 26s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 16s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 51s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 227m 49s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 333m 59s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3559 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 0ed47a568db3 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19
23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e71e324613b7e0268b927ae05b1df4882ebafd7d |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/1/testReport/ |
| Max. process+thread count | 3379 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3559/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about
[GitHub] [hadoop] akshatb1 opened a new pull request #3558: YARN-10978. Fixing ApplicationClassLoader to Correctly Expand Glob for Windows Path
akshatb1 opened a new pull request #3558: URL: https://github.com/apache/hadoop/pull/3558 ### Description of PR Using File.separator instead of "\" to make glob expansion OS agnostic. Fixes following bug: https://issues.apache.org/jira/browse/YARN-10978 ### How was this patch tested? Verified on a Windows test cluster. Auxiliary service starts correctly when the configuration is provided. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16277) Expose getTokenKind method in FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-16277?focusedWorklogId=665954=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-665954 ] ASF GitHub Bot logged work on HADOOP-16277: --- Author: ASF GitHub Bot Created on: 17/Oct/21 00:48 Start Date: 17/Oct/21 00:48 Worklog Time Spent: 10m Work Description: GuoPhilipse opened a new pull request #3559: URL: https://github.com/apache/hadoop/pull/3559 Hi In product environment,we may meet two or more datanode usage reaches nealy 100%,for exampe 99.99%,98%,97%. if we configure `AvailableSpaceBlockPlacementPolicy` , we also have the chance to choose the 99.99%(assume it is the highest usage),for we treat the two choosen datanode as the same usage if their storage usage are different within 5%. but this is not what we want, so i suggest we can improve the decision. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 665954) Remaining Estimate: 0h Time Spent: 10m > Expose getTokenKind method in FileSystem > > > Key: HADOOP-16277 > URL: https://issues.apache.org/jira/browse/HADOOP-16277 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Venkatesh Sridharan >Priority: Trivial > Time Spent: 10m > Remaining Estimate: 0h > > It would be nice to have a getTokenKind() method exposed in > [FileSystem|[https://github.com/apache/hadoop-common/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java]] > . Currently WebHdfsFileSystem class has getTokenKind() which is protected. > Having it in FileSystem makes it easier to use at runtime when the consumer > doesn't know what the underlying FileSystem implementation is. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16277) Expose getTokenKind method in FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-16277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-16277: Labels: pull-request-available (was: ) > Expose getTokenKind method in FileSystem > > > Key: HADOOP-16277 > URL: https://issues.apache.org/jira/browse/HADOOP-16277 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Venkatesh Sridharan >Priority: Trivial > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > It would be nice to have a getTokenKind() method exposed in > [FileSystem|[https://github.com/apache/hadoop-common/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java]] > . Currently WebHdfsFileSystem class has getTokenKind() which is protected. > Having it in FileSystem makes it easier to use at runtime when the consumer > doesn't know what the underlying FileSystem implementation is. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] GuoPhilipse opened a new pull request #3559: HADOOP-16277 improve decision in AvailableSpaceBlockPlacementPolicy
GuoPhilipse opened a new pull request #3559: URL: https://github.com/apache/hadoop/pull/3559 Hi In product environment,we may meet two or more datanode usage reaches nealy 100%,for exampe 99.99%,98%,97%. if we configure `AvailableSpaceBlockPlacementPolicy` , we also have the chance to choose the 99.99%(assume it is the highest usage),for we treat the two choosen datanode as the same usage if their storage usage are different within 5%. but this is not what we want, so i suggest we can improve the decision. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3548: HDFS-16272. Fix int overflow in computing safe length during EC block recovery
hadoop-yetus commented on pull request #3548:
URL: https://github.com/apache/hadoop/pull/3548#issuecomment-944273038
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 51s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 14s | | trunk passed |
| +1 :green_heart: | compile | 7m 25s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 6m 52s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 55s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 46s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 25s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 27m 11s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 8s | | the patch passed |
| +1 :green_heart: | compile | 5m 28s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 5m 28s | | the patch passed |
| +1 :green_heart: | compile | 5m 4s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 5m 4s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 7s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 4s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 46s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 5m 57s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 52s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 15s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 361m 34s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 511m 14s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3548/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3548 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux ece51c58113b 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 178404aede7a84a8ef8ba8dbe92cb3673a2482fa |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3548/4/testReport/ |
| Max. process+thread count | 1992 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client
hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3548/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log
[GitHub] [hadoop] hadoop-yetus commented on pull request #3551: YARN-10904. Investigate: Remove unnecessary fields from AbstractCSQueue or group fields by feature if possible
hadoop-yetus commented on pull request #3551: URL: https://github.com/apache/hadoop/pull/3551#issuecomment-944226087 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] szilard-nemeth opened a new pull request #3560: YARN-10958. Use correct configuration for Group service init in CSMappingPlacementRule
szilard-nemeth opened a new pull request #3560: URL: https://github.com/apache/hadoop/pull/3560 ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17242) S3A (async) ObjectListingIterator to block in hasNext() for results
[ https://issues.apache.org/jira/browse/HADOOP-17242?focusedWorklogId=665818=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-665818 ] ASF GitHub Bot logged work on HADOOP-17242: --- Author: ASF GitHub Bot Created on: 17/Oct/21 00:36 Start Date: 17/Oct/21 00:36 Worklog Time Spent: 10m Work Description: steveloughran closed pull request #2273: URL: https://github.com/apache/hadoop/pull/2273 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 665818) Time Spent: 1h 40m (was: 1.5h) > S3A (async) ObjectListingIterator to block in hasNext() for results > --- > > Key: HADOOP-17242 > URL: https://issues.apache.org/jira/browse/HADOOP-17242 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > HADOOP-17074 made listing async in S3A, but the iterator's hasNext Call > doesn't wait for the result. If invoked on an empty path it *may* return when > it should be failing. > Note: surfaced in code review, not seen in the wild and all our tests were > happy -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17967) Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom
[ https://issues.apache.org/jira/browse/HADOOP-17967?focusedWorklogId=665893=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-665893 ] ASF GitHub Bot logged work on HADOOP-17967: --- Author: ASF GitHub Bot Created on: 17/Oct/21 00:42 Start Date: 17/Oct/21 00:42 Worklog Time Spent: 10m Work Description: amahussein commented on pull request #3555: URL: https://github.com/apache/hadoop/pull/3555#issuecomment-944517593 Hi @virajjasani , Can we hold-on this change for a little bit? We are going to need the per-module enforcer to remove other Guava classes. At least the `Preconditions`. I cannot remember top of my head which other classes need to be incrementally replaced. Maybe we need to generate a list of all guava classes that are widely used across the modules. If we get that PR merged then we will reintroduce the same lines again very soon which is going to be redundant work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 665893) Time Spent: 0.5h (was: 20m) > Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in > hadoop-main pom > -- > > Key: HADOOP-17967 > URL: https://issues.apache.org/jira/browse/HADOOP-17967 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-16752) ABFS: test failure testLastModifiedTime()
[ https://issues.apache.org/jira/browse/HADOOP-16752?focusedWorklogId=665885=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-665885 ] ASF GitHub Bot logged work on HADOOP-16752: --- Author: ASF GitHub Bot Created on: 17/Oct/21 00:42 Start Date: 17/Oct/21 00:42 Worklog Time Spent: 10m Work Description: steveloughran closed pull request #3045: URL: https://github.com/apache/hadoop/pull/3045 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 665885) Time Spent: 1.5h (was: 1h 20m) > ABFS: test failure testLastModifiedTime() > - > > Key: HADOOP-16752 > URL: https://issues.apache.org/jira/browse/HADOOP-16752 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Da Zhou >Assignee: Sneha Vijayarajan >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > java.lang.AssertionError: lastModifiedTime should be after minCreateStartTime > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFileStatus.testLastModifiedTime(ITestAzureBlobFileSystemFileStatus.java:138) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] amahussein commented on pull request #3555: HADOOP-17967. Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom
amahussein commented on pull request #3555: URL: https://github.com/apache/hadoop/pull/3555#issuecomment-944517593 Hi @virajjasani , Can we hold-on this change for a little bit? We are going to need the per-module enforcer to remove other Guava classes. At least the `Preconditions`. I cannot remember top of my head which other classes need to be incrementally replaced. Maybe we need to generate a list of all guava classes that are widely used across the modules. If we get that PR merged then we will reintroduce the same lines again very soon which is going to be redundant work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #3045: HADOOP-16752. assertion failure in ITestAzureBlobFileSystemFileStatus
steveloughran closed pull request #3045: URL: https://github.com/apache/hadoop/pull/3045 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hemanthboyina merged pull request #3554: HDFS-16274. Improve error msg for FSNamesystem#startFileInt
hemanthboyina merged pull request #3554: URL: https://github.com/apache/hadoop/pull/3554 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17880) Build 2.10.x with docker
[ https://issues.apache.org/jira/browse/HADOOP-17880?focusedWorklogId=665855=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-665855 ] ASF GitHub Bot logged work on HADOOP-17880: --- Author: ASF GitHub Bot Created on: 17/Oct/21 00:39 Start Date: 17/Oct/21 00:39 Worklog Time Spent: 10m Work Description: GauthamBanasandra commented on a change in pull request #3535: URL: https://github.com/apache/hadoop/pull/3535#discussion_r730300925 ## File path: dev-support/docker/pkg-resolver/resolve.py ## @@ -0,0 +1,62 @@ +#!/usr/bin/env python3 + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Platform package dependency resolver for building Apache Hadoop. +""" + +from __future__ import print_function Review comment: Where is this package getting used? I'm not able to locate it here. ## File path: dev-support/docker/pkg-resolver/install-openssl.sh ## @@ -0,0 +1,30 @@ +#!/usr/bin/env bash Review comment: This file isn't being used anywhere. Please delete it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 665855) Time Spent: 9h (was: 8h 50m) > Build 2.10.x with docker > > > Key: HADOOP-17880 > URL: https://issues.apache.org/jira/browse/HADOOP-17880 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.10.0 > Environment: mac os x86_64 >Reporter: baizhendong >Priority: Major > Labels: pull-request-available > Time Spent: 9h > Remaining Estimate: 0h > > 1. currently, we build the hadoop 2.10.0 with docker machine, and must > install Virtual Box, and for hadoop 3.x, just build with docker only. > 2. besides this, the docker image dependency is out of date, and some of them > is unavaialble, for example – jdk7 > 3. but just building hadoop 2.10.0 with hadoop 3.x build script without > modification is not working, for the protocol buffer version is not 2.5.0, > and it's not work for native build. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #3554: HDFS-16274. Improve error msg for FSNamesystem#startFileInt
virajjasani commented on pull request #3554: URL: https://github.com/apache/hadoop/pull/3554#issuecomment-944616480 Belated +1 (non-binding) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] GauthamBanasandra commented on a change in pull request #3535: HADOOP-17880. Build 2.10.x with docker
GauthamBanasandra commented on a change in pull request #3535: URL: https://github.com/apache/hadoop/pull/3535#discussion_r730300925 ## File path: dev-support/docker/pkg-resolver/resolve.py ## @@ -0,0 +1,62 @@ +#!/usr/bin/env python3 + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Platform package dependency resolver for building Apache Hadoop. +""" + +from __future__ import print_function Review comment: Where is this package getting used? I'm not able to locate it here. ## File path: dev-support/docker/pkg-resolver/install-openssl.sh ## @@ -0,0 +1,30 @@ +#!/usr/bin/env bash Review comment: This file isn't being used anywhere. Please delete it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17947) Provide alternative to Guava VisibleForTesting
[ https://issues.apache.org/jira/browse/HADOOP-17947?focusedWorklogId=665849=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-665849 ] ASF GitHub Bot logged work on HADOOP-17947: --- Author: ASF GitHub Bot Created on: 17/Oct/21 00:38 Start Date: 17/Oct/21 00:38 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #3505: URL: https://github.com/apache/hadoop/pull/3505#issuecomment-932088478 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 665849) Time Spent: 6h 20m (was: 6h 10m) > Provide alternative to Guava VisibleForTesting > -- > > Key: HADOOP-17947 > URL: https://issues.apache.org/jira/browse/HADOOP-17947 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2, 3.2.4 > > Time Spent: 6h 20m > Remaining Estimate: 0h > > In an attempt to reduce the dependency on Guava, we should remove > VisibleForTesting annotation usages as it has very high usage in our > codebase. This Jira is to provide Hadoop's own alternative and use it in > hadoop-common-project modules. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17559) S3Guard import can OOM on large imports
[ https://issues.apache.org/jira/browse/HADOOP-17559?focusedWorklogId=665844=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-665844 ] ASF GitHub Bot logged work on HADOOP-17559: --- Author: ASF GitHub Bot Created on: 17/Oct/21 00:38 Start Date: 17/Oct/21 00:38 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2734: URL: https://github.com/apache/hadoop/pull/2734#issuecomment-93616 closing as unmerged; leaving the PR up -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 665844) Time Spent: 1h 40m (was: 1.5h) > S3Guard import can OOM on large imports > --- > > Key: HADOOP-17559 > URL: https://issues.apache.org/jira/browse/HADOOP-17559 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > I know I'm closing ~all S3Guard issues as wontfix, but this is pressing so > I'm going to do it anyway > S3guard import of directory tree containing many, many files will OOM. > Looking at the code this is going to be because > * import tool builds a map of all dirs imported, which as the comments note > "superfluous for DDB". - *cut* > * DDB AncestorState tracks files as well as dirs, purely as a safety check to > make sure current op doesn't somehow write a file entry above a dir entry in > the same operation > We've been running S3Guard for a long time, and condition #2 has never arisen. > Propose: don't store filenames there, so memory consumption goes from O(files > + dirs) to O(dirs) > Code straightforward, can't think of any tests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3505: HADOOP-17947. Provide alternative to Guava VisibleForTesting
hadoop-yetus removed a comment on pull request #3505: URL: https://github.com/apache/hadoop/pull/3505#issuecomment-932088478 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2734: HADOOP-17559. S3guard import OOM.
steveloughran commented on pull request #2734: URL: https://github.com/apache/hadoop/pull/2734#issuecomment-93616 closing as unmerged; leaving the PR up -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17967) Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom
[ https://issues.apache.org/jira/browse/HADOOP-17967?focusedWorklogId=665837=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-665837 ] ASF GitHub Bot logged work on HADOOP-17967: --- Author: ASF GitHub Bot Created on: 17/Oct/21 00:37 Start Date: 17/Oct/21 00:37 Worklog Time Spent: 10m Work Description: virajjasani commented on pull request #3555: URL: https://github.com/apache/hadoop/pull/3555#issuecomment-944614992 I am fine either way but I think we might have to make this change soon for sure, because not every single pom of all sub-modules are covered, only the ones where VFT is being used are covered. Hence, someone can still technically use Guava provided VFT annotation in any of those modules, it is only after this change we can restrict in every single module just because of the hadoop-main pom, which is directly or indirectly parent pom for every module/sub-module. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 665837) Time Spent: 20m (was: 10m) > Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in > hadoop-main pom > -- > > Key: HADOOP-17967 > URL: https://issues.apache.org/jira/browse/HADOOP-17967 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] GuoPhilipse commented on pull request #3559: HDFS-16277 improve decision in AvailableSpaceBlockPlacementPolicy
GuoPhilipse commented on pull request #3559: URL: https://github.com/apache/hadoop/pull/3559#issuecomment-945021829 @ayushtkn Thanks for you kindly review, will improve it later -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17953) S3A: ITestS3AFileContextStatistics test to lookup global or per-bucket configuration for encryption algorithm
[
https://issues.apache.org/jira/browse/HADOOP-17953?focusedWorklogId=665831=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-665831
]
ASF GitHub Bot logged work on HADOOP-17953:
---
Author: ASF GitHub Bot
Created on: 17/Oct/21 00:37
Start Date: 17/Oct/21 00:37
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#3525:
URL: https://github.com/apache/hadoop/pull/3525#discussion_r729975058
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/ITestSessionDelegationInFileystem.java
##
@@ -145,9 +145,13 @@ protected Configuration createConfiguration() {
// disable if assume role opts are off
assumeSessionTestsEnabled(conf);
disableFilesystemCaching(conf);
-String s3EncryptionMethod =
-conf.getTrimmed(Constants.S3_ENCRYPTION_ALGORITHM,
-S3AEncryptionMethods.SSE_KMS.getMethod());
+String s3EncryptionMethod;
+try {
+ s3EncryptionMethod =
+ getEncryptionAlgorithm(getTestBucketName(conf), conf).getMethod();
+} catch (IOException e) {
+ throw new RuntimeException("Failed to lookup encryption algorithm.");
Review comment:
nit: should throw UncheckedIOException, as it's a good habit to get into
-certainly I'd expect in production code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 665831)
Time Spent: 40m (was: 0.5h)
> S3A: ITestS3AFileContextStatistics test to lookup global or per-bucket
> configuration for encryption algorithm
> -
>
> Key: HADOOP-17953
> URL: https://issues.apache.org/jira/browse/HADOOP-17953
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Mehakmeet Singh
>Assignee: Mehakmeet Singh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> ITestS3AFileContextStatistics uses the conf.get() method to get the global
> configuration for encryption algorithm and keys, but the per-bucket
> configuration would be ignored in this case.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3558: YARN-10978. Fixing ApplicationClassLoader to Correctly Expand Glob for Windows Path
hadoop-yetus commented on pull request #3558:
URL: https://github.com/apache/hadoop/pull/3558#issuecomment-944977116
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 16m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 22s | | trunk passed |
| +1 :green_heart: | compile | 23m 45s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 20m 26s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 44s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 32s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 1s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 57s | | the patch passed |
| +1 :green_heart: | compile | 23m 5s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 23m 5s | | the patch passed |
| +1 :green_heart: | compile | 20m 33s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 20m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 59s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 11s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 34s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 6s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not
generate ASF License warnings. |
| | | 216m 42s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3558/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3558 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 1aa379bed76f 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17
17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8521737769361e0f7fb417379a5470b11e482880 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3558/1/testReport/ |
| Max. process+thread count | 1268 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3558/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For
[GitHub] [hadoop] virajjasani commented on pull request #3555: HADOOP-17967. Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom
virajjasani commented on pull request #3555: URL: https://github.com/apache/hadoop/pull/3555#issuecomment-944614992 I am fine either way but I think we might have to make this change soon for sure, because not every single pom of all sub-modules are covered, only the ones where VFT is being used are covered. Hence, someone can still technically use Guava provided VFT annotation in any of those modules, it is only after this change we can restrict in every single module just because of the hadoop-main pom, which is directly or indirectly parent pom for every module/sub-module. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #3525: HADOOP-17953. S3A: Tests to lookup global or per-bucket configuration for encryption algorithm
steveloughran commented on a change in pull request #3525:
URL: https://github.com/apache/hadoop/pull/3525#discussion_r729975058
##
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/ITestSessionDelegationInFileystem.java
##
@@ -145,9 +145,13 @@ protected Configuration createConfiguration() {
// disable if assume role opts are off
assumeSessionTestsEnabled(conf);
disableFilesystemCaching(conf);
-String s3EncryptionMethod =
-conf.getTrimmed(Constants.S3_ENCRYPTION_ALGORITHM,
-S3AEncryptionMethods.SSE_KMS.getMethod());
+String s3EncryptionMethod;
+try {
+ s3EncryptionMethod =
+ getEncryptionAlgorithm(getTestBucketName(conf), conf).getMethod();
+} catch (IOException e) {
+ throw new RuntimeException("Failed to lookup encryption algorithm.");
Review comment:
nit: should throw UncheckedIOException, as it's a good habit to get into
-certainly I'd expect in production code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hemanthboyina commented on pull request #3554: HDFS-16274. Improve error msg for FSNamesystem#startFileInt
hemanthboyina commented on pull request #3554: URL: https://github.com/apache/hadoop/pull/3554#issuecomment-944462036 +1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17559) S3Guard import can OOM on large imports
[ https://issues.apache.org/jira/browse/HADOOP-17559?focusedWorklogId=665814=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-665814 ] ASF GitHub Bot logged work on HADOOP-17559: --- Author: ASF GitHub Bot Created on: 17/Oct/21 00:35 Start Date: 17/Oct/21 00:35 Worklog Time Spent: 10m Work Description: steveloughran closed pull request #2734: URL: https://github.com/apache/hadoop/pull/2734 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 665814) Time Spent: 1.5h (was: 1h 20m) > S3Guard import can OOM on large imports > --- > > Key: HADOOP-17559 > URL: https://issues.apache.org/jira/browse/HADOOP-17559 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > I know I'm closing ~all S3Guard issues as wontfix, but this is pressing so > I'm going to do it anyway > S3guard import of directory tree containing many, many files will OOM. > Looking at the code this is going to be because > * import tool builds a map of all dirs imported, which as the comments note > "superfluous for DDB". - *cut* > * DDB AncestorState tracks files as well as dirs, purely as a safety check to > make sure current op doesn't somehow write a file entry above a dir entry in > the same operation > We've been running S3Guard for a long time, and condition #2 has never arisen. > Propose: don't store filenames there, so memory consumption goes from O(files > + dirs) to O(dirs) > Code straightforward, can't think of any tests -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #2273: HADOOP-17242. S3A ObjectListingIterator to block in hasNext() for results
steveloughran closed pull request #2273: URL: https://github.com/apache/hadoop/pull/2273 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #2734: HADOOP-17559. S3guard import OOM.
steveloughran closed pull request #2734: URL: https://github.com/apache/hadoop/pull/2734 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3556: HDFS-16275. Enable considerLoad for localWrite
hadoop-yetus commented on pull request #3556:
URL: https://github.com/apache/hadoop/pull/3556#issuecomment-944418982
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 14s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 14s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 24s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 7s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 20s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 0s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 327m 45s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 431m 6s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3556/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3556 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 716fe136357d 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fef6a535dceee468ac8b79603e3ce0ac8701b7eb |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3556/1/testReport/ |
| Max. process+thread count | 1983 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3556/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3560: YARN-10958. Use correct configuration for Group service init in CSMappingPlacementRule
hadoop-yetus commented on pull request #3560:
URL: https://github.com/apache/hadoop/pull/3560#issuecomment-945013282
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 19m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 6s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 19s | | trunk passed |
| +1 :green_heart: | compile | 29m 30s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 24m 43s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 13s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 18s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 44s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 5m 32s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 15s | | the patch passed |
| +1 :green_heart: | compile | 28m 28s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 28m 28s | | the patch passed |
| +1 :green_heart: | compile | 24m 34s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 24m 34s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3560/1/artifact/out/blanks-eol.txt)
| The patch has 10 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 28s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3560/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 9 new + 12 unchanged - 0 fixed = 21 total (was
12) |
| +1 :green_heart: | mvnsite | 3m 9s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 16s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 45s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 5m 59s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 19s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 55s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 110m 25s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 1m 0s | | The patch does not
generate ASF License warnings. |
| | | 390m 34s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3560/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3560 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux ef9befb88768 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4079229698c3f494b05c817230c30d5d482989fe |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3560/1/testReport/ |
| Max. process+thread count | 3137 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common
[GitHub] [hadoop] tomscut commented on pull request #3554: HDFS-16274. Improve error msg for FSNamesystem#startFileInt
tomscut commented on pull request #3554: URL: https://github.com/apache/hadoop/pull/3554#issuecomment-944639170 Thanks @hemanthboyina @virajjasani for your review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17201) Spark job with s3acommitter stuck at the last stage
[
https://issues.apache.org/jira/browse/HADOOP-17201?focusedWorklogId=665771=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-665771
]
ASF GitHub Bot logged work on HADOOP-17201:
---
Author: ASF GitHub Bot
Created on: 17/Oct/21 00:31
Start Date: 17/Oct/21 00:31
Worklog Time Spent: 10m
Work Description: steveloughran closed pull request #2402:
URL: https://github.com/apache/hadoop/pull/2402
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 665771)
Time Spent: 0.5h (was: 20m)
> Spark job with s3acommitter stuck at the last stage
> ---
>
> Key: HADOOP-17201
> URL: https://issues.apache.org/jira/browse/HADOOP-17201
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.2.1
> Environment: we are on spark 2.4.5/hadoop 3.2.1 with s3a committer.
> spark.hadoop.fs.s3a.committer.magic.enabled: 'true'
> spark.hadoop.fs.s3a.committer.name: magic
>Reporter: Dyno
>Priority: Major
> Labels: pull-request-available
> Attachments: exec-120.log, exec-125.log, exec-25.log, exec-31.log,
> exec-36.log, exec-44.log, exec-5.log, exec-64.log, exec-7.log
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> usually our spark job took 1 hour or 2 to finish, occasionally it runs for
> more than 3 hour and then we know it's stuck and usually the executor has
> stack like this
> {{
> "Executor task launch worker for task 78620" #265 daemon prio=5 os_prio=0
> tid=0x7f73e0005000 nid=0x12d waiting on condition [0x7f74cb291000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:349)
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:285)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObjects(S3AFileSystem.java:1457)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:1717)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.deleteUnnecessaryFakeDirectories(S3AFileSystem.java:2785)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.finishedWrite(S3AFileSystem.java:2751)
> at
> org.apache.hadoop.fs.s3a.WriteOperationHelper.lambda$finalizeMultipartUpload$1(WriteOperationHelper.java:238)
> at
> org.apache.hadoop.fs.s3a.WriteOperationHelper$$Lambda$210/1059071691.execute(Unknown
> Source)
> at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
> at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:265)
> at
> org.apache.hadoop.fs.s3a.Invoker$$Lambda$23/586859139.execute(Unknown Source)
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:261)
> at
> org.apache.hadoop.fs.s3a.WriteOperationHelper.finalizeMultipartUpload(WriteOperationHelper.java:226)
> at
> org.apache.hadoop.fs.s3a.WriteOperationHelper.completeMPUwithRetries(WriteOperationHelper.java:271)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload.complete(S3ABlockOutputStream.java:660)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload.access$200(S3ABlockOutputStream.java:521)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.close(S3ABlockOutputStream.java:385)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> at
> org.apache.parquet.hadoop.util.HadoopPositionOutputStream.close(HadoopPositionOutputStream.java:64)
> at
> org.apache.parquet.hadoop.ParquetFileWriter.end(ParquetFileWriter.java:685)
> at
> org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:122)
> at
> org.apache.parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:165)
> at
> org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.close(ParquetOutputWriter.scala:42)
> at
> org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseResources(FileFormatDataWriter.scala:57)
> at
> org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:74)
> at
>
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #1747: HDFS-15042 Add more tests for ByteBufferPositionedReadable.
hadoop-yetus removed a comment on pull request #1747: URL: https://github.com/apache/hadoop/pull/1747#issuecomment-734970323 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #2402: HADOOP-17201. eliminate double retry around delete/finishedWrite
steveloughran closed pull request #2402: URL: https://github.com/apache/hadoop/pull/2402 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17967) Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom
[ https://issues.apache.org/jira/browse/HADOOP-17967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-17967: Labels: pull-request-available (was: ) > Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in > hadoop-main pom > -- > > Key: HADOOP-17967 > URL: https://issues.apache.org/jira/browse/HADOOP-17967 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17967) Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom
[ https://issues.apache.org/jira/browse/HADOOP-17967?focusedWorklogId=665760=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-665760 ] ASF GitHub Bot logged work on HADOOP-17967: --- Author: ASF GitHub Bot Created on: 17/Oct/21 00:30 Start Date: 17/Oct/21 00:30 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #3555: URL: https://github.com/apache/hadoop/pull/3555#issuecomment-944319975 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 665760) Remaining Estimate: 0h Time Spent: 10m > Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in > hadoop-main pom > -- > > Key: HADOOP-17967 > URL: https://issues.apache.org/jira/browse/HADOOP-17967 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3555: HADOOP-17967. Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom
hadoop-yetus commented on pull request #3555: URL: https://github.com/apache/hadoop/pull/3555#issuecomment-944319975 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
