[jira] [Commented] (HADOOP-18816) Rebuild Exceptions on Client side to get genuine exceptions
[
https://issues.apache.org/jira/browse/HADOOP-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17750106#comment-17750106
]
ASF GitHub Bot commented on HADOOP-18816:
-
hadoop-yetus commented on PR #5863:
URL: https://github.com/apache/hadoop/pull/5863#issuecomment-1661528034
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 49s | | trunk passed |
| +1 :green_heart: | compile | 17m 24s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 16m 20s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 48s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 25s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 0s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 4s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 36m 32s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 57s | | the patch passed |
| +1 :green_heart: | compile | 16m 32s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | cc | 16m 32s | | the patch passed |
| +1 :green_heart: | javac | 16m 32s | | the patch passed |
| +1 :green_heart: | compile | 16m 21s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | cc | 16m 21s | | the patch passed |
| +1 :green_heart: | javac | 16m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 21s | |
hadoop-common-project/hadoop-common: The patch generated 0 new + 249 unchanged
- 6 fixed = 249 total (was 255) |
| +1 :green_heart: | mvnsite | 1m 49s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 20s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 49s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 53s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 15s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 14s | | The patch does not
generate ASF License warnings. |
| | | 224m 48s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5863/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5863 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint
bufcompat |
| uname | Linux 5421522b5377 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b6b0b7f75f07c82c3985b47d3b91b60a485a67d9 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
[GitHub] [hadoop] hadoop-yetus commented on pull request #5863: HADOOP-18816. Rebuild exceptions on Client side
hadoop-yetus commented on PR #5863:
URL: https://github.com/apache/hadoop/pull/5863#issuecomment-1661528034
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 49s | | trunk passed |
| +1 :green_heart: | compile | 17m 24s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 16m 20s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 48s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 25s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 0s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 4s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 36m 32s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 57s | | the patch passed |
| +1 :green_heart: | compile | 16m 32s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | cc | 16m 32s | | the patch passed |
| +1 :green_heart: | javac | 16m 32s | | the patch passed |
| +1 :green_heart: | compile | 16m 21s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | cc | 16m 21s | | the patch passed |
| +1 :green_heart: | javac | 16m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 21s | |
hadoop-common-project/hadoop-common: The patch generated 0 new + 249 unchanged
- 6 fixed = 249 total (was 255) |
| +1 :green_heart: | mvnsite | 1m 49s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 20s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 49s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 53s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 15s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 14s | | The patch does not
generate ASF License warnings. |
| | | 224m 48s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5863/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5863 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint
bufcompat |
| uname | Linux 5421522b5377 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b6b0b7f75f07c82c3985b47d3b91b60a485a67d9 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5863/9/testReport/ |
| Max. process+thread count | 3149 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
[GitHub] [hadoop] zhangshuyan0 commented on pull request #5855: HDFS-17093. In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incomplete bloc
zhangshuyan0 commented on PR #5855: URL: https://github.com/apache/hadoop/pull/5855#issuecomment-1661459596 > Two of the cases will come into this logic > > 1. The namenode is restarting while receiving FBRS from all datanodes and is in safe mode > 2. When the namenode is in secure mode for some reason while it has been running for a long time >In the first case, if the datanode has a failed disk, the datanode will send the FBR for the normal disk and the namenode will handle it normally >In the second case, blockReportCount == 0 will always be false if no new disks are added to the datanode >So I recommend keeping the code as it is and not using blockReportCount == 0 @Tre2878 If a disk is failed, its' state will be set to `FAILED` and it will be removed from `storageMap`. So the check of `blockReportCount` will not involve failed storages, and `blockReportCount == 0` is not always be false. Thus, I think my plan can work here. What's your opinion? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Tre2878 commented on a diff in pull request #5855: HDFS-17093. In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incomplete
Tre2878 commented on code in PR #5855: URL: https://github.com/apache/hadoop/pull/5855#discussion_r1281302320 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java: ## @@ -1650,7 +1650,7 @@ public DatanodeCommand blockReport(final DatanodeRegistration nodeReg, final int index = r; noStaleStorages = bm.runBlockOp(() -> bm.processReport(nodeReg, reports[index].getStorage(), -blocks, context)); +blocks, context, reports.length, index + 1)); Review Comment: Thanks for reminding me. I think we'll have to think of something else -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18829) s3a prefetch LRU cache eviction metric
[ https://issues.apache.org/jira/browse/HADOOP-18829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17750036#comment-17750036 ] ASF GitHub Bot commented on HADOOP-18829: - virajjasani commented on PR #5893: URL: https://github.com/apache/hadoop/pull/5893#issuecomment-1661213292 this (and other future follow-ups from HADOOP-18291) can go after #5832 > s3a prefetch LRU cache eviction metric > -- > > Key: HADOOP-18829 > URL: https://issues.apache.org/jira/browse/HADOOP-18829 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > Follow-up from HADOOP-18291: > Add new IO statistics metric to capture s3a prefetch LRU cache eviction. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5893: HADOOP-18829. S3A prefetch LRU cache eviction metric
virajjasani commented on PR #5893: URL: https://github.com/apache/hadoop/pull/5893#issuecomment-1661213292 this (and other future follow-ups from HADOOP-18291) can go after #5832 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18829) s3a prefetch LRU cache eviction metric
[ https://issues.apache.org/jira/browse/HADOOP-18829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17750035#comment-17750035 ] Viraj Jasani commented on HADOOP-18829: --- sure thing, i think this can wait for sure. thanks > s3a prefetch LRU cache eviction metric > -- > > Key: HADOOP-18829 > URL: https://issues.apache.org/jira/browse/HADOOP-18829 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > Follow-up from HADOOP-18291: > Add new IO statistics metric to capture s3a prefetch LRU cache eviction. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18184) s3a prefetching stream to support unbuffer()
[
https://issues.apache.org/jira/browse/HADOOP-18184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17750029#comment-17750029
]
ASF GitHub Bot commented on HADOOP-18184:
-
hadoop-yetus commented on PR #5832:
URL: https://github.com/apache/hadoop/pull/5832#issuecomment-1661161006
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 22 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 5s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 18s | | trunk passed |
| +1 :green_heart: | compile | 11m 7s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 10m 13s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 2m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 54s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 21s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 51s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 59s | | the patch passed |
| +1 :green_heart: | compile | 9m 51s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 9m 51s | | the patch passed |
| +1 :green_heart: | compile | 9m 37s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 9m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 2m 17s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5832/7/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 24 new + 8 unchanged - 0 fixed = 32 total (was 8)
|
| +1 :green_heart: | mvnsite | 1m 52s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 47s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 16m 43s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 2m 19s |
[/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5832/7/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not
generate ASF License warnings. |
| | | 167m 8s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.fs.s3a.prefetch.TestS3ACachingBlockManager |
| | hadoop.fs.s3a.prefetch.TestS3ARemoteInputStream |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5832/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5832 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 91de6937f614 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c0e4f1cd5ab715d087a7d0d67f27730f024ab19e |
| Default Java | Private
[GitHub] [hadoop] hadoop-yetus commented on pull request #5832: HADOOP-18184. S3A Prefetching unbuffer.
hadoop-yetus commented on PR #5832:
URL: https://github.com/apache/hadoop/pull/5832#issuecomment-1661161006
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 22 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 5s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 18s | | trunk passed |
| +1 :green_heart: | compile | 11m 7s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 10m 13s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 2m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 54s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 21s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 51s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 59s | | the patch passed |
| +1 :green_heart: | compile | 9m 51s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 9m 51s | | the patch passed |
| +1 :green_heart: | compile | 9m 37s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 9m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 2m 17s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5832/7/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 24 new + 8 unchanged - 0 fixed = 32 total (was 8)
|
| +1 :green_heart: | mvnsite | 1m 52s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 47s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 16m 43s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 2m 19s |
[/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5832/7/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not
generate ASF License warnings. |
| | | 167m 8s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.fs.s3a.prefetch.TestS3ACachingBlockManager |
| | hadoop.fs.s3a.prefetch.TestS3ARemoteInputStream |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5832/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5832 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 91de6937f614 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c0e4f1cd5ab715d087a7d0d67f27730f024ab19e |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
|
[GitHub] [hadoop] hadoop-yetus commented on pull request #5884: YARN-11535: Snakeyaml should be excluded from jackson-dataformat-yaml-2.12.7 as it may cause transitive dependency issue.
hadoop-yetus commented on PR #5884:
URL: https://github.com/apache/hadoop/pull/5884#issuecomment-1661026258
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 28s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 33s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 24s | | trunk passed |
| +1 :green_heart: | compile | 10m 8s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 9m 35s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | mvnsite | 1m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 26s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 21s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | shadedclient | 80m 7s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 41s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 46s | | the patch passed |
| +1 :green_heart: | compile | 9m 29s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 9m 29s | | the patch passed |
| +1 :green_heart: | compile | 9m 28s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 9m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 1m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 22s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | shadedclient | 27m 20s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 32s | | hadoop-project in the patch
passed. |
| -1 :x: | unit | 84m 37s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5884/2/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 52s | | The patch does not
generate ASF License warnings. |
| | | 217m 19s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher |
| | hadoop.yarn.server.resourcemanager.TestRMRestart |
| |
hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisherForV2 |
| | hadoop.yarn.server.resourcemanager.TestResourceTrackerService |
| | hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher |
| | hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5884/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5884 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint |
| uname | Linux 283fec32d995 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 726e0fc92587b63bbc91107f5b52b18ffe79187b |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
|
[jira] [Commented] (HADOOP-18829) s3a prefetch LRU cache eviction metric
[ https://issues.apache.org/jira/browse/HADOOP-18829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749996#comment-17749996 ] Steve Loughran commented on HADOOP-18829: - can i get my /HADOOP-18184 unbuffer patch in before anything else? i've already had to rebase it a couple of times. > s3a prefetch LRU cache eviction metric > -- > > Key: HADOOP-18829 > URL: https://issues.apache.org/jira/browse/HADOOP-18829 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > Follow-up from HADOOP-18291: > Add new IO statistics metric to capture s3a prefetch LRU cache eviction. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18836) Some properties are missing from hadoop-policy.xml.
[
https://issues.apache.org/jira/browse/HADOOP-18836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-18836:
Component/s: security
> Some properties are missing from hadoop-policy.xml.
> ---
>
> Key: HADOOP-18836
> URL: https://issues.apache.org/jira/browse/HADOOP-18836
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, documentation, security
>Reporter: WangYuanben
>Assignee: WangYuanben
>Priority: Major
>
>
> {noformat}
> security.datanode.lifeline.protocol.acl
> security.get.user.mappings.protocol.acl
> security.reconfiguration.protocol.acl
> security.refresh.callqueue.protocol.acl
> security.refresh.generic.protocol.acl{noformat}
> The properties above are missing from hadoop-policy.xml.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18184) s3a prefetching stream to support unbuffer()
[ https://issues.apache.org/jira/browse/HADOOP-18184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749994#comment-17749994 ] ASF GitHub Bot commented on HADOOP-18184: - steveloughran commented on PR #5832: URL: https://github.com/apache/hadoop/pull/5832#issuecomment-1660936697 updated patch. the test that caching is failing because files aren't being added to the buffer dir. theory: they are going somewhere else. trivia; rebased patch wouldn't push to the repo until I updated my github oauth tokens; see https://docs.github.com/en/get-started/getting-started-with-git/caching-your-github-credentials-in-git > s3a prefetching stream to support unbuffer() > > > Key: HADOOP-18184 > URL: https://issues.apache.org/jira/browse/HADOOP-18184 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: pull-request-available > > Apache Impala uses unbuffer() to free up all client side resources held by a > stream, so allowing it to have a map of available (path -> stream) objects, > retained across queries. > This saves on having to reopen the files, with the cost of HEAD checks etc. > S3AInputStream just closes its http connection. here there is a lot more > state to discard, but all memory and file storage must be freed. > until this done, ITestS3AContractUnbuffer must skip when the prefetch stream > is used. > its notable that the other tests don't fail, even though the stream doesn't > implement the interface; the graceful degradation handles that. it should > fail if the test xml resource says the stream does it, but that the stream > capabilities say it doesn't. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5832: HADOOP-18184. S3A Prefetching unbuffer.
steveloughran commented on PR #5832: URL: https://github.com/apache/hadoop/pull/5832#issuecomment-1660936697 updated patch. the test that caching is failing because files aren't being added to the buffer dir. theory: they are going somewhere else. trivia; rebased patch wouldn't push to the repo until I updated my github oauth tokens; see https://docs.github.com/en/get-started/getting-started-with-git/caching-your-github-credentials-in-git -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18832) Upgrade aws-java-sdk to 1.12.499+
[ https://issues.apache.org/jira/browse/HADOOP-18832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749985#comment-17749985 ] ASF GitHub Bot commented on HADOOP-18832: - virajjasani commented on PR #5908: URL: https://github.com/apache/hadoop/pull/5908#issuecomment-1660890327 load tests are passing: ``` org.apache.hadoop.fs.s3a.scale.ILoadTestS3ABulkDeleteThrottling org.apache.hadoop.fs.s3a.auth.delegation.ILoadTestSessionCredentials org.apache.hadoop.fs.s3a.auth.delegation.ILoadTestRoleCredentials ``` > Upgrade aws-java-sdk to 1.12.499+ > - > > Key: HADOOP-18832 > URL: https://issues.apache.org/jira/browse/HADOOP-18832 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > aws sdk versions < 1.12.499 uses a vulnerable version of netty and hence > showing up in security CVE scans (CVE-2023-34462). The safe version for netty > is 4.1.94.Final and this is used by aws-java-sdk:1.12.499+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5908: HADOOP-18832. Upgrade aws-java-sdk to 1.12.499
virajjasani commented on PR #5908: URL: https://github.com/apache/hadoop/pull/5908#issuecomment-1660890327 load tests are passing: ``` org.apache.hadoop.fs.s3a.scale.ILoadTestS3ABulkDeleteThrottling org.apache.hadoop.fs.s3a.auth.delegation.ILoadTestSessionCredentials org.apache.hadoop.fs.s3a.auth.delegation.ILoadTestRoleCredentials ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18184) s3a prefetching stream to support unbuffer()
[
https://issues.apache.org/jira/browse/HADOOP-18184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749924#comment-17749924
]
ASF GitHub Bot commented on HADOOP-18184:
-
steveloughran commented on code in PR #5832:
URL: https://github.com/apache/hadoop/pull/5832#discussion_r1280864279
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/prefetch/S3ARemoteInputStream.java:
##
@@ -339,49 +367,67 @@ public int read(byte[] buffer, int offset, int len)
throws IOException {
numBytesRemaining -= bytesToRead;
numBytesRead += bytesToRead;
}
-
return numBytesRead;
}
- protected S3ARemoteObject getFile() {
+ /**
+ * Forward to superclass after updating the read fully IOStatistics
+ * {@inheritDoc}
+ */
+ @Override
+ public void readFully(final long position,
+ final byte[] buffer,
+ final int offset,
+ final int length) throws IOException {
+throwIfClosed();
+validatePositionedReadArgs(position, buffer, offset, length);
+streamStatistics.readFullyOperationStarted(position, length);
+super.readFully(position, buffer, offset, length);
+ }
+
+ protected final S3ARemoteObject getRemoteObject() {
return remoteObject;
}
- protected S3ARemoteObjectReader getReader() {
+ protected final S3ARemoteObjectReader getReader() {
return reader;
}
- protected S3ObjectAttributes getS3ObjectAttributes() {
+ protected final S3ObjectAttributes getS3ObjectAttributes() {
return s3Attributes;
}
- protected FilePosition getFilePosition() {
+ protected final FilePosition getFilePosition() {
return fpos;
}
- protected String getName() {
+ protected final String getName() {
return name;
}
- protected boolean isClosed() {
-return closed;
+ protected final boolean isClosed() {
+return closed.get();
+ }
+
+ protected final boolean underlyingResourcesClosed() {
+return underlyingResourcesClosed.get();
}
- protected long getNextReadPos() {
+ protected final long getNextReadPos() {
return nextReadPos;
}
- protected BlockData getBlockData() {
+ protected final BlockData getBlockData() {
return blockData;
}
- protected S3AReadOpContext getContext() {
+ protected final S3AReadOpContext getContext() {
return context;
}
private void incrementBytesRead(int bytesRead) {
if (bytesRead > 0) {
- streamStatistics.bytesRead(bytesRead);
+ streamStatistics.bytesReadFromBuffer(bytesRead);
Review Comment:
trying to differentiate bytes we read remotely vs bytes which were returned
from cached data.
> s3a prefetching stream to support unbuffer()
>
>
> Key: HADOOP-18184
> URL: https://issues.apache.org/jira/browse/HADOOP-18184
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Labels: pull-request-available
>
> Apache Impala uses unbuffer() to free up all client side resources held by a
> stream, so allowing it to have a map of available (path -> stream) objects,
> retained across queries.
> This saves on having to reopen the files, with the cost of HEAD checks etc.
> S3AInputStream just closes its http connection. here there is a lot more
> state to discard, but all memory and file storage must be freed.
> until this done, ITestS3AContractUnbuffer must skip when the prefetch stream
> is used.
> its notable that the other tests don't fail, even though the stream doesn't
> implement the interface; the graceful degradation handles that. it should
> fail if the test xml resource says the stream does it, but that the stream
> capabilities say it doesn't.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5832: HADOOP-18184. S3A Prefetching unbuffer.
steveloughran commented on code in PR #5832:
URL: https://github.com/apache/hadoop/pull/5832#discussion_r1280864279
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/prefetch/S3ARemoteInputStream.java:
##
@@ -339,49 +367,67 @@ public int read(byte[] buffer, int offset, int len)
throws IOException {
numBytesRemaining -= bytesToRead;
numBytesRead += bytesToRead;
}
-
return numBytesRead;
}
- protected S3ARemoteObject getFile() {
+ /**
+ * Forward to superclass after updating the read fully IOStatistics
+ * {@inheritDoc}
+ */
+ @Override
+ public void readFully(final long position,
+ final byte[] buffer,
+ final int offset,
+ final int length) throws IOException {
+throwIfClosed();
+validatePositionedReadArgs(position, buffer, offset, length);
+streamStatistics.readFullyOperationStarted(position, length);
+super.readFully(position, buffer, offset, length);
+ }
+
+ protected final S3ARemoteObject getRemoteObject() {
return remoteObject;
}
- protected S3ARemoteObjectReader getReader() {
+ protected final S3ARemoteObjectReader getReader() {
return reader;
}
- protected S3ObjectAttributes getS3ObjectAttributes() {
+ protected final S3ObjectAttributes getS3ObjectAttributes() {
return s3Attributes;
}
- protected FilePosition getFilePosition() {
+ protected final FilePosition getFilePosition() {
return fpos;
}
- protected String getName() {
+ protected final String getName() {
return name;
}
- protected boolean isClosed() {
-return closed;
+ protected final boolean isClosed() {
+return closed.get();
+ }
+
+ protected final boolean underlyingResourcesClosed() {
+return underlyingResourcesClosed.get();
}
- protected long getNextReadPos() {
+ protected final long getNextReadPos() {
return nextReadPos;
}
- protected BlockData getBlockData() {
+ protected final BlockData getBlockData() {
return blockData;
}
- protected S3AReadOpContext getContext() {
+ protected final S3AReadOpContext getContext() {
return context;
}
private void incrementBytesRead(int bytesRead) {
if (bytesRead > 0) {
- streamStatistics.bytesRead(bytesRead);
+ streamStatistics.bytesReadFromBuffer(bytesRead);
Review Comment:
trying to differentiate bytes we read remotely vs bytes which were returned
from cached data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18184) s3a prefetching stream to support unbuffer()
[
https://issues.apache.org/jira/browse/HADOOP-18184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749922#comment-17749922
]
ASF GitHub Bot commented on HADOOP-18184:
-
steveloughran commented on code in PR #5832:
URL: https://github.com/apache/hadoop/pull/5832#discussion_r1280863384
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/prefetch/S3ACachingInputStream.java:
##
@@ -76,36 +79,75 @@ public S3ACachingInputStream(
S3AInputStreamStatistics streamStatistics,
Configuration conf,
LocalDirAllocator localDirAllocator) {
-super(context, s3Attributes, client, streamStatistics);
-this.numBlocksToPrefetch = this.getContext().getPrefetchBlockCount();
-int bufferPoolSize = this.numBlocksToPrefetch + 1;
-this.blockManager = this.createBlockManager(
-this.getContext().getFuturePool(),
-this.getReader(),
-this.getBlockData(),
-bufferPoolSize,
-conf,
-localDirAllocator);
+super(context, s3Attributes, client, streamStatistics);
+this.conf = conf;
+this.localDirAllocator = localDirAllocator;
+this.numBlocksToPrefetch = getContext().getPrefetchBlockCount();
+demandCreateBlockManager();
int fileSize = (int) s3Attributes.getLen();
LOG.debug("Created caching input stream for {} (size = {})",
this.getName(),
fileSize);
+streamStatistics.setPrefetchState(numBlocksToPrefetch > 0,
Review Comment:
what do you think we should publish? that we are prefetching the entire file?
> s3a prefetching stream to support unbuffer()
>
>
> Key: HADOOP-18184
> URL: https://issues.apache.org/jira/browse/HADOOP-18184
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Labels: pull-request-available
>
> Apache Impala uses unbuffer() to free up all client side resources held by a
> stream, so allowing it to have a map of available (path -> stream) objects,
> retained across queries.
> This saves on having to reopen the files, with the cost of HEAD checks etc.
> S3AInputStream just closes its http connection. here there is a lot more
> state to discard, but all memory and file storage must be freed.
> until this done, ITestS3AContractUnbuffer must skip when the prefetch stream
> is used.
> its notable that the other tests don't fail, even though the stream doesn't
> implement the interface; the graceful degradation handles that. it should
> fail if the test xml resource says the stream does it, but that the stream
> capabilities say it doesn't.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5832: HADOOP-18184. S3A Prefetching unbuffer.
steveloughran commented on code in PR #5832:
URL: https://github.com/apache/hadoop/pull/5832#discussion_r1280863384
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/prefetch/S3ACachingInputStream.java:
##
@@ -76,36 +79,75 @@ public S3ACachingInputStream(
S3AInputStreamStatistics streamStatistics,
Configuration conf,
LocalDirAllocator localDirAllocator) {
-super(context, s3Attributes, client, streamStatistics);
-this.numBlocksToPrefetch = this.getContext().getPrefetchBlockCount();
-int bufferPoolSize = this.numBlocksToPrefetch + 1;
-this.blockManager = this.createBlockManager(
-this.getContext().getFuturePool(),
-this.getReader(),
-this.getBlockData(),
-bufferPoolSize,
-conf,
-localDirAllocator);
+super(context, s3Attributes, client, streamStatistics);
+this.conf = conf;
+this.localDirAllocator = localDirAllocator;
+this.numBlocksToPrefetch = getContext().getPrefetchBlockCount();
+demandCreateBlockManager();
int fileSize = (int) s3Attributes.getLen();
LOG.debug("Created caching input stream for {} (size = {})",
this.getName(),
fileSize);
+streamStatistics.setPrefetchState(numBlocksToPrefetch > 0,
Review Comment:
what do you think we should publish? that we are prefetching the entire file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18184) s3a prefetching stream to support unbuffer()
[
https://issues.apache.org/jira/browse/HADOOP-18184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749919#comment-17749919
]
ASF GitHub Bot commented on HADOOP-18184:
-
steveloughran commented on code in PR #5832:
URL: https://github.com/apache/hadoop/pull/5832#discussion_r1280859452
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/PrefetchingStatistics.java:
##
@@ -24,14 +24,24 @@
import org.apache.hadoop.fs.statistics.DurationTracker;
import org.apache.hadoop.fs.statistics.IOStatisticsSource;
-public interface PrefetchingStatistics extends IOStatisticsSource {
+import static
org.apache.hadoop.fs.statistics.IOStatisticsSupport.stubDurationTracker;
+
+ public interface PrefetchingStatistics extends IOStatisticsSource {
/**
* A prefetch operation has started.
* @return duration tracker
*/
DurationTracker prefetchOperationStarted();
+ /**
+ * A block fetch operation has started.
+ * @return duration tracker
+ */
+ default DurationTracker blockFetchOperationStarted() {
+return stubDurationTracker();
Review Comment:
there's multiple empty back ends and putting it in here fixed them all. a
bit lazy, really. the existing code already exists; no need to change
> s3a prefetching stream to support unbuffer()
>
>
> Key: HADOOP-18184
> URL: https://issues.apache.org/jira/browse/HADOOP-18184
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Labels: pull-request-available
>
> Apache Impala uses unbuffer() to free up all client side resources held by a
> stream, so allowing it to have a map of available (path -> stream) objects,
> retained across queries.
> This saves on having to reopen the files, with the cost of HEAD checks etc.
> S3AInputStream just closes its http connection. here there is a lot more
> state to discard, but all memory and file storage must be freed.
> until this done, ITestS3AContractUnbuffer must skip when the prefetch stream
> is used.
> its notable that the other tests don't fail, even though the stream doesn't
> implement the interface; the graceful degradation handles that. it should
> fail if the test xml resource says the stream does it, but that the stream
> capabilities say it doesn't.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5832: HADOOP-18184. S3A Prefetching unbuffer.
steveloughran commented on code in PR #5832:
URL: https://github.com/apache/hadoop/pull/5832#discussion_r1280859452
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/PrefetchingStatistics.java:
##
@@ -24,14 +24,24 @@
import org.apache.hadoop.fs.statistics.DurationTracker;
import org.apache.hadoop.fs.statistics.IOStatisticsSource;
-public interface PrefetchingStatistics extends IOStatisticsSource {
+import static
org.apache.hadoop.fs.statistics.IOStatisticsSupport.stubDurationTracker;
+
+ public interface PrefetchingStatistics extends IOStatisticsSource {
/**
* A prefetch operation has started.
* @return duration tracker
*/
DurationTracker prefetchOperationStarted();
+ /**
+ * A block fetch operation has started.
+ * @return duration tracker
+ */
+ default DurationTracker blockFetchOperationStarted() {
+return stubDurationTracker();
Review Comment:
there's multiple empty back ends and putting it in here fixed them all. a
bit lazy, really. the existing code already exists; no need to change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18184) s3a prefetching stream to support unbuffer()
[
https://issues.apache.org/jira/browse/HADOOP-18184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749918#comment-17749918
]
ASF GitHub Bot commented on HADOOP-18184:
-
steveloughran commented on code in PR #5832:
URL: https://github.com/apache/hadoop/pull/5832#discussion_r1280859452
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/PrefetchingStatistics.java:
##
@@ -24,14 +24,24 @@
import org.apache.hadoop.fs.statistics.DurationTracker;
import org.apache.hadoop.fs.statistics.IOStatisticsSource;
-public interface PrefetchingStatistics extends IOStatisticsSource {
+import static
org.apache.hadoop.fs.statistics.IOStatisticsSupport.stubDurationTracker;
+
+ public interface PrefetchingStatistics extends IOStatisticsSource {
/**
* A prefetch operation has started.
* @return duration tracker
*/
DurationTracker prefetchOperationStarted();
+ /**
+ * A block fetch operation has started.
+ * @return duration tracker
+ */
+ default DurationTracker blockFetchOperationStarted() {
+return stubDurationTracker();
Review Comment:
there's multiple empty back ends and putting it in here fixed them all.
> s3a prefetching stream to support unbuffer()
>
>
> Key: HADOOP-18184
> URL: https://issues.apache.org/jira/browse/HADOOP-18184
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Labels: pull-request-available
>
> Apache Impala uses unbuffer() to free up all client side resources held by a
> stream, so allowing it to have a map of available (path -> stream) objects,
> retained across queries.
> This saves on having to reopen the files, with the cost of HEAD checks etc.
> S3AInputStream just closes its http connection. here there is a lot more
> state to discard, but all memory and file storage must be freed.
> until this done, ITestS3AContractUnbuffer must skip when the prefetch stream
> is used.
> its notable that the other tests don't fail, even though the stream doesn't
> implement the interface; the graceful degradation handles that. it should
> fail if the test xml resource says the stream does it, but that the stream
> capabilities say it doesn't.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5832: HADOOP-18184. S3A Prefetching unbuffer.
steveloughran commented on code in PR #5832:
URL: https://github.com/apache/hadoop/pull/5832#discussion_r1280859452
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/PrefetchingStatistics.java:
##
@@ -24,14 +24,24 @@
import org.apache.hadoop.fs.statistics.DurationTracker;
import org.apache.hadoop.fs.statistics.IOStatisticsSource;
-public interface PrefetchingStatistics extends IOStatisticsSource {
+import static
org.apache.hadoop.fs.statistics.IOStatisticsSupport.stubDurationTracker;
+
+ public interface PrefetchingStatistics extends IOStatisticsSource {
/**
* A prefetch operation has started.
* @return duration tracker
*/
DurationTracker prefetchOperationStarted();
+ /**
+ * A block fetch operation has started.
+ * @return duration tracker
+ */
+ default DurationTracker blockFetchOperationStarted() {
+return stubDurationTracker();
Review Comment:
there's multiple empty back ends and putting it in here fixed them all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18184) s3a prefetching stream to support unbuffer()
[
https://issues.apache.org/jira/browse/HADOOP-18184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749916#comment-17749916
]
ASF GitHub Bot commented on HADOOP-18184:
-
steveloughran commented on code in PR #5832:
URL: https://github.com/apache/hadoop/pull/5832#discussion_r1280857264
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/CachingBlockManager.java:
##
@@ -494,19 +546,25 @@ private void addToCacheAndRelease(BufferData data,
Future blockFuture,
prefetchingStatistics.executorAcquired(
Duration.between(taskQueuedStartTime, Instant.now()));
-if (closed) {
+if (isClosed()) {
return;
}
-if (cachingDisabled.get()) {
+final int blockNumber = data.getBlockNumber();
+LOG.debug("Block {}: Preparing to cache block", blockNumber);
+
+if (isCachingDisabled()) {
Review Comment:
there's a .get() on the future...which blocks until the data is received.
the checks on L577 are if caching changed during that time.
added some more comments and reviewed/tuned log messages
> s3a prefetching stream to support unbuffer()
>
>
> Key: HADOOP-18184
> URL: https://issues.apache.org/jira/browse/HADOOP-18184
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Labels: pull-request-available
>
> Apache Impala uses unbuffer() to free up all client side resources held by a
> stream, so allowing it to have a map of available (path -> stream) objects,
> retained across queries.
> This saves on having to reopen the files, with the cost of HEAD checks etc.
> S3AInputStream just closes its http connection. here there is a lot more
> state to discard, but all memory and file storage must be freed.
> until this done, ITestS3AContractUnbuffer must skip when the prefetch stream
> is used.
> its notable that the other tests don't fail, even though the stream doesn't
> implement the interface; the graceful degradation handles that. it should
> fail if the test xml resource says the stream does it, but that the stream
> capabilities say it doesn't.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5832: HADOOP-18184. S3A Prefetching unbuffer.
steveloughran commented on code in PR #5832:
URL: https://github.com/apache/hadoop/pull/5832#discussion_r1280857264
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/CachingBlockManager.java:
##
@@ -494,19 +546,25 @@ private void addToCacheAndRelease(BufferData data,
Future blockFuture,
prefetchingStatistics.executorAcquired(
Duration.between(taskQueuedStartTime, Instant.now()));
-if (closed) {
+if (isClosed()) {
return;
}
-if (cachingDisabled.get()) {
+final int blockNumber = data.getBlockNumber();
+LOG.debug("Block {}: Preparing to cache block", blockNumber);
+
+if (isCachingDisabled()) {
Review Comment:
there's a .get() on the future...which blocks until the data is received.
the checks on L577 are if caching changed during that time.
added some more comments and reviewed/tuned log messages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5869: HDFS-17113. Reconfig transfer and write bandwidth for datanode.
hadoop-yetus commented on PR #5869:
URL: https://github.com/apache/hadoop/pull/5869#issuecomment-1660559722
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 48m 6s | | trunk passed |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 10s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 35s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 3m 23s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 50s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 59s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 16s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 3m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 41m 18s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 240m 56s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 0s | | The patch does not
generate ASF License warnings. |
| | | 396m 17s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5869/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5869 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 374acd603c7e 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d363ec82661cb14bf717378db90e65f6e19fd701 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5869/3/testReport/ |
| Max. process+thread count | 2260 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5869/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about
[GitHub] [hadoop] hadoop-yetus commented on pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
hadoop-yetus commented on PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#issuecomment-1660551459
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 28s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 0m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 35s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 53s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 27s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 15s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 8s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 86m 0s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 175m 30s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5320 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux c53b65f809ad 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f98c387125a6e3bbe4ffb4010a68df379d0dbf32 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/9/testReport/ |
| Max. process+thread count | 938 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5320/9/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
[GitHub] [hadoop] hadoop-yetus commented on pull request #5912: HADOOP-18810. Document missing a lot of properties in core-default.xml.
hadoop-yetus commented on PR #5912:
URL: https://github.com/apache/hadoop/pull/5912#issuecomment-1660496806
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 4s | | trunk passed |
| +1 :green_heart: | compile | 20m 17s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 19m 0s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | mvnsite | 1m 48s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 22s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| -1 :x: | shadedclient | 127m 4s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 25s |
[/patch-mvninstall-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5912/1/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| -1 :x: | compile | 0m 25s |
[/patch-compile-root-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5912/1/artifact/out/patch-compile-root-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt)
| root in the patch failed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | javac | 0m 25s |
[/patch-compile-root-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5912/1/artifact/out/patch-compile-root-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt)
| root in the patch failed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | compile | 0m 25s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5912/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt)
| root in the patch failed with JDK Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09. |
| -1 :x: | javac | 0m 25s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5912/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt)
| root in the patch failed with JDK Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09. |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| -1 :x: | mvnsite | 0m 25s |
[/patch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5912/1/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| -1 :x: | javadoc | 0m 24s |
[/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5912/1/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt)
| hadoop-common in the patch failed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1. |
| +1 :green_heart: | javadoc | 0m 35s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| -1 :x: | shadedclient | 6m 0s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 2m 25s |
[/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5912/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not
[jira] [Commented] (HADOOP-18810) Document missing a lot of properties in core-default.xml
[
https://issues.apache.org/jira/browse/HADOOP-18810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749873#comment-17749873
]
ASF GitHub Bot commented on HADOOP-18810:
-
hadoop-yetus commented on PR #5912:
URL: https://github.com/apache/hadoop/pull/5912#issuecomment-1660496806
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 4s | | trunk passed |
| +1 :green_heart: | compile | 20m 17s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 19m 0s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | mvnsite | 1m 48s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 22s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| -1 :x: | shadedclient | 127m 4s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 25s |
[/patch-mvninstall-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5912/1/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| -1 :x: | compile | 0m 25s |
[/patch-compile-root-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5912/1/artifact/out/patch-compile-root-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt)
| root in the patch failed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | javac | 0m 25s |
[/patch-compile-root-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5912/1/artifact/out/patch-compile-root-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt)
| root in the patch failed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | compile | 0m 25s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5912/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt)
| root in the patch failed with JDK Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09. |
| -1 :x: | javac | 0m 25s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5912/1/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt)
| root in the patch failed with JDK Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09. |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| -1 :x: | mvnsite | 0m 25s |
[/patch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5912/1/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| -1 :x: | javadoc | 0m 24s |
[/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5912/1/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt)
| hadoop-common in the patch failed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1. |
| +1 :green_heart: | javadoc | 0m 35s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| -1 :x: | shadedclient | 6m 0s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 2m 25s |
[GitHub] [hadoop] hadoop-yetus commented on pull request #5913: HDFS-17137. Standby/Observer NameNode should not handle redundant rep…
hadoop-yetus commented on PR #5913:
URL: https://github.com/apache/hadoop/pull/5913#issuecomment-1660455782
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 37m 18s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5913/1/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 1m 30s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 2m 0s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 4m 24s | | trunk passed |
| -1 :x: | shadedclient | 9m 56s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 24s |
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5913/1/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| -1 :x: | compile | 0m 24s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5913/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | javac | 0m 24s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5913/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | compile | 0m 24s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5913/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt)
| hadoop-hdfs in the patch failed with JDK Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09. |
| -1 :x: | javac | 0m 24s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5913/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt)
| hadoop-hdfs in the patch failed with JDK Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09. |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 22s |
[/buildtool-patch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5913/1/artifact/out/buildtool-patch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| The patch fails to run checkstyle in hadoop-hdfs |
| -1 :x: | mvnsite | 0m 24s |
[/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5913/1/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| -1 :x: | javadoc | 0m 24s |
[/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5913/1/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x:
[jira] [Commented] (HADOOP-18820) AWS SDK v2: make the v1 bridging support optional
[ https://issues.apache.org/jira/browse/HADOOP-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749817#comment-17749817 ] ASF GitHub Bot commented on HADOOP-18820: - steveloughran commented on PR #5872: URL: https://github.com/apache/hadoop/pull/5872#issuecomment-1660437266 last test run s3 london, only failure was the expected ITestS3AHugeFilesNoMultipart suite ``` -Dparallel-tests -DtestsThreadCount=8 -Dprefetch -Dscale ``` > AWS SDK v2: make the v1 bridging support optional > - > > Key: HADOOP-18820 > URL: https://issues.apache.org/jira/browse/HADOOP-18820 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > > The AWS SDK v2 code includes the v1 sdk core for plugin support of > * existing credential providers > * delegation token binding > I propose we break #2 and rely on those who have implemented to to upgrade. > apart from all the needless changes the v2 SDK did to the api (why?) this is > fairly straighforward > for #1: fix through reflection, retaining a v1 sdk dependency at test time so > we can verify that the binder works. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5872: HADOOP-18820. Cut AWS v1 support
steveloughran commented on PR #5872: URL: https://github.com/apache/hadoop/pull/5872#issuecomment-1660437266 last test run s3 london, only failure was the expected ITestS3AHugeFilesNoMultipart suite ``` -Dparallel-tests -DtestsThreadCount=8 -Dprefetch -Dscale ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5901: YARN-7402. BackPort [GPG] Fix potential connection leak in GPGUtils.
hadoop-yetus commented on PR #5901:
URL: https://github.com/apache/hadoop/pull/5901#issuecomment-1660434061
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 49s | | trunk passed |
| +1 :green_heart: | compile | 0m 22s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 0m 21s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 0m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 22s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 0m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 55s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 15s | | the patch passed |
| +1 :green_heart: | compile | 0m 15s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 0m 15s | | the patch passed |
| +1 :green_heart: | compile | 0m 14s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 0m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 12s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 16s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 16s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 14s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 0m 30s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 57s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 32s | |
hadoop-yarn-server-globalpolicygenerator in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not
generate ASF License warnings. |
| | | 88m 42s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5901/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5901 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux cfb8957a60fc 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9273188537954f868b7a08adc83ad7b4bbb6acda |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5901/4/testReport/ |
| Max. process+thread count | 641 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5901/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to
[jira] [Updated] (HADOOP-18810) Document missing a lot of properties in core-default.xml
[ https://issues.apache.org/jira/browse/HADOOP-18810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangYuanben updated HADOOP-18810: - Component/s: common > Document missing a lot of properties in core-default.xml > > > Key: HADOOP-18810 > URL: https://issues.apache.org/jira/browse/HADOOP-18810 > Project: Hadoop Common > Issue Type: Improvement > Components: common, documentation >Reporter: WangYuanben >Assignee: WangYuanben >Priority: Major > Labels: pull-request-available > Attachments: image-2023-07-19-18-02-23-700.png > > > !image-2023-07-19-18-02-23-700.png|width=600,height=333! > As the picture shows, there are many configs in core-site.xml, which should > be documented. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18836) Some properties are missing from hadoop-policy.xml.
[
https://issues.apache.org/jira/browse/HADOOP-18836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
WangYuanben updated HADOOP-18836:
-
Component/s: documentation
> Some properties are missing from hadoop-policy.xml.
> ---
>
> Key: HADOOP-18836
> URL: https://issues.apache.org/jira/browse/HADOOP-18836
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, documentation
>Reporter: WangYuanben
>Assignee: WangYuanben
>Priority: Major
>
>
> {noformat}
> security.datanode.lifeline.protocol.acl
> security.get.user.mappings.protocol.acl
> security.reconfiguration.protocol.acl
> security.refresh.callqueue.protocol.acl
> security.refresh.generic.protocol.acl{noformat}
> The properties above are missing from hadoop-policy.xml.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18836) Some properties are missing from hadoop-policy.xml.
[
https://issues.apache.org/jira/browse/HADOOP-18836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
WangYuanben updated HADOOP-18836:
-
Target Version/s: 3.4.0
> Some properties are missing from hadoop-policy.xml.
> ---
>
> Key: HADOOP-18836
> URL: https://issues.apache.org/jira/browse/HADOOP-18836
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Reporter: WangYuanben
>Assignee: WangYuanben
>Priority: Major
>
>
> {noformat}
> security.datanode.lifeline.protocol.acl
> security.get.user.mappings.protocol.acl
> security.reconfiguration.protocol.acl
> security.refresh.callqueue.protocol.acl
> security.refresh.generic.protocol.acl{noformat}
> The properties above are missing from hadoop-policy.xml.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18836) Some properties are missing from hadoop-policy.xml.
[
https://issues.apache.org/jira/browse/HADOOP-18836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
WangYuanben updated HADOOP-18836:
-
Component/s: common
> Some properties are missing from hadoop-policy.xml.
> ---
>
> Key: HADOOP-18836
> URL: https://issues.apache.org/jira/browse/HADOOP-18836
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Reporter: WangYuanben
>Assignee: WangYuanben
>Priority: Major
>
>
> {noformat}
> security.datanode.lifeline.protocol.acl
> security.get.user.mappings.protocol.acl
> security.reconfiguration.protocol.acl
> security.refresh.callqueue.protocol.acl
> security.refresh.generic.protocol.acl{noformat}
> The properties above are missing from hadoop-policy.xml.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18836) Some properties are missing from hadoop-policy.xml.
WangYuanben created HADOOP-18836:
Summary: Some properties are missing from hadoop-policy.xml.
Key: HADOOP-18836
URL: https://issues.apache.org/jira/browse/HADOOP-18836
Project: Hadoop Common
Issue Type: Improvement
Reporter: WangYuanben
Assignee: WangYuanben
{noformat}
security.datanode.lifeline.protocol.acl
security.get.user.mappings.protocol.acl
security.reconfiguration.protocol.acl
security.refresh.callqueue.protocol.acl
security.refresh.generic.protocol.acl{noformat}
The properties above are missing from hadoop-policy.xml.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] haiyang1987 opened a new pull request, #5913: HDFS-17137. Standby/Observer NameNode should not handle redundant rep…
haiyang1987 opened a new pull request, #5913: URL: https://github.com/apache/hadoop/pull/5913 ### Description of PR https://issues.apache.org/jira/browse/HDFS-17137 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18820) AWS SDK v2: make the v1 bridging support optional
[ https://issues.apache.org/jira/browse/HADOOP-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749724#comment-17749724 ] ASF GitHub Bot commented on HADOOP-18820: - steveloughran commented on PR #5872: URL: https://github.com/apache/hadoop/pull/5872#issuecomment-1660262564 created https://issues.apache.org/jira/browse/YARN-11546 for the TestTimelineAuthFilterForV2 failure > AWS SDK v2: make the v1 bridging support optional > - > > Key: HADOOP-18820 > URL: https://issues.apache.org/jira/browse/HADOOP-18820 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > > The AWS SDK v2 code includes the v1 sdk core for plugin support of > * existing credential providers > * delegation token binding > I propose we break #2 and rely on those who have implemented to to upgrade. > apart from all the needless changes the v2 SDK did to the api (why?) this is > fairly straighforward > for #1: fix through reflection, retaining a v1 sdk dependency at test time so > we can verify that the binder works. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5872: HADOOP-18820. Cut AWS v1 support
steveloughran commented on PR #5872: URL: https://github.com/apache/hadoop/pull/5872#issuecomment-1660262564 created https://issues.apache.org/jira/browse/YARN-11546 for the TestTimelineAuthFilterForV2 failure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on a diff in pull request #5855: HDFS-17093. In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incompl
Hexiaoqiao commented on code in PR #5855: URL: https://github.com/apache/hadoop/pull/5855#discussion_r1280597069 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java: ## @@ -1650,7 +1650,7 @@ public DatanodeCommand blockReport(final DatanodeRegistration nodeReg, final int index = r; noStaleStorages = bm.runBlockOp(() -> bm.processReport(nodeReg, reports[index].getStorage(), -blocks, context)); +blocks, context, reports.length, index + 1)); Review Comment: Great catch. +1, We have to solve this case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5901: YARN-7402. BackPort [GPG] Fix potential connection leak in GPGUtils.
slfan1989 commented on code in PR #5901:
URL: https://github.com/apache/hadoop/pull/5901#discussion_r1280592893
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/src/main/java/org/apache/hadoop/yarn/server/globalpolicygenerator/GPGUtils.java:
##
@@ -57,15 +58,23 @@ public static T invokeRMWebService(String webAddr,
String path, final Class<
T obj = null;
WebResource webResource = client.resource(webAddr);
-ClientResponse response = webResource.path("ws/v1/cluster").path(path)
-.accept(MediaType.APPLICATION_XML).get(ClientResponse.class);
-if (response.getStatus() == HttpServletResponse.SC_OK) {
- obj = response.getEntity(returnType);
-} else {
- throw new YarnRuntimeException("Bad response from remote web service: "
- + response.getStatus());
+ClientResponse response = null;
+try {
+ response = webResource.path("ws/v1/cluster").path(path)
+ .accept(MediaType.APPLICATION_XML).get(ClientResponse.class);
+ if (response.getStatus() == SC_OK) {
+obj = response.getEntity(returnType);
+ } else {
+throw new YarnRuntimeException(
+"Bad response from remote web service: " + response.getStatus());
+ }
+ return obj;
+} finally {
+ if (response != null) {
+response.close();
Review Comment:
Thank you very much for reviewing the code! I will fix it.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/src/test/java/org/apache/hadoop/yarn/server/globalpolicygenerator/policygenerator/TestPolicyGenerator.java:
##
@@ -44,10 +44,7 @@
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration;
import org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWSConsts;
-import
org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.CapacitySchedulerInfo;
-import
org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.ClusterMetricsInfo;
-import org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.SchedulerInfo;
-import
org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.SchedulerTypeInfo;
+import org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.*;
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18810) Document missing a lot of properties in core-default.xml
[ https://issues.apache.org/jira/browse/HADOOP-18810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18810: Labels: pull-request-available (was: ) > Document missing a lot of properties in core-default.xml > > > Key: HADOOP-18810 > URL: https://issues.apache.org/jira/browse/HADOOP-18810 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Reporter: WangYuanben >Assignee: WangYuanben >Priority: Major > Labels: pull-request-available > Attachments: image-2023-07-19-18-02-23-700.png > > > !image-2023-07-19-18-02-23-700.png|width=600,height=333! > As the picture shows, there are many configs in core-site.xml, which should > be documented. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18810) Document missing a lot of properties in core-default.xml
[ https://issues.apache.org/jira/browse/HADOOP-18810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749689#comment-17749689 ] ASF GitHub Bot commented on HADOOP-18810: - YuanbenWang opened a new pull request, #5912: URL: https://github.com/apache/hadoop/pull/5912 JIRA: [HADOOP-18810](https://issues.apache.org/jira/browse/HADOOP-18810). Document missing a lot of properties in core-default.xml. ### Description of PR there are many missing configs in core-site.xml, which should be documented. > Document missing a lot of properties in core-default.xml > > > Key: HADOOP-18810 > URL: https://issues.apache.org/jira/browse/HADOOP-18810 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Reporter: WangYuanben >Assignee: WangYuanben >Priority: Major > Attachments: image-2023-07-19-18-02-23-700.png > > > !image-2023-07-19-18-02-23-700.png|width=600,height=333! > As the picture shows, there are many configs in core-site.xml, which should > be documented. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] YuanbenWang opened a new pull request, #5912: HADOOP-18810. Document missing a lot of properties in core-default.xml.
YuanbenWang opened a new pull request, #5912: URL: https://github.com/apache/hadoop/pull/5912 JIRA: [HADOOP-18810](https://issues.apache.org/jira/browse/HADOOP-18810). Document missing a lot of properties in core-default.xml. ### Description of PR there are many missing configs in core-site.xml, which should be documented. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18810) Document missing a lot of properties in core-default.xml
[ https://issues.apache.org/jira/browse/HADOOP-18810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangYuanben updated HADOOP-18810: - Description: !image-2023-07-19-18-02-23-700.png|width=600,height=333! As the picture shows, there are many configs in core-site.xml, which should be documented. was: !image-2023-07-19-18-02-23-700.png|width=600,height=333! As the picture shows, there are 83 missing configs in core-site.xml, which should be documented. > Document missing a lot of properties in core-default.xml > > > Key: HADOOP-18810 > URL: https://issues.apache.org/jira/browse/HADOOP-18810 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Reporter: WangYuanben >Assignee: WangYuanben >Priority: Major > Attachments: image-2023-07-19-18-02-23-700.png > > > !image-2023-07-19-18-02-23-700.png|width=600,height=333! > As the picture shows, there are many configs in core-site.xml, which should > be documented. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5862: YARN-11536. [Federation] Router CLI Supports Batch Save the SubClusterPolicyConfiguration Of Queues.
hadoop-yetus commented on PR #5862:
URL: https://github.com/apache/hadoop/pull/5862#issuecomment-1660176139
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 4s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 7 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 6s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 36m 49s | | trunk passed |
| +1 :green_heart: | compile | 9m 18s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 7m 47s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 2m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 30s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 4m 59s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 11m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 23s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 4m 1s | | the patch passed |
| +1 :green_heart: | compile | 9m 45s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | cc | 9m 45s | | the patch passed |
| +1 :green_heart: | javac | 9m 45s | | the patch passed |
| +1 :green_heart: | compile | 8m 56s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | cc | 8m 56s | | the patch passed |
| +1 :green_heart: | javac | 8m 56s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 2m 6s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5862/13/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 64 unchanged -
0 fixed = 67 total (was 64) |
| +1 :green_heart: | mvnsite | 5m 8s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 1s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 4m 42s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 12m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 15s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 47s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 42s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 109m 43s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 28m 44s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | unit | 0m 44s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 4s | | The patch does not
generate ASF License warnings. |
| | | 387m 58s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5862/13/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5862 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint
bufcompat xmllint |
| uname | Linux 884f9758968d
[GitHub] [hadoop] hadoop-yetus commented on pull request #5862: YARN-11536. [Federation] Router CLI Supports Batch Save the SubClusterPolicyConfiguration Of Queues.
hadoop-yetus commented on PR #5862:
URL: https://github.com/apache/hadoop/pull/5862#issuecomment-1660173776
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 3s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 7 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 16s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 36m 10s | | trunk passed |
| +1 :green_heart: | compile | 8m 58s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 8m 0s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 2m 3s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 23s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 5m 0s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 11m 16s | | trunk passed |
| +1 :green_heart: | shadedclient | 41m 23s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 58s | | the patch passed |
| +1 :green_heart: | compile | 9m 34s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | cc | 9m 34s | | the patch passed |
| +1 :green_heart: | javac | 9m 34s | | the patch passed |
| +1 :green_heart: | compile | 9m 10s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | cc | 9m 10s | | the patch passed |
| +1 :green_heart: | javac | 9m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 2m 2s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5862/12/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 64 unchanged -
0 fixed = 67 total (was 64) |
| +1 :green_heart: | mvnsite | 5m 7s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 58s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 4m 42s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 12m 3s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 9s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 51s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 42s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 109m 29s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 28m 32s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | unit | 0m 41s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 10s | | The patch does not
generate ASF License warnings. |
| | | 388m 9s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5862/12/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5862 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint
bufcompat xmllint |
| uname | Linux 76a70e876e61
[GitHub] [hadoop] zhangshuyan0 commented on a diff in pull request #5855: HDFS-17093. In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incom
zhangshuyan0 commented on code in PR #5855: URL: https://github.com/apache/hadoop/pull/5855#discussion_r1280525397 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java: ## @@ -1650,7 +1650,7 @@ public DatanodeCommand blockReport(final DatanodeRegistration nodeReg, final int index = r; noStaleStorages = bm.runBlockOp(() -> bm.processReport(nodeReg, reports[index].getStorage(), -blocks, context)); +blocks, context, reports.length, index + 1)); Review Comment: Thanks for reminding me of the shortcomings of my plan. I will try to improve it. However, the solution in this PR may not work. If datanode send one storage report per RPC, `reports.length` will be 1 here. Your code `totalReportNum == currentReportNum` will always be true. So block report lease will be removed as before. This repairing will be ineffective. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] brumi1024 commented on pull request #5320: YARN-11416. FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
brumi1024 commented on PR #5320: URL: https://github.com/apache/hadoop/pull/5320#issuecomment-1660169676 Thanks @susheel-gupta, the latest state LGTM. Can you please fix the checkstyle issue and that line ending? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18810) Document missing a lot of properties in core-default.xml
[ https://issues.apache.org/jira/browse/HADOOP-18810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangYuanben updated HADOOP-18810: - Summary: Document missing a lot of properties in core-default.xml (was: Document missing lots of properties in core-default.xml) > Document missing a lot of properties in core-default.xml > > > Key: HADOOP-18810 > URL: https://issues.apache.org/jira/browse/HADOOP-18810 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Reporter: WangYuanben >Assignee: WangYuanben >Priority: Major > Attachments: image-2023-07-19-18-02-23-700.png > > > !image-2023-07-19-18-02-23-700.png|width=600,height=333! > As the picture shows, there are 83 missing configs in core-site.xml, which > should be documented. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18810) Document missing lots of properties in core-default.xml
[ https://issues.apache.org/jira/browse/HADOOP-18810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] WangYuanben updated HADOOP-18810: - Summary: Document missing lots of properties in core-default.xml (was: Document missing many properties in core-default.xml) > Document missing lots of properties in core-default.xml > --- > > Key: HADOOP-18810 > URL: https://issues.apache.org/jira/browse/HADOOP-18810 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Reporter: WangYuanben >Assignee: WangYuanben >Priority: Major > Attachments: image-2023-07-19-18-02-23-700.png > > > !image-2023-07-19-18-02-23-700.png|width=600,height=333! > As the picture shows, there are 83 missing configs in core-site.xml, which > should be documented. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] susheel-gupta commented on a diff in pull request #5884: YARN-11535: Snakeyaml should be excluded from jackson-dataformat-yaml-2.12.7 as it may cause transitive dependency issue.
susheel-gupta commented on code in PR #5884: URL: https://github.com/apache/hadoop/pull/5884#discussion_r1280477691 ## hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml: ## @@ -333,6 +333,16 @@ com.fasterxml.jackson.dataformat jackson-dataformat-yaml + + + org.yaml + snakeyaml + + + + + org.yaml + snakeyaml Review Comment: @tomicooler Yes, but this needed to be fixed for other jackson related dependency altogether having 2.12.7 , https://github.com/apache/hadoop/blob/5aee0e0c0a2c1d4181e92a20b311885a5c552121/hadoop-project/pom.xml#L72 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5862: YARN-11536. [Federation] Router CLI Supports Batch Save the SubClusterPolicyConfiguration Of Queues.
hadoop-yetus commented on PR #5862:
URL: https://github.com/apache/hadoop/pull/5862#issuecomment-1660090683
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 7 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 5s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 35m 27s | | trunk passed |
| +1 :green_heart: | compile | 8m 7s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 7m 20s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 14s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 4s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 4m 42s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 9m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 20s | | the patch passed |
| +1 :green_heart: | compile | 7m 26s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | cc | 7m 26s | | the patch passed |
| +1 :green_heart: | javac | 7m 26s | | the patch passed |
| +1 :green_heart: | compile | 7m 14s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | cc | 7m 14s | | the patch passed |
| +1 :green_heart: | javac | 7m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 50s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5862/11/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 65 unchanged -
0 fixed = 68 total (was 65) |
| +1 :green_heart: | mvnsite | 4m 48s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 36s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| -1 :x: | javadoc | 0m 34s |
[/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5862/11/artifact/out/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09
with JDK Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 generated 5
new + 164 unchanged - 0 fixed = 169 total (was 164) |
| +1 :green_heart: | spotbugs | 10m 31s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 19s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 8s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 41s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 28s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 103m 15s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 28m 12s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | unit | 0m 41s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not
generate ASF License warnings. |
| | | 364m 56s |
[GitHub] [hadoop] tomicooler commented on a diff in pull request #5884: YARN-11535: Snakeyaml should be excluded from jackson-dataformat-yaml-2.12.7 as it may cause transitive dependency issue.
tomicooler commented on code in PR #5884: URL: https://github.com/apache/hadoop/pull/5884#discussion_r1280451636 ## hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml: ## @@ -333,6 +333,16 @@ com.fasterxml.jackson.dataformat jackson-dataformat-yaml + + + org.yaml + snakeyaml + + + + + org.yaml + snakeyaml Review Comment: I think instead of this exclusion, we should update `com.fasterxml.jackson.dataformat` to 2.15.2 version which uses snakeyaml 2.0. https://mvnrepository.com/artifact/com.fasterxml.jackson.dataformat/jackson-dataformat-yaml/2.15.2 Can we try that option? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5862: YARN-11536. [Federation] Router CLI Supports Batch Save the SubClusterPolicyConfiguration Of Queues.
hadoop-yetus commented on PR #5862:
URL: https://github.com/apache/hadoop/pull/5862#issuecomment-1659997281
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 7 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 57s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 3s | | trunk passed |
| +1 :green_heart: | compile | 5m 1s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 4m 17s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 20s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 4m 7s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 7m 2s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 18s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 29s | | the patch passed |
| +1 :green_heart: | compile | 4m 10s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | cc | 4m 10s | | the patch passed |
| +1 :green_heart: | javac | 4m 10s | | the patch passed |
| +1 :green_heart: | compile | 4m 12s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | cc | 4m 12s | | the patch passed |
| +1 :green_heart: | javac | 4m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 3s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5862/14/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 64 unchanged -
0 fixed = 67 total (was 64) |
| +1 :green_heart: | mvnsite | 3m 55s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 51s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 41s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 7m 54s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 31s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 55s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 4m 48s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 49s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 85m 55s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 26m 8s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | unit | 0m 37s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not
generate ASF License warnings. |
| | | 266m 17s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5862/14/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5862 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint
bufcompat xmllint |
| uname | Linux 46c95d590129
[GitHub] [hadoop] Hexiaoqiao commented on pull request #5900: HDFS-17134. RBF: Fix duplicate results of getListing through Router.
Hexiaoqiao commented on PR #5900: URL: https://github.com/apache/hadoop/pull/5900#issuecomment-1659972653 Committed to trunk. Thanks @zhangshuyan0 for your contribution and @ayushtkn for your reviews! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao merged pull request #5900: HDFS-17134. RBF: Fix duplicate results of getListing through Router.
Hexiaoqiao merged PR #5900: URL: https://github.com/apache/hadoop/pull/5900 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sourav-biggboss closed pull request #5911: Update NOTICE.txt
sourav-biggboss closed pull request #5911: Update NOTICE.txt URL: https://github.com/apache/hadoop/pull/5911 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sourav-biggboss opened a new pull request, #5911: Update NOTICE.txt
sourav-biggboss opened a new pull request, #5911: URL: https://github.com/apache/hadoop/pull/5911 ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5855: HDFS-17093. In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incomplete bloc
hadoop-yetus commented on PR #5855:
URL: https://github.com/apache/hadoop/pull/5855#issuecomment-1659779990
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 44s | | trunk passed |
| +1 :green_heart: | compile | 0m 53s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 0m 50s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 0m 44s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 57s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 49s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 14s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 47s | | the patch passed |
| +1 :green_heart: | compile | 0m 46s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 0m 46s | | the patch passed |
| +1 :green_heart: | compile | 0m 42s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 0m 42s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 34s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5855/7/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 147 unchanged
- 0 fixed = 149 total (was 147) |
| +1 :green_heart: | mvnsite | 0m 47s | | the patch passed |
| -1 :x: | javadoc | 0m 38s |
[/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5855/7/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1. |
| +1 :green_heart: | javadoc | 1m 1s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 55s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 16s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 202m 21s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5855/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 298m 9s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestObserverNode |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5855/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5855 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d2ee3a6dcd98 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3b72336f88597c20cc566f7cef0a95a6be74e4ce |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
[GitHub] [hadoop] Tre2878 commented on a diff in pull request #5855: HDFS-17093. In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incomplete
Tre2878 commented on code in PR #5855:
URL: https://github.com/apache/hadoop/pull/5855#discussion_r1280190216
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:
##
@@ -2904,7 +2908,8 @@ public boolean processReport(final DatanodeID nodeID,
}
if (namesystem.isInStartupSafeMode()
&& !StorageType.PROVIDED.equals(storageInfo.getStorageType())
- && storageInfo.getBlockReportCount() > 0) {
+ && storageInfo.getBlockReportCount() > 0
+ && totalReportNum == currentReportNum) {
Review Comment:
Two of the cases will come into this logic
1. The namenode is restarting while receiving FBRS from all datanodes and is
in safe mode
2. When the namenode is in secure mode for some reason while it has been
running for a long time
In the first case, if the datanode has a failed disk, the datanode will send
the FBR for the normal disk and the namenode will handle it normally
In the second case, blockReportCount == 0 will always be false if no new
disks are added to the datanode
So I recommend keeping the code as it is and not using blockReportCount == 0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18820) AWS SDK v2: make the v1 bridging support optional
[
https://issues.apache.org/jira/browse/HADOOP-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17749485#comment-17749485
]
ASF GitHub Bot commented on HADOOP-18820:
-
hadoop-yetus commented on PR #5872:
URL: https://github.com/apache/hadoop/pull/5872#issuecomment-1659666215
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 11 new or modified test files. |
_ feature-HADOOP-18073-s3a-sdk-upgrade Compile Tests _ |
| +0 :ok: | mvndep | 14m 36s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 4s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed |
| +1 :green_heart: | compile | 11m 9s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 9m 36s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 2m 32s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed |
| +1 :green_heart: | mvnsite | 15m 36s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed |
| +1 :green_heart: | javadoc | 6m 8s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 5m 18s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +0 :ok: | spotbugs | 0m 17s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 0m 42s |
[/branch-spotbugs-hadoop-tools_hadoop-aws-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5872/9/artifact/out/branch-spotbugs-hadoop-tools_hadoop-aws-warnings.html)
| hadoop-tools/hadoop-aws in feature-HADOOP-18073-s3a-sdk-upgrade has 1
extant spotbugs warnings. |
| -1 :x: | spotbugs | 19m 15s |
[/branch-spotbugs-root-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5872/9/artifact/out/branch-spotbugs-root-warnings.html)
| root in feature-HADOOP-18073-s3a-sdk-upgrade has 1 extant spotbugs
warnings. |
| +1 :green_heart: | shadedclient | 41m 11s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 41m 30s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 20m 53s | | the patch passed |
| +1 :green_heart: | compile | 10m 7s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 10m 7s | | the patch passed |
| +1 :green_heart: | compile | 9m 31s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 9m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 2m 22s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5872/9/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 12 unchanged - 12 fixed = 13 total (was
24) |
| +1 :green_heart: | mvnsite | 9m 25s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 5m 54s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| -1 :x: | javadoc | 4m 43s |
[/patch-javadoc-root-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5872/9/artifact/out/patch-javadoc-root-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt)
| root in the patch failed with JDK Private
[GitHub] [hadoop] hadoop-yetus commented on pull request #5872: HADOOP-18820. Cut AWS v1 support
hadoop-yetus commented on PR #5872:
URL: https://github.com/apache/hadoop/pull/5872#issuecomment-1659666215
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 11 new or modified test files. |
_ feature-HADOOP-18073-s3a-sdk-upgrade Compile Tests _ |
| +0 :ok: | mvndep | 14m 36s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 4s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed |
| +1 :green_heart: | compile | 11m 9s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 9m 36s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 2m 32s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed |
| +1 :green_heart: | mvnsite | 15m 36s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed |
| +1 :green_heart: | javadoc | 6m 8s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 5m 18s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +0 :ok: | spotbugs | 0m 17s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 0m 42s |
[/branch-spotbugs-hadoop-tools_hadoop-aws-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5872/9/artifact/out/branch-spotbugs-hadoop-tools_hadoop-aws-warnings.html)
| hadoop-tools/hadoop-aws in feature-HADOOP-18073-s3a-sdk-upgrade has 1
extant spotbugs warnings. |
| -1 :x: | spotbugs | 19m 15s |
[/branch-spotbugs-root-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5872/9/artifact/out/branch-spotbugs-root-warnings.html)
| root in feature-HADOOP-18073-s3a-sdk-upgrade has 1 extant spotbugs
warnings. |
| +1 :green_heart: | shadedclient | 41m 11s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 41m 30s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 20m 53s | | the patch passed |
| +1 :green_heart: | compile | 10m 7s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 10m 7s | | the patch passed |
| +1 :green_heart: | compile | 9m 31s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 9m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 2m 22s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5872/9/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 12 unchanged - 12 fixed = 13 total (was
24) |
| +1 :green_heart: | mvnsite | 9m 25s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 5m 54s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| -1 :x: | javadoc | 4m 43s |
[/patch-javadoc-root-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5872/9/artifact/out/patch-javadoc-root-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt)
| root in the patch failed with JDK Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09. |
| +0 :ok: | spotbugs | 0m 17s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 38m 15s | | patch has no errors
when building and testing our client
[GitHub] [hadoop] Hexiaoqiao commented on pull request #5900: HDFS-17134. RBF: Fix duplicate results of getListing through Router.
Hexiaoqiao commented on PR #5900: URL: https://github.com/apache/hadoop/pull/5900#issuecomment-1659638104 Let's wait another 24h, I will check in if no more comments. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on a diff in pull request #5855: HDFS-17093. In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incompl
Hexiaoqiao commented on code in PR #5855:
URL: https://github.com/apache/hadoop/pull/5855#discussion_r1280154310
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:
##
@@ -2904,7 +2908,8 @@ public boolean processReport(final DatanodeID nodeID,
}
if (namesystem.isInStartupSafeMode()
&& !StorageType.PROVIDED.equals(storageInfo.getStorageType())
- && storageInfo.getBlockReportCount() > 0) {
+ && storageInfo.getBlockReportCount() > 0
+ && totalReportNum == currentReportNum) {
Review Comment:
I am not sure if this will be a good solution with condition
`blockReportCount == 0`, consider that one disk failed but not checked in time.
Will it affect this logic here? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xinglin commented on a diff in pull request #5855: HDFS-17093. In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incomplete
xinglin commented on code in PR #5855:
URL: https://github.com/apache/hadoop/pull/5855#discussion_r1280143136
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:
##
@@ -2904,7 +2908,8 @@ public boolean processReport(final DatanodeID nodeID,
}
if (namesystem.isInStartupSafeMode()
&& !StorageType.PROVIDED.equals(storageInfo.getStorageType())
- && storageInfo.getBlockReportCount() > 0) {
+ && storageInfo.getBlockReportCount() > 0
+ && totalReportNum == currentReportNum) {
Review Comment:
I like @zhangshuyan0's proposal better.
The following section of code can also be separated to be a function on its
own.
```
// Remove the lease when we have received block reports for all storages for
a particular DN.
void removeLease() {
for (DatanodeStorageInfo sInfo : node.getStorageInfos()) {
if (sInfo.getBlockReportCount() == 0) {
needRemoveLease = false;
}
}
if (needRemoveLease) {
blockReportLeaseManager.removeLease(node);
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
