[PR] YARN-11616. Fast fail for NodeConstraintParser when having multi attribute kvs [hadoop]
zuston opened a new pull request, #6282: URL: https://github.com/apache/hadoop/pull/6282 ### Description of PR In the `NodeConstraintParser`, it won't throw exception when multiple attribute kvs are specified. It will return incorrect placement constraint, which will mislead users. Like the `rm.yarn.io/foo=1,rm.yarn.io/bar=2`, it will parse it to `node,EQ,rm.yarn.io/bar=[1:2]` ### How was this patch tested? Verified in our internal cluster. ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] YARN-11616. Fast fail for NodeConstraintParser when having multi attribute kvs [hadoop]
zuston commented on PR #6282: URL: https://github.com/apache/hadoop/pull/6282#issuecomment-1818425405 Could you help review this ? @slfan1989 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18959 Use builder for prefetch CachingBlockManager [hadoop]
hadoop-yetus commented on PR #6240:
URL: https://github.com/apache/hadoop/pull/6240#issuecomment-1818459112
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 35s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 18m 45s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6240/5/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 8m 20s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 7m 30s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 59s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 30s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 47s | | the patch passed |
| +1 :green_heart: | compile | 7m 55s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 7m 55s | | the patch passed |
| +1 :green_heart: | compile | 7m 33s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 7m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 54s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 1s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 22s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 43s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 15m 47s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 13s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 141m 36s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6240/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6240 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e8edd1dac598 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6f217d2c9bb795ee7b19ac333ddf5d5ac377c9d7 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6240/5/testReport/ |
| Max. process+thread count | 3152 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws
U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6240/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.or
[jira] [Commented] (HADOOP-18959) Use builder for prefetch CachingBlockManager
[
https://issues.apache.org/jira/browse/HADOOP-18959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787881#comment-17787881
]
ASF GitHub Bot commented on HADOOP-18959:
-
hadoop-yetus commented on PR #6240:
URL: https://github.com/apache/hadoop/pull/6240#issuecomment-1818459112
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 35s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 18m 45s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6240/5/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 8m 20s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 7m 30s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 59s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 30s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 47s | | the patch passed |
| +1 :green_heart: | compile | 7m 55s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 7m 55s | | the patch passed |
| +1 :green_heart: | compile | 7m 33s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 7m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 54s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 1s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 22s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 43s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 15m 47s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 13s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 141m 36s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6240/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6240 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e8edd1dac598 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6f217d2c9bb795ee7b19ac333ddf5d5ac377c9d7 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6240/5/testReport/ |
| Max. process+thread count | 3152 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/h
[PR] YARN-11617. Remove noisy log in SingleConstraintAppPlacementAllocator [hadoop]
zuston opened a new pull request, #6283: URL: https://github.com/apache/hadoop/pull/6283 ### Description of PR Too many noisy log in SingleConstraintAppPlacementAllocator like that: ``` 2023-11-20 15:14:30,493 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.SingleConstraintAppPlacementAllocator: Successfully added SchedulingRequest to app=appattempt_1700464328807_0002_01 placementConstraint=[ node,EQ,nm.yarn.io/lifecycle=[reserved:true]]. nodePartition= ``` ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18981. moved oncrpc/portmap from hadoop-common-project/hadoop-nfs to hadoop-common-project/hadoop-common [hadoop]
hadoop-yetus commented on PR #6280:
URL: https://github.com/apache/hadoop/pull/6280#issuecomment-1818574211
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 2s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 22 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 4s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 35m 13s | | trunk passed |
| +1 :green_heart: | compile | 18m 3s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 16m 31s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 15s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 51s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 26s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 37m 50s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 17m 10s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 17m 10s | | the patch passed |
| +1 :green_heart: | compile | 16m 16s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 16m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 19s | | hadoop-common-project:
The patch generated 0 new + 100 unchanged - 2 fixed = 100 total (was 102) |
| +1 :green_heart: | mvnsite | 2m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 45s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 26s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 37m 57s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 11s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 0m 40s | | hadoop-nfs in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not
generate ASF License warnings. |
| | | 243m 24s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6280/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6280 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d3695b4c0f33 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8b2bb0a199f0d28b8a887a40030ff2d1290c7ff8 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6280/3/testReport/ |
| Max. process+thread count | 1261 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common
hadoop-common-project/hadoop-nfs U: hadoop-common-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6280/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This messa
[jira] [Commented] (HADOOP-18981) Move oncrpc/portmap from hadoop-nfs to hadoop-common
[
https://issues.apache.org/jira/browse/HADOOP-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787901#comment-17787901
]
ASF GitHub Bot commented on HADOOP-18981:
-
hadoop-yetus commented on PR #6280:
URL: https://github.com/apache/hadoop/pull/6280#issuecomment-1818574211
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 2s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 22 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 4s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 35m 13s | | trunk passed |
| +1 :green_heart: | compile | 18m 3s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 16m 31s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 15s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 51s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 26s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 37m 50s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 17m 10s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 17m 10s | | the patch passed |
| +1 :green_heart: | compile | 16m 16s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 16m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 19s | | hadoop-common-project:
The patch generated 0 new + 100 unchanged - 2 fixed = 100 total (was 102) |
| +1 :green_heart: | mvnsite | 2m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 45s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 26s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 37m 57s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 11s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 0m 40s | | hadoop-nfs in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not
generate ASF License warnings. |
| | | 243m 24s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6280/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6280 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d3695b4c0f33 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8b2bb0a199f0d28b8a887a40030ff2d1290c7ff8 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6280/3/testReport/ |
| Max. process+thread count | 1261 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common
hadoop-common-project/hadoop-nfs U: hado
Re: [PR] YARN-11616. Fast fail for NodeConstraintParser when having multi attribute kvs [hadoop]
hadoop-yetus commented on PR #6282:
URL: https://github.com/apache/hadoop/pull/6282#issuecomment-1818647088
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 7m 13s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 10s | | trunk passed |
| +1 :green_heart: | compile | 0m 30s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 19s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 29s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 21s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 14s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 52s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 23s | | the patch passed |
| +1 :green_heart: | compile | 0m 19s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 10s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 21s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 33s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 34s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 22s | | The patch does not
generate ASF License warnings. |
| | | 86m 47s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6282/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6282 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux fd7270686a4c 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bbfd47a1f559ac60a37a260512af55e071db0ca7 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6282/1/testReport/ |
| Max. process+thread count | 560 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6282/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please cont
Re: [PR] HDFS-17223. Add journalnode maintenance node list [hadoop]
hadoop-yetus commented on PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1818691838
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 16m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 48m 33s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 14s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/10/artifact/out/blanks-eol.txt)
| The patch has 6 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 1m 4s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/10/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 8 new + 322 unchanged
- 1 fixed = 330 total (was 323) |
| +1 :green_heart: | mvnsite | 1m 16s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 30s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 0s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 248m 28s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/10/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 417m 55s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6183 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 5b0c9ebc7458 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / dc94c93398be3a6559898faa377dc7b8f9091244 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Res
[jira] [Commented] (HADOOP-18980) S3A credential provider remapping: make extensible
[ https://issues.apache.org/jira/browse/HADOOP-18980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787914#comment-17787914 ] Steve Loughran commented on HADOOP-18980: - exactly; though i'd expect the remapping to be from com.amazonaws to software.amazonaws or private implementations key goal: you can use the same credentials.provider list for v1 and v2 sdk clients. > S3A credential provider remapping: make extensible > -- > > Key: HADOOP-18980 > URL: https://issues.apache.org/jira/browse/HADOOP-18980 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Priority: Minor > > s3afs will now remap the common com.amazonaws credential providers to > equivalents in the v2 sdk or in hadoop-aws > We could do the same for third party credential providers by taking a > key=value list in a configuration property and adding to the map. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17250. EditLogTailer#triggerActiveLogRoll should handle thread Interrupted [hadoop]
haiyang1987 commented on PR #6266: URL: https://github.com/apache/hadoop/pull/6266#issuecomment-1818872859 Thanks @xinglin @ZanderXu for your detailed review. > > when we submit a task to an executorService and later on we cancel it due to timeout, and we don't capture interrupted exception in our task, what state will the thread from executorService be left? Do we have to capture interrupted exception in our task in this case? > > This is very nice question, you can refer to the source code of `ThreadPoolExecutor#Worker#runWorker()`, the worker in the `ThreadPoolExecutor` will clear the `Interrupt` flag. Yeah, totally agree with this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18975. AWS SDK v2: extend support for FIPS endpoints [hadoop]
steveloughran commented on PR #6277: URL: https://github.com/apache/hadoop/pull/6277#issuecomment-1818873757 i'll say ok to those tests, we just need to make sure that they work/can be disabled during third-party testing. proposed 1. add some test.fips.bucket and test.fips.region options; 2. create an s3a client with auth set to anon (maybe we need a new test.fips.credentials) optin 3. do the tests you've proposed. can you point me at the interceptor stuff to understand how that could be used? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17250. EditLogTailer#triggerActiveLogRoll should handle thread Interrupted [hadoop]
haiyang1987 commented on PR #6266: URL: https://github.com/apache/hadoop/pull/6266#issuecomment-1818874038 > `InterruptedIOException` is a subClass of `IOException`. In the try_catch block in `getActiveNodeProxy`, we do catch for `IOException`. So, interruptedIOException will be captured there. > > https://docs.oracle.com/javase/8/docs/api/java/io/InterruptedIOException.html > > What I think should be happening is as following. > > ``` > main_Thread calling `triggerActiveLogRoll`, wait for 60 secs, timeout, cancel this task, and return. > > MultipleNameNodeProxy.call() thread: >-> getActiveNodeProxy() > -> nnLookup.next = ob2 (down node) > -> RPC.waitForProxy(ob2) > -> after 60 secs, interrupted. > -> output "Failed to reach ob2", increment nnLoopCount. Ideally, we should just stop here, since we already time out. > -> nnLookup.next = n1 (live node). > then, it should succeed to connect to n1. > ``` > > This does not seem to be the case from the logs you shared. thoughts? > > A possible fix might be to explicitly capture `InterruptedIOException` in `getActiveNodeProxy`, and just finish for this thread (assuming all `InterruptedIOExceptions` are invoked from `triggerActiveLogRoll`). For the following triggerActiveLogRoll calls, we should be good, since we will move the nnLookup to next one. Yeah, For MultipleNameNodeProxy#call() explicitly capture InterruptedIOException and then exit execution is also a solution -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18969. S3A: AbstractS3ACostTest to clear bucket fs.s3a.create.performance [hadoop]
steveloughran commented on PR #6264: URL: https://github.com/apache/hadoop/pull/6264#issuecomment-1818874863 Can I get a quick review of this @mukund-thakur @ahmarsuhail @HarshitGupta11 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18969) S3A: AbstractS3ACostTest to clear bucket fs.s3a.create.performance flag
[ https://issues.apache.org/jira/browse/HADOOP-18969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787959#comment-17787959 ] ASF GitHub Bot commented on HADOOP-18969: - steveloughran commented on PR #6264: URL: https://github.com/apache/hadoop/pull/6264#issuecomment-1818874863 Can I get a quick review of this @mukund-thakur @ahmarsuhail @HarshitGupta11 > S3A: AbstractS3ACostTest to clear bucket fs.s3a.create.performance flag > > > Key: HADOOP-18969 > URL: https://issues.apache.org/jira/browse/HADOOP-18969 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3, test >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Priority: Minor > Labels: pull-request-available > > If there's a bucket-specific fs.s3a.create.performance flag then the create > tests can fail as the costs are lower than expected. > trivial fix: add to the removeBaseAndBucketOverrides list -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18975) AWS SDK v2: extend support for FIPS endpoints
[ https://issues.apache.org/jira/browse/HADOOP-18975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787957#comment-17787957 ] ASF GitHub Bot commented on HADOOP-18975: - steveloughran commented on PR #6277: URL: https://github.com/apache/hadoop/pull/6277#issuecomment-1818873757 i'll say ok to those tests, we just need to make sure that they work/can be disabled during third-party testing. proposed 1. add some test.fips.bucket and test.fips.region options; 2. create an s3a client with auth set to anon (maybe we need a new test.fips.credentials) optin 3. do the tests you've proposed. can you point me at the interceptor stuff to understand how that could be used? > AWS SDK v2: extend support for FIPS endpoints > -- > > Key: HADOOP-18975 > URL: https://issues.apache.org/jira/browse/HADOOP-18975 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > > v1 SDK supported FIPS just by changing the endpoint. > Now we have a new builder setting to use. > * add new fs.s3a.endpoint.fips option > * pass it down > * test -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18975. AWS SDK v2: extend support for FIPS endpoints [hadoop]
ahmarsuhail commented on PR #6277: URL: https://github.com/apache/hadoop/pull/6277#issuecomment-1818889613 It's[ here](https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java#L221) in ITestS3AEndpointRegion.java . The interceptor already has an [assertion for endpoint](https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java#L239) so hopefully shouldn't be too much work to add the new tests in. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18975) AWS SDK v2: extend support for FIPS endpoints
[ https://issues.apache.org/jira/browse/HADOOP-18975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787963#comment-17787963 ] ASF GitHub Bot commented on HADOOP-18975: - ahmarsuhail commented on PR #6277: URL: https://github.com/apache/hadoop/pull/6277#issuecomment-1818889613 It's[ here](https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java#L221) in ITestS3AEndpointRegion.java . The interceptor already has an [assertion for endpoint](https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java#L239) so hopefully shouldn't be too much work to add the new tests in. > AWS SDK v2: extend support for FIPS endpoints > -- > > Key: HADOOP-18975 > URL: https://issues.apache.org/jira/browse/HADOOP-18975 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > > v1 SDK supported FIPS just by changing the endpoint. > Now we have a new builder setting to use. > * add new fs.s3a.endpoint.fips option > * pass it down > * test -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17218. NameNode should process time out excess redundancy blocks [hadoop]
hadoop-yetus commented on PR #6176:
URL: https://github.com/apache/hadoop/pull/6176#issuecomment-1818909919
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 53s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 54s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 4s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 32s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 29s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 253m 16s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/13/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 404m 59s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/13/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6176 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 67b6b0c7dd15 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b7f9b236cd9e38657ac2b64d215ec688320c8f6b |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/13/testReport/ |
| Max. process+thread count | 2471 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6176/13/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org
Re: [PR] HADOOP-18959 Use builder for prefetch CachingBlockManager [hadoop]
ahmarsuhail commented on code in PR #6240:
URL: https://github.com/apache/hadoop/pull/6240#discussion_r1399090061
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/BlockManagerParameters.java:
##
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.fs.impl.prefetch;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.LocalDirAllocator;
+import org.apache.hadoop.fs.statistics.DurationTrackerFactory;
+
+/**
+ * This class is used to provide parameters to {@link BlockManager}.
+ */
+@InterfaceAudience.Private
+public final class BlockManagerParameters {
+
+ /**
+ * Asynchronous tasks are performed in this pool.
+ */
+ private ExecutorServiceFuturePool futurePool;
+
+ /**
+ * Information about each block of the underlying file.
+ */
+ private BlockData blockData;
+
+ /**
+ * Size of the in-memory cache in terms of number of blocks.
+ */
+ private int bufferPoolSize;
+
+ /**
+ * Statistics for the stream.
+ */
+ private PrefetchingStatistics prefetchingStatistics;
+
+ /**
+ * The configuration object.
+ */
+ private Configuration conf;
+
+ /**
+ * The local dir allocator instance.
+ */
+ private LocalDirAllocator localDirAllocator;
+
+ /**
+ * Max blocks count to be kept in cache at any time.
+ */
+ private int maxBlocksCount;
+
+ /**
+ * Tracker with statistics to update.
+ */
+ private DurationTrackerFactory trackerFactory;
+
+ public ExecutorServiceFuturePool getFuturePool() {
+return futurePool;
+ }
+
+ public BlockData getBlockData() {
+return blockData;
+ }
+
+ public int getBufferPoolSize() {
+return bufferPoolSize;
+ }
+
+ public PrefetchingStatistics getPrefetchingStatistics() {
+return prefetchingStatistics;
+ }
+
+ public Configuration getConf() {
+return conf;
+ }
+
+ public LocalDirAllocator getLocalDirAllocator() {
+return localDirAllocator;
+ }
+
+ public int getMaxBlocksCount() {
+return maxBlocksCount;
+ }
+
+ public DurationTrackerFactory getTrackerFactory() {
+return trackerFactory;
+ }
+
+ public BlockManagerParameters setFuturePool(
Review Comment:
could you rename the setters to "withFuturePool" instead of set, keeps
things uniform with other builders in the project.
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/prefetch/TestS3ACachingBlockManager.java:
##
@@ -61,50 +65,93 @@ public class TestS3ACachingBlockManager extends
AbstractHadoopTestBase {
private final BlockData blockData = new BlockData(FILE_SIZE, BLOCK_SIZE);
+ private static final Configuration CONF =
+ S3ATestUtils.prepareTestConfiguration(new Configuration());
+
@Test
public void testArgChecks() throws Exception {
MockS3ARemoteObject s3File = new MockS3ARemoteObject(FILE_SIZE, false);
S3ARemoteObjectReader reader = new S3ARemoteObjectReader(s3File);
Configuration conf = new Configuration();
+BlockManagerParameters blockManagerParamsBuilder1 =
Review Comment:
with the builder, this test is getting really long now. How about we split
it into separate tests? One test for the valid blockManager and the assertions
on it that are done. And then split each of the illegal arg ones into a
separate test? eg: testNullFuturePool(), etc.
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/BlockManagerParameters.java:
##
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS,
[jira] [Commented] (HADOOP-18959) Use builder for prefetch CachingBlockManager
[
https://issues.apache.org/jira/browse/HADOOP-18959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787983#comment-17787983
]
ASF GitHub Bot commented on HADOOP-18959:
-
ahmarsuhail commented on code in PR #6240:
URL: https://github.com/apache/hadoop/pull/6240#discussion_r1399090061
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/BlockManagerParameters.java:
##
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.fs.impl.prefetch;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.LocalDirAllocator;
+import org.apache.hadoop.fs.statistics.DurationTrackerFactory;
+
+/**
+ * This class is used to provide parameters to {@link BlockManager}.
+ */
+@InterfaceAudience.Private
+public final class BlockManagerParameters {
+
+ /**
+ * Asynchronous tasks are performed in this pool.
+ */
+ private ExecutorServiceFuturePool futurePool;
+
+ /**
+ * Information about each block of the underlying file.
+ */
+ private BlockData blockData;
+
+ /**
+ * Size of the in-memory cache in terms of number of blocks.
+ */
+ private int bufferPoolSize;
+
+ /**
+ * Statistics for the stream.
+ */
+ private PrefetchingStatistics prefetchingStatistics;
+
+ /**
+ * The configuration object.
+ */
+ private Configuration conf;
+
+ /**
+ * The local dir allocator instance.
+ */
+ private LocalDirAllocator localDirAllocator;
+
+ /**
+ * Max blocks count to be kept in cache at any time.
+ */
+ private int maxBlocksCount;
+
+ /**
+ * Tracker with statistics to update.
+ */
+ private DurationTrackerFactory trackerFactory;
+
+ public ExecutorServiceFuturePool getFuturePool() {
+return futurePool;
+ }
+
+ public BlockData getBlockData() {
+return blockData;
+ }
+
+ public int getBufferPoolSize() {
+return bufferPoolSize;
+ }
+
+ public PrefetchingStatistics getPrefetchingStatistics() {
+return prefetchingStatistics;
+ }
+
+ public Configuration getConf() {
+return conf;
+ }
+
+ public LocalDirAllocator getLocalDirAllocator() {
+return localDirAllocator;
+ }
+
+ public int getMaxBlocksCount() {
+return maxBlocksCount;
+ }
+
+ public DurationTrackerFactory getTrackerFactory() {
+return trackerFactory;
+ }
+
+ public BlockManagerParameters setFuturePool(
Review Comment:
could you rename the setters to "withFuturePool" instead of set, keeps
things uniform with other builders in the project.
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/prefetch/TestS3ACachingBlockManager.java:
##
@@ -61,50 +65,93 @@ public class TestS3ACachingBlockManager extends
AbstractHadoopTestBase {
private final BlockData blockData = new BlockData(FILE_SIZE, BLOCK_SIZE);
+ private static final Configuration CONF =
+ S3ATestUtils.prepareTestConfiguration(new Configuration());
+
@Test
public void testArgChecks() throws Exception {
MockS3ARemoteObject s3File = new MockS3ARemoteObject(FILE_SIZE, false);
S3ARemoteObjectReader reader = new S3ARemoteObjectReader(s3File);
Configuration conf = new Configuration();
+BlockManagerParameters blockManagerParamsBuilder1 =
Review Comment:
with the builder, this test is getting really long now. How about we split
it into separate tests? One test for the valid blockManager and the assertions
on it that are done. And then split each of the illegal arg ones into a
separate test? eg: testNullFuturePool(), etc.
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/BlockManagerParameters.java:
##
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *
Re: [PR] YARN-11617. Remove noisy log in SingleConstraintAppPlacementAllocator [hadoop]
slfan1989 commented on code in PR #6283:
URL: https://github.com/apache/hadoop/pull/6283#discussion_r1399159027
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/SingleConstraintAppPlacementAllocator.java:
##
@@ -225,11 +225,13 @@ private void
validateAndSetSchedulingRequest(SchedulingRequest
((SchedulingRequestPBImpl) newSchedulingRequest).getProto());
-LOG.info("Successfully added SchedulingRequest to app="
-+ appSchedulingInfo.getApplicationAttemptId()
-+ " placementConstraint=["
-+ schedulingRequest.getPlacementConstraint()
-+ "]. nodePartition=" + targetNodePartition);
+if (LOG.isDebugEnabled()) {
Review Comment:
We should not need to add this condition.
```
LOG.debug("Successfully added SchedulingRequest to app={},
placementConstraint={}. nodePartition={}",
appSchedulingInfo.getApplicationAttemptId(),
schedulingRequest.getPlacementConstraint(), targetNodePartition)");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] YARN-11577. Improve FederationInterceptorREST Method Result. [hadoop]
slfan1989 merged PR #6190: URL: https://github.com/apache/hadoop/pull/6190 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] YARN-11577. Improve FederationInterceptorREST Method Result. [hadoop]
slfan1989 commented on PR #6190: URL: https://github.com/apache/hadoop/pull/6190#issuecomment-1819015330 @goiri Thank you very much for your help reviewing the code! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] YARN-11617. Remove noisy log in SingleConstraintAppPlacementAllocator [hadoop]
hadoop-yetus commented on PR #6283:
URL: https://github.com/apache/hadoop/pull/6283#issuecomment-1819088870
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 14m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 48m 43s | | trunk passed |
| +1 :green_heart: | compile | 1m 2s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 54s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 58s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 46s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 47m 26s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 49s | | the patch passed |
| +1 :green_heart: | compile | 0m 54s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 54s | | the patch passed |
| +1 :green_heart: | compile | 0m 47s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 47s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 45s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 49s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 43s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 40s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 25s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 119m 8s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 285m 19s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6283/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6283 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux af362ea63e90 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ae0957498603e62a51af4369ca6fe83c016b847a |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6283/1/testReport/ |
| Max. process+thread count | 884 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6283/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Se
[jira] [Created] (HADOOP-18983) S3A: remove @deprecated tags where no longer needed
Steve Loughran created HADOOP-18983:
---
Summary: S3A: remove @deprecated tags where no longer needed
Key: HADOOP-18983
URL: https://issues.apache.org/jira/browse/HADOOP-18983
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.4.0
Reporter: Steve Loughran
{{S3ClientFactory}} is tagged as deprecated "to be replaced" when we just
changed it instead. probably the same elsewhere so scan for the tag and remove
as appropriate
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18984) S3A: Support dynamic region resolution
Steve Loughran created HADOOP-18984: --- Summary: S3A: Support dynamic region resolution Key: HADOOP-18984 URL: https://issues.apache.org/jira/browse/HADOOP-18984 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.4.0 Reporter: Steve Loughran Assignee: Steve Loughran AWS v1 SDK used to do dynamic region inference including EC2 metadata. See HADOOP-17771 for that at work with the chain of: sysprops, env vars, EC2 metadata. I think we *try* to do this, but I'm not sure it is working, maybe related to the region=null vs region="" options: its very hard to differentiate the two -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18931. FileSystem.getFileSystemClass() to log at debug the jar the .class came from [hadoop]
virajjasani commented on PR #6197: URL: https://github.com/apache/hadoop/pull/6197#issuecomment-1819539985 It is assertj: `org.assertj.core.api.Assertions` (unless i misinterpreted something) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18931) FileSystem.getFileSystemClass() to log at debug the jar the .class came from
[ https://issues.apache.org/jira/browse/HADOOP-18931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788110#comment-17788110 ] ASF GitHub Bot commented on HADOOP-18931: - virajjasani commented on PR #6197: URL: https://github.com/apache/hadoop/pull/6197#issuecomment-1819539985 It is assertj: `org.assertj.core.api.Assertions` (unless i misinterpreted something) > FileSystem.getFileSystemClass() to log at debug the jar the .class came from > > > Key: HADOOP-18931 > URL: https://issues.apache.org/jira/browse/HADOOP-18931 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 3.3.6 >Reporter: Steve Loughran >Priority: Minor > Labels: pull-request-available > > we want to be able to log the jar the filesystem implementation class, so > that we can identify which version of a module the class came from. > this is to help track down problems where different machines in the cluster > or the .tar.gz bundle is out of date. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18957. Use StandardCharsets.UTF_8 [hadoop]
ayushtkn merged PR #6231: URL: https://github.com/apache/hadoop/pull/6231 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18957) Use StandardCharsets.UTF_8 constant
[ https://issues.apache.org/jira/browse/HADOOP-18957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena reassigned HADOOP-18957: - Assignee: PJ Fanning > Use StandardCharsets.UTF_8 constant > --- > > Key: HADOOP-18957 > URL: https://issues.apache.org/jira/browse/HADOOP-18957 > Project: Hadoop Common > Issue Type: Improvement >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > > * there are some places in the code that have to check for > UnsupportedCharsetException when explicitly using the charset name "UTF-8" > * using StandardCharsets.UTF_8 is more efficient because the Java libs > usually have to look up the charsets when you provide it as String param > instead > * also stop using Guava Charsets and use StandardCharsets -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18957) Use StandardCharsets.UTF_8 constant
[ https://issues.apache.org/jira/browse/HADOOP-18957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788117#comment-17788117 ] ASF GitHub Bot commented on HADOOP-18957: - ayushtkn merged PR #6231: URL: https://github.com/apache/hadoop/pull/6231 > Use StandardCharsets.UTF_8 constant > --- > > Key: HADOOP-18957 > URL: https://issues.apache.org/jira/browse/HADOOP-18957 > Project: Hadoop Common > Issue Type: Improvement >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > > * there are some places in the code that have to check for > UnsupportedCharsetException when explicitly using the charset name "UTF-8" > * using StandardCharsets.UTF_8 is more efficient because the Java libs > usually have to look up the charsets when you provide it as String param > instead > * also stop using Guava Charsets and use StandardCharsets -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18957. Use StandardCharsets.UTF_8 [hadoop]
ayushtkn commented on PR #6231: URL: https://github.com/apache/hadoop/pull/6231#issuecomment-1819576205 Failed test passes locally ``` [INFO] --- [INFO] T E S T S [INFO] --- [INFO] Running org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl [INFO] Tests run: 33, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.9 s - in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl [INFO] [INFO] Results: [INFO] [INFO] Tests run: 33, Failures: 0, Errors: 0, Skipped: 0 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18957) Use StandardCharsets.UTF_8 constant
[ https://issues.apache.org/jira/browse/HADOOP-18957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HADOOP-18957. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Use StandardCharsets.UTF_8 constant > --- > > Key: HADOOP-18957 > URL: https://issues.apache.org/jira/browse/HADOOP-18957 > Project: Hadoop Common > Issue Type: Improvement >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > * there are some places in the code that have to check for > UnsupportedCharsetException when explicitly using the charset name "UTF-8" > * using StandardCharsets.UTF_8 is more efficient because the Java libs > usually have to look up the charsets when you provide it as String param > instead > * also stop using Guava Charsets and use StandardCharsets -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18957) Use StandardCharsets.UTF_8 constant
[ https://issues.apache.org/jira/browse/HADOOP-18957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788118#comment-17788118 ] ASF GitHub Bot commented on HADOOP-18957: - ayushtkn commented on PR #6231: URL: https://github.com/apache/hadoop/pull/6231#issuecomment-1819576205 Failed test passes locally ``` [INFO] --- [INFO] T E S T S [INFO] --- [INFO] Running org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl [INFO] Tests run: 33, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.9 s - in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl [INFO] [INFO] Results: [INFO] [INFO] Tests run: 33, Failures: 0, Errors: 0, Skipped: 0 ``` > Use StandardCharsets.UTF_8 constant > --- > > Key: HADOOP-18957 > URL: https://issues.apache.org/jira/browse/HADOOP-18957 > Project: Hadoop Common > Issue Type: Improvement >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > > * there are some places in the code that have to check for > UnsupportedCharsetException when explicitly using the charset name "UTF-8" > * using StandardCharsets.UTF_8 is more efficient because the Java libs > usually have to look up the charsets when you provide it as String param > instead > * also stop using Guava Charsets and use StandardCharsets -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18957) Use StandardCharsets.UTF_8 constant
[ https://issues.apache.org/jira/browse/HADOOP-18957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788119#comment-17788119 ] Ayush Saxena commented on HADOOP-18957: --- Committed to trunk. Thanx [~pj.fanning] for the contribution!!! > Use StandardCharsets.UTF_8 constant > --- > > Key: HADOOP-18957 > URL: https://issues.apache.org/jira/browse/HADOOP-18957 > Project: Hadoop Common > Issue Type: Improvement >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > > * there are some places in the code that have to check for > UnsupportedCharsetException when explicitly using the charset name "UTF-8" > * using StandardCharsets.UTF_8 is more efficient because the Java libs > usually have to look up the charsets when you provide it as String param > instead > * also stop using Guava Charsets and use StandardCharsets -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18972) Bug in SaslPropertiesResolver allows mutation of internal state
[
https://issues.apache.org/jira/browse/HADOOP-18972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charles Connell updated HADOOP-18972:
-
Description:
When {{SaslDataTransferServer}} or {{SaslDataTranferClient}} want to get a SASL
properties map to do a handshake, they call
{{SaslPropertiesResolver#getServerProperties()}} or
{{SaslPropertiesResolver#getClientProperties()}}, and they get back a
{{Map}}. Every call gets the same {{Map}} object back, and then
the callers sometimes call
[put()|https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferServer.java#L385]
on it. This means that future users of {{SaslPropertiesResolver}} get back the
wrong information.
I propose that {{SaslPropertiesResolver}} should pass a copy of its internal
map, so that users can safety modify them.
I discovered this problem in my company's testing environment as we began to
enable {{dfs.data.transfer.protection}} on our DataNodes, while our NameNodes
were using {{IngressPortBasedResolver}} to give out block tokens with different
QOPs depending on the port used. Then our HDFS client applications became
unable to read or write to HDFS because they could not find a QOP in common
with the DataNodes during SASL handshake. With multiple threads executing SASL
handshakes at the same time, the properties map used in
{{SaslDataTransferServer}} in a DataNode could be clobbered during usage, since
the same map was used by all threads. Also, future clients that do not have a
QOP embedded in their block tokens would connect to a server with the wrong
SASL properties map. I think that one or both of these issues explains the
problem that I saw. I eliminated this unsafety and saw the problem go away.
was:
When {{SaslDataTransferServer}} or {{SaslDataTranferClient}} want to get a SASL
properties map to do a handshake, they call
{{SaslPropertiesResolver#getServerProperties()}} or
{{SaslPropertiesResolver#getClientProperties()}}, and they get back a
{{Map}}. Every call gets the same {{Map}} object back, and then
the callers sometimes call
[put()|https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferServer.java#L385]
on it. This means that future users of {{SaslPropertiesResolver}} get back the
wrong information.
I propose that {{SaslPropertiesResolver}} should pass a copy of its internal
map, so that users can safety modify them.
I discovered this problem in my company's testing environment as we began to
enable {{dfs.data.transfer.protection}} on our DataNodes, while our NameNodes
were using {{IngressPortBasedResolver}} to give out block tokens with different
QOPs depending on the port used. The our HDFS client applications became unable
to read or write to HDFS because they could not find a QOP in common with the
DataNodes during SASL handshake. With multiple threads executing SASL
handshakes at the same time, the properties map used in
{{SaslDataTransferServer}} in a DataNode could be clobbered during usage, since
the same map was used by all threads. Also, future clients that do not have a
QOP embedded in their block tokens would connect to a server with the wrong
SASL properties map. I think that one or both of these issues explains the
problem that I saw. I eliminated this unsafety and saw the problem go away.
> Bug in SaslPropertiesResolver allows mutation of internal state
> ---
>
> Key: HADOOP-18972
> URL: https://issues.apache.org/jira/browse/HADOOP-18972
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Charles Connell
>Priority: Minor
> Labels: pull-request-available
>
> When {{SaslDataTransferServer}} or {{SaslDataTranferClient}} want to get a
> SASL properties map to do a handshake, they call
> {{SaslPropertiesResolver#getServerProperties()}} or
> {{SaslPropertiesResolver#getClientProperties()}}, and they get back a
> {{Map}}. Every call gets the same {{Map}} object back, and
> then the callers sometimes call
> [put()|https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferServer.java#L385]
> on it. This means that future users of {{SaslPropertiesResolver}} get back
> the wrong information.
> I propose that {{SaslPropertiesResolver}} should pass a copy of its internal
> map, so that users can safety modify them.
> I discovered this problem in my company's testing environment as we began to
> enable {{dfs.data.transfer.protection}} on our DataNodes, while our NameNodes
> were using {{IngressPortBasedResolver}} to give o
[jira] [Comment Edited] (HADOOP-18980) S3A credential provider remapping: make extensible
[
https://issues.apache.org/jira/browse/HADOOP-18980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788128#comment-17788128
]
Viraj Jasani edited comment on HADOOP-18980 at 11/20/23 6:44 PM:
-
In addition to having same credentials.provider list for v1 and v2 sdk, maybe
we can also remove static mapping for v1 to v2 credential providers and let new
config have default key value pairs:
{code:java}
fs.s3a.aws.credentials.provider.mapping
com.amazonaws.auth.AnonymousAWSCredentials=org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider,
com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper=org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider,
com.amazonaws.auth.InstanceProfileCredentialsProvider=org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider,
com.amazonaws.auth.EnvironmentVariableCredentialsProvider=software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider,
com.amazonaws.auth.profile.ProfileCredentialsProvider=software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider
{code}
With this being default value, any new third-party credential provider can be
added to this list by users. Does that sound good?
was (Author: vjasani):
{quote}exactly; though i'd expect the remapping to be from com.amazonaws to
software.amazonaws or private implementations
key goal: you can use the same credentials.provider list for v1 and v2 sdk
clients.
{quote}
In addition to having same credentials.provider list for v1 and v2 sdk, maybe
we can also remove static mapping for v1 to v2 credential providers and let new
config have default key value pairs:
{code:java}
fs.s3a.aws.credentials.provider.mapping
com.amazonaws.auth.AnonymousAWSCredentials=org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider,
com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper=org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider,
com.amazonaws.auth.InstanceProfileCredentialsProvider=org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider,
com.amazonaws.auth.EnvironmentVariableCredentialsProvider=software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider,
com.amazonaws.auth.profile.ProfileCredentialsProvider=software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider
{code}
With this being default value, any new third-party credential provider can be
added to this list by users. Does that sound good?
> S3A credential provider remapping: make extensible
> --
>
> Key: HADOOP-18980
> URL: https://issues.apache.org/jira/browse/HADOOP-18980
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Priority: Minor
>
> s3afs will now remap the common com.amazonaws credential providers to
> equivalents in the v2 sdk or in hadoop-aws
> We could do the same for third party credential providers by taking a
> key=value list in a configuration property and adding to the map.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18980) S3A credential provider remapping: make extensible
[
https://issues.apache.org/jira/browse/HADOOP-18980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788128#comment-17788128
]
Viraj Jasani commented on HADOOP-18980:
---
{quote}exactly; though i'd expect the remapping to be from com.amazonaws to
software.amazonaws or private implementations
key goal: you can use the same credentials.provider list for v1 and v2 sdk
clients.
{quote}
In addition to having same credentials.provider list for v1 and v2 sdk, maybe
we can also remove static mapping for v1 to v2 credential providers and let new
config have default key value pairs:
{code:java}
fs.s3a.aws.credentials.provider.mapping
com.amazonaws.auth.AnonymousAWSCredentials=org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider,
com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper=org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider,
com.amazonaws.auth.InstanceProfileCredentialsProvider=org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider,
com.amazonaws.auth.EnvironmentVariableCredentialsProvider=software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider,
com.amazonaws.auth.profile.ProfileCredentialsProvider=software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider
{code}
With this being default value, any new third-party credential provider can be
added to this list by users. Does that sound good?
> S3A credential provider remapping: make extensible
> --
>
> Key: HADOOP-18980
> URL: https://issues.apache.org/jira/browse/HADOOP-18980
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Priority: Minor
>
> s3afs will now remap the common com.amazonaws credential providers to
> equivalents in the v2 sdk or in hadoop-aws
> We could do the same for third party credential providers by taking a
> key=value list in a configuration property and adding to the map.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18981. moved oncrpc/portmap from hadoop-common-project/hadoop-nfs to hadoop-common-project/hadoop-common [hadoop]
xinglin commented on PR #6280: URL: https://github.com/apache/hadoop/pull/6280#issuecomment-1819853710 @goiri / @ZanderXu, can I get a review? thanks, -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18981) Move oncrpc/portmap from hadoop-nfs to hadoop-common
[ https://issues.apache.org/jira/browse/HADOOP-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788161#comment-17788161 ] ASF GitHub Bot commented on HADOOP-18981: - xinglin commented on PR #6280: URL: https://github.com/apache/hadoop/pull/6280#issuecomment-1819853710 @goiri / @ZanderXu, can I get a review? thanks, > Move oncrpc/portmap from hadoop-nfs to hadoop-common > > > Key: HADOOP-18981 > URL: https://issues.apache.org/jira/browse/HADOOP-18981 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.4.0 >Reporter: Xing Lin >Assignee: Xing Lin >Priority: Major > Labels: pull-request-available > > We want to use udpserver/client for other use cases, rather than only for > NFS. One such use case is to export NameNodeHAState for NameNodes via a UDP > server. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18985) Reversed (or previously applied) patch detected for hadoop-common-project/hadoop-common/dev-support/jdiff-workaround.patch
Xing Lin created HADOOP-18985:
-
Summary: Reversed (or previously applied) patch detected for
hadoop-common-project/hadoop-common/dev-support/jdiff-workaround.patch
Key: HADOOP-18985
URL: https://issues.apache.org/jira/browse/HADOOP-18985
Project: Hadoop Common
Issue Type: Improvement
Reporter: Xing Lin
tried to compile under Hadoop-common-project and noticed the following.
{code:java}
$ mvn clean package -Pdist,docs -DskipTests -Dtar
patching file
'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java'
Reversed (or previously applied) patch detected! Assume -R? [y]
1 out of 2 hunks failed--saving rejects to
'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java.rej'
patching file
'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java'
Reversed (or previously applied) patch detected! Assume -R? [y]
patching file
'hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestGangliaMetrics.java{code}
also reproducible by
{code:java}
xinglin@xinglin-mn2 ~/p/h/trunk (trunk)> patch -p1 <
hadoop-common-project/hadoop-common/dev-support/jdiff-workaround.patch
patching file
'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java'
Reversed (or previously applied) patch detected! Assume -R? [y]
1 out of 2 hunks failed--saving rejects to
'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java.rej'
patching file
'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java'
Reversed (or previously applied) patch detected! Assume -R? [y]
patching file
'hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestGangliaMetrics.java'{code}
Build succeeded but the jdiff-workaround patch probably needs a fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18985) Reversed (or previously applied) patch detected for hadoop-common-project/hadoop-common/dev-support/jdiff-workaround.patch
[
https://issues.apache.org/jira/browse/HADOOP-18985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xing Lin updated HADOOP-18985:
--
Description:
tried to compile under Hadoop-common-project and noticed the following.
trunk: commit f609460bda0c2bd87dd3580158e549e2f34f14d5
{code:java}
$ mvn clean package -Pdist,docs -DskipTests -Dtar
patching file
'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java'
Reversed (or previously applied) patch detected! Assume -R? [y]
1 out of 2 hunks failed--saving rejects to
'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java.rej'
patching file
'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java'
Reversed (or previously applied) patch detected! Assume -R? [y]
patching file
'hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestGangliaMetrics.java{code}
also reproducible by
{code:java}
xinglin@xinglin-mn2 ~/p/h/trunk (trunk)> patch -p1 <
hadoop-common-project/hadoop-common/dev-support/jdiff-workaround.patch
patching file
'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java'
Reversed (or previously applied) patch detected! Assume -R? [y]
1 out of 2 hunks failed--saving rejects to
'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java.rej'
patching file
'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java'
Reversed (or previously applied) patch detected! Assume -R? [y]
patching file
'hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestGangliaMetrics.java'{code}
Build succeeded but the jdiff-workaround patch probably needs a fix.
was:
tried to compile under Hadoop-common-project and noticed the following.
{code:java}
$ mvn clean package -Pdist,docs -DskipTests -Dtar
patching file
'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java'
Reversed (or previously applied) patch detected! Assume -R? [y]
1 out of 2 hunks failed--saving rejects to
'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java.rej'
patching file
'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java'
Reversed (or previously applied) patch detected! Assume -R? [y]
patching file
'hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestGangliaMetrics.java{code}
also reproducible by
{code:java}
xinglin@xinglin-mn2 ~/p/h/trunk (trunk)> patch -p1 <
hadoop-common-project/hadoop-common/dev-support/jdiff-workaround.patch
patching file
'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java'
Reversed (or previously applied) patch detected! Assume -R? [y]
1 out of 2 hunks failed--saving rejects to
'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java.rej'
patching file
'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java'
Reversed (or previously applied) patch detected! Assume -R? [y]
patching file
'hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestGangliaMetrics.java'{code}
Build succeeded but the jdiff-workaround patch probably needs a fix.
> Reversed (or previously applied) patch detected for
> hadoop-common-project/hadoop-common/dev-support/jdiff-workaround.patch
> ---
>
> Key: HADOOP-18985
> URL: https://issues.apache.org/jira/browse/HADOOP-18985
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Xing Lin
>Priority: Minor
>
> tried to compile under Hadoop-common-project and noticed the following.
> trunk: commit f609460bda0c2bd87dd3580158e549e2f34f14d5
> {code:java}
> $ mvn clean package -Pdist,docs -DskipTests -Dtar
> patching file
> 'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java'
> Reversed (or previously applied) patch detected! Assume -R? [y]
> 1 out of 2 hunks failed--saving rejects to
> 'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java.rej'
> patching file
> 'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java'
> Reversed (or previously applied) patch detected! Assume -R? [y]
> patching file
> 'hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestGangliaMetrics.java{code}
> also reproducible by
> {code:java}
> xinglin@xinglin-mn2 ~/p/h/trunk (trunk)>
[jira] [Updated] (HADOOP-18985) Reversed (or previously applied) patch detected for hadoop-common-project/hadoop-common/dev-support/jdiff-workaround.patch
[
https://issues.apache.org/jira/browse/HADOOP-18985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xing Lin updated HADOOP-18985:
--
Target Version/s: 3.4.0, 3.3.9
Affects Version/s: 3.4.0
> Reversed (or previously applied) patch detected for
> hadoop-common-project/hadoop-common/dev-support/jdiff-workaround.patch
> ---
>
> Key: HADOOP-18985
> URL: https://issues.apache.org/jira/browse/HADOOP-18985
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.4.0
>Reporter: Xing Lin
>Priority: Minor
>
> tried to compile under Hadoop-common-project and noticed the following.
> trunk: commit f609460bda0c2bd87dd3580158e549e2f34f14d5
> {code:java}
> $ mvn clean package -Pdist,docs -DskipTests -Dtar
> patching file
> 'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java'
> Reversed (or previously applied) patch detected! Assume -R? [y]
> 1 out of 2 hunks failed--saving rejects to
> 'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java.rej'
> patching file
> 'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java'
> Reversed (or previously applied) patch detected! Assume -R? [y]
> patching file
> 'hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestGangliaMetrics.java{code}
> also reproducible by
> {code:java}
> xinglin@xinglin-mn2 ~/p/h/trunk (trunk)> patch -p1 <
> hadoop-common-project/hadoop-common/dev-support/jdiff-workaround.patch
> patching file
> 'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java'
> Reversed (or previously applied) patch detected! Assume -R? [y]
> 1 out of 2 hunks failed--saving rejects to
> 'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java.rej'
> patching file
> 'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java'
> Reversed (or previously applied) patch detected! Assume -R? [y]
> patching file
> 'hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestGangliaMetrics.java'{code}
> Build succeeded but the jdiff-workaround patch probably needs a fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18985) "Reversed (or previously applied) patch detected" for hadoop-common-project/hadoop-common/dev-support/jdiff-workaround.patch
[
https://issues.apache.org/jira/browse/HADOOP-18985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xing Lin updated HADOOP-18985:
--
Summary: "Reversed (or previously applied) patch detected" for
hadoop-common-project/hadoop-common/dev-support/jdiff-workaround.patch (was:
Reversed (or previously applied) patch detected for
hadoop-common-project/hadoop-common/dev-support/jdiff-workaround.patch )
> "Reversed (or previously applied) patch detected" for
> hadoop-common-project/hadoop-common/dev-support/jdiff-workaround.patch
> -
>
> Key: HADOOP-18985
> URL: https://issues.apache.org/jira/browse/HADOOP-18985
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.4.0
>Reporter: Xing Lin
>Priority: Minor
>
> tried to compile under Hadoop-common-project and noticed the following.
> trunk: commit f609460bda0c2bd87dd3580158e549e2f34f14d5
> {code:java}
> $ mvn clean package -Pdist,docs -DskipTests -Dtar
> patching file
> 'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java'
> Reversed (or previously applied) patch detected! Assume -R? [y]
> 1 out of 2 hunks failed--saving rejects to
> 'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java.rej'
> patching file
> 'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java'
> Reversed (or previously applied) patch detected! Assume -R? [y]
> patching file
> 'hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestGangliaMetrics.java{code}
> also reproducible by
> {code:java}
> xinglin@xinglin-mn2 ~/p/h/trunk (trunk)> patch -p1 <
> hadoop-common-project/hadoop-common/dev-support/jdiff-workaround.patch
> patching file
> 'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java'
> Reversed (or previously applied) patch detected! Assume -R? [y]
> 1 out of 2 hunks failed--saving rejects to
> 'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java.rej'
> patching file
> 'hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java'
> Reversed (or previously applied) patch detected! Assume -R? [y]
> patching file
> 'hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestGangliaMetrics.java'{code}
> Build succeeded but the jdiff-workaround patch probably needs a fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18959 S3A : Use builder for prefetch CachingBlockManager [hadoop]
virajjasani commented on code in PR #6240:
URL: https://github.com/apache/hadoop/pull/6240#discussion_r1399837597
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/BlockManagerParameters.java:
##
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.fs.impl.prefetch;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.LocalDirAllocator;
+import org.apache.hadoop.fs.statistics.DurationTrackerFactory;
+
+/**
+ * This class is used to provide parameters to {@link BlockManager}.
+ */
+@InterfaceAudience.Private
+public final class BlockManagerParameters {
+
+ /**
+ * Asynchronous tasks are performed in this pool.
+ */
+ private ExecutorServiceFuturePool futurePool;
+
+ /**
+ * Information about each block of the underlying file.
+ */
+ private BlockData blockData;
+
+ /**
+ * Size of the in-memory cache in terms of number of blocks.
+ */
+ private int bufferPoolSize;
+
+ /**
+ * Statistics for the stream.
+ */
+ private PrefetchingStatistics prefetchingStatistics;
+
+ /**
+ * The configuration object.
+ */
+ private Configuration conf;
+
+ /**
+ * The local dir allocator instance.
+ */
+ private LocalDirAllocator localDirAllocator;
+
+ /**
+ * Max blocks count to be kept in cache at any time.
+ */
+ private int maxBlocksCount;
+
+ /**
+ * Tracker with statistics to update.
+ */
+ private DurationTrackerFactory trackerFactory;
+
+ public ExecutorServiceFuturePool getFuturePool() {
+return futurePool;
+ }
+
+ public BlockData getBlockData() {
+return blockData;
+ }
+
+ public int getBufferPoolSize() {
+return bufferPoolSize;
+ }
+
+ public PrefetchingStatistics getPrefetchingStatistics() {
+return prefetchingStatistics;
+ }
+
+ public Configuration getConf() {
+return conf;
+ }
+
+ public LocalDirAllocator getLocalDirAllocator() {
+return localDirAllocator;
+ }
+
+ public int getMaxBlocksCount() {
+return maxBlocksCount;
+ }
+
+ public DurationTrackerFactory getTrackerFactory() {
+return trackerFactory;
+ }
+
+ public BlockManagerParameters setFuturePool(
Review Comment:
done
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/prefetch/TestS3ACachingBlockManager.java:
##
@@ -61,50 +65,93 @@ public class TestS3ACachingBlockManager extends
AbstractHadoopTestBase {
private final BlockData blockData = new BlockData(FILE_SIZE, BLOCK_SIZE);
+ private static final Configuration CONF =
+ S3ATestUtils.prepareTestConfiguration(new Configuration());
+
@Test
public void testArgChecks() throws Exception {
MockS3ARemoteObject s3File = new MockS3ARemoteObject(FILE_SIZE, false);
S3ARemoteObjectReader reader = new S3ARemoteObjectReader(s3File);
Configuration conf = new Configuration();
+BlockManagerParameters blockManagerParamsBuilder1 =
Review Comment:
done
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/prefetch/TestS3ACachingBlockManager.java:
##
@@ -244,9 +292,18 @@ private void testPrefetchHelper(boolean
forcePrefetchFailure)
throws IOException, InterruptedException {
MockS3ARemoteObject s3File = new MockS3ARemoteObject(FILE_SIZE, false);
S3ARemoteObjectReader reader = new S3ARemoteObjectReader(s3File);
+BlockManagerParameters blockManagerParamsBuilder =
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18959) Use builder for prefetch CachingBlockManager
[
https://issues.apache.org/jira/browse/HADOOP-18959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788176#comment-17788176
]
ASF GitHub Bot commented on HADOOP-18959:
-
virajjasani commented on code in PR #6240:
URL: https://github.com/apache/hadoop/pull/6240#discussion_r1399837597
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/BlockManagerParameters.java:
##
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.fs.impl.prefetch;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.LocalDirAllocator;
+import org.apache.hadoop.fs.statistics.DurationTrackerFactory;
+
+/**
+ * This class is used to provide parameters to {@link BlockManager}.
+ */
+@InterfaceAudience.Private
+public final class BlockManagerParameters {
+
+ /**
+ * Asynchronous tasks are performed in this pool.
+ */
+ private ExecutorServiceFuturePool futurePool;
+
+ /**
+ * Information about each block of the underlying file.
+ */
+ private BlockData blockData;
+
+ /**
+ * Size of the in-memory cache in terms of number of blocks.
+ */
+ private int bufferPoolSize;
+
+ /**
+ * Statistics for the stream.
+ */
+ private PrefetchingStatistics prefetchingStatistics;
+
+ /**
+ * The configuration object.
+ */
+ private Configuration conf;
+
+ /**
+ * The local dir allocator instance.
+ */
+ private LocalDirAllocator localDirAllocator;
+
+ /**
+ * Max blocks count to be kept in cache at any time.
+ */
+ private int maxBlocksCount;
+
+ /**
+ * Tracker with statistics to update.
+ */
+ private DurationTrackerFactory trackerFactory;
+
+ public ExecutorServiceFuturePool getFuturePool() {
+return futurePool;
+ }
+
+ public BlockData getBlockData() {
+return blockData;
+ }
+
+ public int getBufferPoolSize() {
+return bufferPoolSize;
+ }
+
+ public PrefetchingStatistics getPrefetchingStatistics() {
+return prefetchingStatistics;
+ }
+
+ public Configuration getConf() {
+return conf;
+ }
+
+ public LocalDirAllocator getLocalDirAllocator() {
+return localDirAllocator;
+ }
+
+ public int getMaxBlocksCount() {
+return maxBlocksCount;
+ }
+
+ public DurationTrackerFactory getTrackerFactory() {
+return trackerFactory;
+ }
+
+ public BlockManagerParameters setFuturePool(
Review Comment:
done
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/prefetch/TestS3ACachingBlockManager.java:
##
@@ -61,50 +65,93 @@ public class TestS3ACachingBlockManager extends
AbstractHadoopTestBase {
private final BlockData blockData = new BlockData(FILE_SIZE, BLOCK_SIZE);
+ private static final Configuration CONF =
+ S3ATestUtils.prepareTestConfiguration(new Configuration());
+
@Test
public void testArgChecks() throws Exception {
MockS3ARemoteObject s3File = new MockS3ARemoteObject(FILE_SIZE, false);
S3ARemoteObjectReader reader = new S3ARemoteObjectReader(s3File);
Configuration conf = new Configuration();
+BlockManagerParameters blockManagerParamsBuilder1 =
Review Comment:
done
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/prefetch/TestS3ACachingBlockManager.java:
##
@@ -244,9 +292,18 @@ private void testPrefetchHelper(boolean
forcePrefetchFailure)
throws IOException, InterruptedException {
MockS3ARemoteObject s3File = new MockS3ARemoteObject(FILE_SIZE, false);
S3ARemoteObjectReader reader = new S3ARemoteObjectReader(s3File);
+BlockManagerParameters blockManagerParamsBuilder =
Review Comment:
done
> Use builder for prefetch CachingBlockManager
>
>
> Key: HADOOP-18959
> URL: https://issues.apache.org/jira/browse/HADOOP-18959
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-availab
Re: [PR] YARN-11485. [Federation] Router Supports Yarn Admin CLI Cmds. [hadoop]
hadoop-yetus commented on PR #6265:
URL: https://github.com/apache/hadoop/pull/6265#issuecomment-1820022322
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 38s | | trunk passed |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 56s | | trunk passed |
| +1 :green_heart: | shadedclient | 31m 58s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 24s | | the patch passed |
| +1 :green_heart: | compile | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 23s | | the patch passed |
| +1 :green_heart: | compile | 0m 22s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 19s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6265/6/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt)
| hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The patch generated 6
new + 70 unchanged - 1 fixed = 76 total (was 71) |
| +1 :green_heart: | mvnsite | 0m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 0m 55s | | the patch passed |
| +1 :green_heart: | shadedclient | 32m 2s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 28m 9s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 148m 8s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6265/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6265 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 4c70666c0ccc 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e1105241d6508b11a60d4dab8e4acc5a976c70fe |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6265/6/testReport/ |
| Max. process+thread count | 578 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6265/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org
Re: [PR] HADOOP-18959 S3A : Use builder for prefetch CachingBlockManager [hadoop]
hadoop-yetus commented on PR #6240:
URL: https://github.com/apache/hadoop/pull/6240#issuecomment-1820053567
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 12s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 31s | | trunk passed |
| +1 :green_heart: | compile | 8m 19s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 7m 24s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 2m 5s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 47s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 53s | | the patch passed |
| +1 :green_heart: | compile | 7m 52s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 7m 52s | | the patch passed |
| +1 :green_heart: | compile | 7m 30s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 7m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 2s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 1s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 1s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 29s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 15m 48s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 11s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 143m 19s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6240/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6240 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d2a94196bca3 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1df9eb70b23aa631bbbe7a598206a847a83a1b82 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6240/6/testReport/ |
| Max. process+thread count | 3153 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws
U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6240/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To resp
[jira] [Commented] (HADOOP-18959) Use builder for prefetch CachingBlockManager
[
https://issues.apache.org/jira/browse/HADOOP-18959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788201#comment-17788201
]
ASF GitHub Bot commented on HADOOP-18959:
-
hadoop-yetus commented on PR #6240:
URL: https://github.com/apache/hadoop/pull/6240#issuecomment-1820053567
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 12s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 31s | | trunk passed |
| +1 :green_heart: | compile | 8m 19s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 7m 24s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 2m 5s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 47s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 53s | | the patch passed |
| +1 :green_heart: | compile | 7m 52s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 7m 52s | | the patch passed |
| +1 :green_heart: | compile | 7m 30s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 7m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 2s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 1s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 1s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 29s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 15m 48s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 11s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 143m 19s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6240/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6240 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d2a94196bca3 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1df9eb70b23aa631bbbe7a598206a847a83a1b82 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6240/6/testReport/ |
| Max. process+thread count | 3153 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws
U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6240/6/co
Re: [PR] YARN-11485. [Federation] Router Supports Yarn Admin CLI Cmds. [hadoop]
goiri commented on code in PR #6265:
URL: https://github.com/apache/hadoop/pull/6265#discussion_r1399917820
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java:
##
@@ -1099,4 +1099,27 @@ public void testNoUnsupportedHACommandsInHelp() throws
Exception {
assertFalse(errOut.contains("-failover"));
dataErr.reset();
}
+
+ @Test
+ public void testParseSubClusterId() throws Exception {
+rmAdminCLI.getConf().setBoolean(YarnConfiguration.FEDERATION_ENABLED,
true);
+
+// replaceLabelsOnNode
+String[] replaceLabelsOnNodeArgs = { "-replaceLabelsOnNode" ,
Review Comment:
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java:1108:
String[] replaceLabelsOnNodeArgs = { "-replaceLabelsOnNode" ,:40: '{' is
followed by whitespace. [NoWhitespaceAfter]
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java:1108:
String[] replaceLabelsOnNodeArgs = { "-replaceLabelsOnNode" ,:65: ',' is
preceded with whitespace. [NoWhitespaceBefore]
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java:1112:
String[] refreshQueuesArgs = { "-refreshQueues" , "-subClusterId", "SC-1"
};:34: '{' is followed by whitespace. [NoWhitespaceAfter]
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java:1112:
String[] refreshQueuesArgs = { "-refreshQueues" , "-subClusterId", "SC-1"
};:53: ',' is preceded with whitespace. [NoWhitespaceBefore]
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java:1115:
String[] refreshNodesResourcesArgs = { "-refreshNodesResources" ,
"-subClusterId", "SC-1" };:42: '{' is followed by whitespace.
[NoWhitespaceAfter]
./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java:1115:
String[] refreshNodesResourcesArgs = { "-refreshNodesResources" ,
"-subClusterId", "SC-1" };:69: ',' is preceded with whitespace.
[NoWhitespaceBefore]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] HADOOP-18872: [Backport to 3.3] [ABFS] [BugFix] Misreporting Retry Count for Sub-sequential and Parallel Operations [hadoop]
anujmodi2021 opened a new pull request, #6284: URL: https://github.com/apache/hadoop/pull/6284 Jira Ticket: https://issues.apache.org/jira/browse/HADOOP-18872 Trunk PR: https://github.com/apache/hadoop/pull/6019 Cherry-picked commit: https://github.com/apache/hadoop/commit/000a39ba2d2131ac158e23b35eae8c1329681bff Description: There was a bug identified where retry count in the client correlation id was wrongly reported for sub-sequential and parallel operations triggered by a single file system call. This was due to reusing same tracing context for all such calls. We create a new tracing context as soon as HDFS call comes. We keep on passing that same TC for all the client calls. For instance, when we get a createFile call, we first call metadata operations. If those metadata operations somehow succeeded after a few retries, the tracing context will have that many retry count in it. Now when actual call for create is made, same retry count will be used to construct the headers(clientCorrelationId). Alhough the create operation never failed, we will still see retry count from the previous request. Fix is to use a new tracing context object for all the network calls made. All the sub-sequential and parallel operations will have same primary request Id to correlate them, yet they will have their own tracing of retry count. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18872) ABFS: Misreporting Retry Count for Sub-sequential and Parallel Operations
[ https://issues.apache.org/jira/browse/HADOOP-18872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788243#comment-17788243 ] ASF GitHub Bot commented on HADOOP-18872: - anujmodi2021 opened a new pull request, #6284: URL: https://github.com/apache/hadoop/pull/6284 Jira Ticket: https://issues.apache.org/jira/browse/HADOOP-18872 Trunk PR: https://github.com/apache/hadoop/pull/6019 Cherry-picked commit: https://github.com/apache/hadoop/commit/000a39ba2d2131ac158e23b35eae8c1329681bff Description: There was a bug identified where retry count in the client correlation id was wrongly reported for sub-sequential and parallel operations triggered by a single file system call. This was due to reusing same tracing context for all such calls. We create a new tracing context as soon as HDFS call comes. We keep on passing that same TC for all the client calls. For instance, when we get a createFile call, we first call metadata operations. If those metadata operations somehow succeeded after a few retries, the tracing context will have that many retry count in it. Now when actual call for create is made, same retry count will be used to construct the headers(clientCorrelationId). Alhough the create operation never failed, we will still see retry count from the previous request. Fix is to use a new tracing context object for all the network calls made. All the sub-sequential and parallel operations will have same primary request Id to correlate them, yet they will have their own tracing of retry count. > ABFS: Misreporting Retry Count for Sub-sequential and Parallel Operations > - > > Key: HADOOP-18872 > URL: https://issues.apache.org/jira/browse/HADOOP-18872 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Affects Versions: 3.3.6 >Reporter: Anmol Asrani >Assignee: Anuj Modi >Priority: Major > Labels: Bug, pull-request-available > Fix For: 3.4.0 > > > There was a bug identified where retry count in the client correlation id was > wrongly reported for sub-sequential and parallel operations triggered by a > single file system call. This was due to reusing same tracing context for all > such calls. > We create a new tracing context as soon as HDFS call comes. We keep on > passing that same TC for all the client calls. > For instance, when we get a createFile call, we first call metadata > operations. If those metadata operations somehow succeeded after a few > retries, the tracing context will have that many retry count in it. Now when > actual call for create is made, same retry count will be used to construct > the headers(clientCorrelationId). Alhough the create operation never failed, > we will still see retry count from the previous request. > Fix is to use a new tracing context object for all the network calls made. > All the sub-sequential and parallel operations will have same primary request > Id to correlate them, yet they will have their own tracing of retry count. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18874) ABFS: Adding Server returned request id in Exception method thrown to caller.
[ https://issues.apache.org/jira/browse/HADOOP-18874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788246#comment-17788246 ] ASF GitHub Bot commented on HADOOP-18874: - anujmodi2021 opened a new pull request, #6285: URL: https://github.com/apache/hadoop/pull/6285 Jira Ticket: https://issues.apache.org/jira/browse/HADOOP-18874 Trunk PR: https://github.com/apache/hadoop/pull/6004 Cherry-picked commit: https://github.com/apache/hadoop/commit/597ceaae3a779e8b31e6c5e1453a56093a455271 > ABFS: Adding Server returned request id in Exception method thrown to caller. > - > > Key: HADOOP-18874 > URL: https://issues.apache.org/jira/browse/HADOOP-18874 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Anuj Modi >Assignee: Anuj Modi >Priority: Major > Labels: pull-request-available > > Each request made to Azure server has its unique ActivityId (rid) which is > returned in response of the request whether is succeed or fails. > When a HDFS call fails due to an error from Azure service, An > ABFSRestOperationException is throws to the caller. This task is to add a > server returned activity id (rid) in the exception message which can be used > to investigate the failure on service side. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] HADOOP-18874: [Backport to 3.3] [ABFS] Adding Server returned request id in Exception Message thrown to caller. [hadoop]
anujmodi2021 opened a new pull request, #6285: URL: https://github.com/apache/hadoop/pull/6285 Jira Ticket: https://issues.apache.org/jira/browse/HADOOP-18874 Trunk PR: https://github.com/apache/hadoop/pull/6004 Cherry-picked commit: https://github.com/apache/hadoop/commit/597ceaae3a779e8b31e6c5e1453a56093a455271 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HDFS-17223. Add journalnode maintenance node list [hadoop]
xinglin commented on code in PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#discussion_r1400039039
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQuorumJournalManager.java:
##
@@ -1171,6 +1179,47 @@ public void testSelectViaRpcAfterJNRestart() throws
Exception {
}
}
+ /**
+ * Tests to throw an exception if the jn maintenance nodes exceeds half of
the journalnode number.
Review Comment:
nit:
"if the jn maintenance nodes exceeds half of the journalnode number." ->
"when more than half of the journal nodes are in maintenance mode".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
Re: [PR] HADOOP-18874: [Backport to 3.3] [ABFS] Adding Server returned request id in Exception Message thrown to caller. [hadoop]
hadoop-yetus commented on PR #6285:
URL: https://github.com/apache/hadoop/pull/6285#issuecomment-1820323305
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 6m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 18s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 36s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 29s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 41s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 34s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 1m 7s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 34m 27s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 18s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6285/1/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 1 new + 0 unchanged - 0
fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 7s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 133m 31s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6285/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6285 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b5e749324cb0 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / 219c61b16eeb914c92b6988b15f91571d61eb0f7 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6285/1/testReport/ |
| Max. process+thread count | 700 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6285/1/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18874) ABFS: Adding Server returned request id in Exception method thrown to caller.
[
https://issues.apache.org/jira/browse/HADOOP-18874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788271#comment-17788271
]
ASF GitHub Bot commented on HADOOP-18874:
-
hadoop-yetus commented on PR #6285:
URL: https://github.com/apache/hadoop/pull/6285#issuecomment-1820323305
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 6m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 18s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 36s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 29s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 41s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 34s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 1m 7s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 34m 27s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 18s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6285/1/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 1 new + 0 unchanged - 0
fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 7s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 133m 31s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6285/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6285 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b5e749324cb0 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / 219c61b16eeb914c92b6988b15f91571d61eb0f7 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6285/1/testReport/ |
| Max. process+thread count | 700 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6285/1/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> ABFS: Adding Server returned request id in Exception method thrown to caller.
> -
>
> Key: HADOOP-18874
> URL: https://issues.apache.org/jira/browse/HADOOP-18874
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Anuj Modi
>Assignee: Anuj Modi
>Priority: Major
> Labels: pull-request-available
>
> Each request made to Azure server has its unique ActivityId (rid) which is
> returned in response of the request whether is succeed or fails.
> When a HDFS call fails due to an error from Azure service, An
> ABFSRestOperati
[jira] [Commented] (HADOOP-18872) ABFS: Misreporting Retry Count for Sub-sequential and Parallel Operations
[
https://issues.apache.org/jira/browse/HADOOP-18872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788273#comment-17788273
]
ASF GitHub Bot commented on HADOOP-18872:
-
hadoop-yetus commented on PR #6284:
URL: https://github.com/apache/hadoop/pull/6284#issuecomment-1820325936
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 16s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 7 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 31s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 36s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 31s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 43s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 35s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 1m 9s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 34m 47s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 19s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6284/1/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 1 new + 5 unchanged - 0
fixed = 6 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 39s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 4s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 139m 42s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6284/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6284 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 83b27307a3a4 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / e4e7d1ba877f931a6c60972fbb11b486af02e2c7 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6284/1/testReport/ |
| Max. process+thread count | 671 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6284/1/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> ABFS: Misreporting Retry Count for Sub-sequential and Parallel Operations
> -
>
> Key: HADOOP-18872
> URL: https://issues.apache.org/jira/browse/HADOOP-18872
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
>Affects Versions: 3.3.6
>Reporter: Anmol Asrani
>Assignee: Anuj Modi
>Priority: Major
> Labels: Bug, pull-request-available
> Fix For: 3.4.0
>
>
> There was a bug identified where retry count in the client correlation id was
> wrongly reported for sub-sequential and parallel operations triggered by a
> single fi
Re: [PR] HADOOP-18872: [Backport to 3.3] [ABFS] [BugFix] Misreporting Retry Count for Sub-sequential and Parallel Operations [hadoop]
hadoop-yetus commented on PR #6284:
URL: https://github.com/apache/hadoop/pull/6284#issuecomment-1820325936
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 16s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 7 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 31s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 36s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 31s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 43s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 35s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 1m 9s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 34m 47s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 19s |
[/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6284/1/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt)
| hadoop-tools/hadoop-azure: The patch generated 1 new + 5 unchanged - 0
fixed = 6 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 39s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 4s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 139m 42s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6284/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6284 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 83b27307a3a4 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / e4e7d1ba877f931a6c60972fbb11b486af02e2c7 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6284/1/testReport/ |
| Max. process+thread count | 671 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6284/1/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[PR] HDFS-17233. The conf dfs.datanode.lifeline.interval.seconds is not co… [hadoop]
eshwitha opened a new pull request, #6286: URL: https://github.com/apache/hadoop/pull/6286 …nsidering time unit seconds ### Description of PR The configuration dfs.datanode.lifeline.interval.seconds is not considering the time unit as seconds. Upon configuring the DFS_DATANODE_LIFELINE_INTERVAL_SECONDS_KEY, the value is not converting to Ms as it should be. ### How was this patch tested? Verified from the datanode logs that the config dfs.datanode.lifeline.interval.seconds is considering the time unit as seconds after the fix. ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17372) S3A AWS Credential provider loading gets confused with isolated classloaders
[ https://issues.apache.org/jira/browse/HADOOP-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788308#comment-17788308 ] Antonio Murgia commented on HADOOP-17372: - Given the issue reported by [~brandonvin] , would you consider a PR that is guarding the "conf.setClassLoader(this.getClass().getClassLoader())" call? Something like: "fs.s3a.fixclassloader" (any advice here is more than welcome) that is by default true and if set to false will skip the call? This would be a huge quality of life improvement at least for mine and Brandon team :) > S3A AWS Credential provider loading gets confused with isolated classloaders > > > Key: HADOOP-17372 > URL: https://issues.apache.org/jira/browse/HADOOP-17372 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.3.1 > > > Problem: exception in loading S3A credentials for an FS, "Class class > com.amazonaws.auth.EnvironmentVariableCredentialsProvider does not implement > AWSCredentialsProvider" > Location: S3A + Spark dataframes test > Hypothesised cause: > Configuration.getClasses() uses the context classloader, and with the spark > isolated CL that's different from the one the s3a FS uses, so it can't load > AWS credential providers. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
