[jira] [Commented] (YARN-11471) FederationStateStoreFacade Cache Support Caffeine
[
https://issues.apache.org/jira/browse/YARN-11471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844654#comment-17844654
]
ASF GitHub Bot commented on YARN-11471:
---
hadoop-yetus commented on PR #6795:
URL: https://github.com/apache/hadoop/pull/6795#issuecomment-2100562189
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +0 :ok: | shellcheck | 0m 01s | | Shellcheck was not available. |
| +0 :ok: | shelldocs | 0m 01s | | Shelldocs was not available. |
| +0 :ok: | xmllint | 0m 01s | | xmllint was not available. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | markdownlint | 0m 01s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 13s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 86m 03s | | trunk passed |
| +1 :green_heart: | compile | 37m 48s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 42s | | trunk passed |
| -1 :x: | mvnsite | 22m 54s |
[/branch-mvnsite-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6795/1/artifact/out/branch-mvnsite-root.txt)
| root in trunk failed. |
| +1 :green_heart: | javadoc | 15m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 184m 46s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 20s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 81m 39s |
[/patch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6795/1/artifact/out/patch-mvninstall-root.txt)
| root in the patch failed. |
| +1 :green_heart: | compile | 38m 56s | | the patch passed |
| +1 :green_heart: | javac | 38m 56s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 7m 00s | | the patch passed |
| -1 :x: | mvnsite | 22m 18s |
[/patch-mvnsite-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6795/1/artifact/out/patch-mvnsite-root.txt)
| root in the patch failed. |
| +1 :green_heart: | javadoc | 14m 54s | | the patch passed |
| -1 :x: | shadedclient | 193m 21s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 10s | | The patch does not
generate ASF License warnings. |
| | | 646m 42s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6795 |
| Optional Tests | dupname asflicense codespell detsecrets shellcheck
shelldocs compile javac javadoc mvninstall mvnsite unit shadedclient xmllint
spotbugs checkstyle markdownlint |
| uname | MINGW64_NT-10.0-17763 efe4716c0f2c 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 46c6ca567e4ed5f6fbad7f2c038711bee74c95ae |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6795/1/testReport/
|
| modules | C: hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6795/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> FederationStateStoreFacade Cache Support Caffeine
> -
>
> Key: YARN-11471
> URL: https://issues.apache.org/jira/browse/YARN-11471
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Affects Versions: 3.4.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
> Lab
[jira] [Commented] (YARN-11693) Refactor Container scheduler
[
https://issues.apache.org/jira/browse/YARN-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844626#comment-17844626
]
ASF GitHub Bot commented on YARN-11693:
---
hadoop-yetus commented on PR #6786:
URL: https://github.com/apache/hadoop/pull/6786#issuecomment-2100318644
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 91m 43s | | trunk passed |
| +1 :green_heart: | compile | 5m 25s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 42s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 150m 59s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 2m 19s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6786/1/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch failed. |
| -1 :x: | compile | 2m 13s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6786/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch failed. |
| -1 :x: | javac | 2m 14s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6786/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch failed. |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 06s | | the patch passed |
| -1 :x: | mvnsite | 2m 14s |
[/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6786/1/artifact/out/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch failed. |
| +1 :green_heart: | javadoc | 2m 32s | | the patch passed |
| -1 :x: | shadedclient | 71m 35s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 4m 26s | | The patch does not
generate ASF License warnings. |
| | | 336m 44s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6786 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 0b8f004d255a 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 3c58167b67cc3776e63d85ddc9987b0de9bd22d2 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6786/1/testReport/
|
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6786/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was auto
[jira] [Commented] (YARN-11694) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[
https://issues.apache.org/jira/browse/YARN-11694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844533#comment-17844533
]
ASF GitHub Bot commented on YARN-11694:
---
hadoop-yetus commented on PR #6793:
URL: https://github.com/apache/hadoop/pull/6793#issuecomment-2099760156
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 50s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 90m 53s | | trunk passed |
| +1 :green_heart: | compile | 41m 07s | | trunk passed |
| +1 :green_heart: | checkstyle | 6m 01s | | trunk passed |
| +1 :green_heart: | mvnsite | 14m 06s | | trunk passed |
| +1 :green_heart: | javadoc | 13m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 170m 11s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 37s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 8m 42s | | the patch passed |
| +1 :green_heart: | compile | 38m 25s | | the patch passed |
| +1 :green_heart: | javac | 38m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 6m 02s | | the patch passed |
| +1 :green_heart: | mvnsite | 14m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 13m 47s | | the patch passed |
| +1 :green_heart: | shadedclient | 180m 14s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 39s | | The patch does not
generate ASF License warnings. |
| | | 547m 46s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6793 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 9beec3ff0f2e 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 48194ae9f729f70d7fa780a21dad2dceadde77c7 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6793/1/testReport/
|
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice
hadoop-tools/hadoop-sls U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6793/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> 2 tests are non-idempotent (passes in the first run but fails in repeated
> runs in the same JVM)
> ---
>
> Key: YARN-11694
> URL: https://issues.apache.org/jira/browse/YARN-11694
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Kaiyao Ke
>Priority: Major
> Labels: pull-request-available
>
> ## TestTimelineReaderMetrics#testTimelineReaderMetrics
> `org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderMetrics#testTimelineReaderMetrics`
> does not perform a source unregistration after test execution, so the
> `TimelineReaderMetrics.getInstance()` call in repeated runs will throw an
> error since the metrics source `TimelineReaderMetrics` already exists.
> Error message in the 2nd run:
> ```
> org.apache.hadoop.metrics2.MetricsException: Metrics source
> TimelineReaderMetrics already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sour
[jira] [Commented] (YARN-11622) ResourceManager asynchronous switch from Standy to Active exception
[
https://issues.apache.org/jira/browse/YARN-11622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844494#comment-17844494
]
ASF GitHub Bot commented on YARN-11622:
---
hiwangzhihui commented on PR #6352:
URL: https://github.com/apache/hadoop/pull/6352#issuecomment-2099626750
hi @slfan1989 @dineshchitlangia All details about this PR have been
processed, If you have time to review it again. Thank you a lot.
> ResourceManager asynchronous switch from Standy to Active exception
> ---
>
> Key: YARN-11622
> URL: https://issues.apache.org/jira/browse/YARN-11622
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha4, 3.1.1, 3.3.0
>Reporter: wangzhihui
>Assignee: wangzhihui
>Priority: Major
> Labels: pull-request-available
> Attachments: rm_ha_solution.png, yuque_diagram (1).jpg,
> yuque_diagram.jpg
>
>
> h1. Two exception cases：
> h2. The first case：
> *The exception desc:*
> {code:java}
> 14:52:57,426 FATAL event.AsyncDispatcher (AsyncDispatcher.java:dispatch(203))
> - Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.access$1200(ResourceManager.java:610)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.handleTransitionToStandByInNewThread(ResourceManager.java:941)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.access$1100(ResourceManager.java:144)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMFatalEventDispatcher.handle(ResourceManager.java:902)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMFatalEventDispatcher.handle(ResourceManager.java:892)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126)
> at java.lang.Thread.run(Thread.java:748){{}} * {code}
>
> * ActiveStandbyElector and ZKRMStateStore triggered toStandy event at
> 14:52:57,
> Two asynchronous events are respectively referred to as Thread_ 1、Thread_ 2.
> * As shown in the following figure， Thread_1 during the toStandby process ，
> reinitializes the activeServices to null. At this point, Thread_2 will use
> the "activeServices" when executing the handleTransitionToStandByInNewThread
> method ultimately resulting in a NullPointerException and the Reosurcemanager
> server exit.
> !yuque_diagram.jpg|width=629,height=100!
> h2. The second case：
> *The exception desc:*
> {code:java}
> 06:17:35,913 WARN ha.ActiveStandbyElector
> (ActiveStandbyElector.java:becomeActive(900)) - Exception handling the
> winning of election
> org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:146)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:896)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:543)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:558)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> Caused by: org.apache.hadoop.ha.ServiceFailedException: Error on refreshAll
> during transition to Active
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:315)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:144)
> ... 4 more
> Caused by: org.apache.hadoop.ha.ServiceFailedException: RefreshAll operation
> failed
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:765)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:307)
> ... 5 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:467)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:423)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:754)
> ... 6 more
> 06:17:35,917 ERROR resourcemanager.ResourceManager
> (ResourceManager.java:handle(898)) - Received RMFatalEvent of type
> TRANSITION_TO_ACTIVE_FAILED, caused by failure to refresh configuration
> settings: org.apache.hadoop.ha.ServiceFailedException: RefreshAll opera
> tion failed{{}} {code}
> * ActiveStandbyElector and ZKRMStateStore tri
[jira] [Commented] (YARN-11622) ResourceManager asynchronous switch from Standy to Active exception
[
https://issues.apache.org/jira/browse/YARN-11622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844405#comment-17844405
]
ASF GitHub Bot commented on YARN-11622:
---
hadoop-yetus commented on PR #6352:
URL: https://github.com/apache/hadoop/pull/6352#issuecomment-2098960920
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 3m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 38s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 36s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 28s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 39s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 34s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 1m 17s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 21m 36s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 21s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 66 unchanged - 2 fixed = 66 total (was 68) |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 15s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 37s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 78m 53s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not
generate ASF License warnings. |
| | | 168m 38s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6352/16/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6352 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux 51964cb3034d 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / fcdd4fddd6ed005e750afd4c399fb98ab976b7a1 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6352/16/testReport/ |
| Max. process+thread count | 942 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6352/16/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> ResourceManager asynchronous switch from Standy to Active exception
> ---
>
> Key: YARN-11622
> URL: https://issues.apache.org/jira/browse/YARN-11622
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha4, 3.1.1, 3.3.0
>Reporter: wangzhihui
>Assignee: wangzhihui
>Priority: Major
> Labels: pull-request-available
> Attachments: rm_ha_solution.png, yuque_diagram (1).jpg,
> yuque_diagram.jpg
>
>
[jira] [Commented] (YARN-11687) Update CGroupsResourceCalculator to track usages using cgroupv2
[
https://issues.apache.org/jira/browse/YARN-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844317#comment-17844317
]
ASF GitHub Bot commented on YARN-11687:
---
hadoop-yetus commented on PR #6780:
URL: https://github.com/apache/hadoop/pull/6780#issuecomment-2098549896
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 4s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 38m 55s | | trunk passed |
| +1 :green_heart: | compile | 8m 23s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 7m 36s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 2m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 54s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 1s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 52s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 29s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 41s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 9s | | the patch passed |
| +1 :green_heart: | compile | 7m 37s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 7m 37s | | the patch passed |
| +1 :green_heart: | compile | 7m 31s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 7m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 56s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/9/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 12 new + 3 unchanged -
0 fixed = 15 total (was 3) |
| +1 :green_heart: | mvnsite | 1m 43s | | the patch passed |
| -1 :x: | javadoc | 0m 49s |
[/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/9/artifact/out/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 generated 1 new + 98
unchanged - 1 fixed = 99 total (was 99) |
| +1 :green_heart: | javadoc | 0m 56s | | hadoop-yarn-common in the
patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06. |
| +1 :green_heart: | javadoc | 0m 47s | |
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 generated 0 new +
195 unchanged - 1 fixed = 195 total (was 196) |
| -1 :x: | spotbugs | 1m 45s |
[/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/9/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 39m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +
[jira] [Commented] (YARN-11687) Update CGroupsResourceCalculator to track usages using cgroupv2
[
https://issues.apache.org/jira/browse/YARN-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844287#comment-17844287
]
ASF GitHub Bot commented on YARN-11687:
---
hadoop-yetus commented on PR #6780:
URL: https://github.com/apache/hadoop/pull/6780#issuecomment-2098376897
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 7m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 50s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 23s | | trunk passed |
| +1 :green_heart: | compile | 4m 5s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 3m 44s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 3s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 1s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 2m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 18s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 47s | | the patch passed |
| +1 :green_heart: | compile | 3m 52s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 3m 52s | | the patch passed |
| +1 :green_heart: | compile | 3m 41s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 3m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 57s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/10/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 12 new + 3 unchanged -
0 fixed = 15 total (was 3) |
| +1 :green_heart: | mvnsite | 0m 54s | | the patch passed |
| -1 :x: | javadoc | 0m 24s |
[/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/10/artifact/out/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 generated 1 new + 98
unchanged - 1 fixed = 99 total (was 99) |
| +1 :green_heart: | javadoc | 0m 32s | | hadoop-yarn-common in the
patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06. |
| +1 :green_heart: | javadoc | 0m 27s | |
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 generated 0 new +
195 unchanged - 1 fixed = 195 total (was 196) |
| -1 :x: | spotbugs | 1m 8s |
[/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/10/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 22m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
[jira] [Commented] (YARN-11687) Update CGroupsResourceCalculator to track usages using cgroupv2
[
https://issues.apache.org/jira/browse/YARN-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844286#comment-17844286
]
ASF GitHub Bot commented on YARN-11687:
---
hadoop-yetus commented on PR #6780:
URL: https://github.com/apache/hadoop/pull/6780#issuecomment-2098375798
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 7m 10s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 50s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 7s | | trunk passed |
| +1 :green_heart: | compile | 3m 56s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 3m 50s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 3s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 2s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 2m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 14s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 47s | | the patch passed |
| +1 :green_heart: | compile | 3m 50s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 3m 50s | | the patch passed |
| +1 :green_heart: | compile | 3m 42s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 3m 42s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 58s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/11/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 12 new + 3 unchanged -
0 fixed = 15 total (was 3) |
| +1 :green_heart: | mvnsite | 0m 55s | | the patch passed |
| -1 :x: | javadoc | 0m 24s |
[/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/11/artifact/out/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 generated 1 new + 98
unchanged - 1 fixed = 99 total (was 99) |
| +1 :green_heart: | javadoc | 0m 28s | | hadoop-yarn-common in the
patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06. |
| +1 :green_heart: | javadoc | 0m 27s | |
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 generated 0 new +
195 unchanged - 1 fixed = 195 total (was 196) |
| -1 :x: | spotbugs | 1m 3s |
[/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/11/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 22m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
[jira] [Commented] (YARN-11687) Update CGroupsResourceCalculator to track usages using cgroupv2
[
https://issues.apache.org/jira/browse/YARN-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844274#comment-17844274
]
ASF GitHub Bot commented on YARN-11687:
---
hadoop-yetus commented on PR #6780:
URL: https://github.com/apache/hadoop/pull/6780#issuecomment-2098324766
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 50s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 44s | | trunk passed |
| +1 :green_heart: | compile | 3m 37s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 3m 22s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 57s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 3s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 6s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 2m 0s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 11s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 3m 18s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 3m 18s | | the patch passed |
| +1 :green_heart: | compile | 3m 20s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 3m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 56s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/8/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 17 new + 3 unchanged -
0 fixed = 20 total (was 3) |
| +1 :green_heart: | mvnsite | 1m 1s | | the patch passed |
| -1 :x: | javadoc | 0m 25s |
[/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/8/artifact/out/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 generated 1 new + 98
unchanged - 1 fixed = 99 total (was 99) |
| +1 :green_heart: | javadoc | 0m 36s | | hadoop-yarn-common in the
patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06. |
| +1 :green_heart: | javadoc | 0m 25s | |
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 generated 0 new +
195 unchanged - 1 fixed = 195 total (was 196) |
| -1 :x: | spotbugs | 1m 5s |
[/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/8/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 20m 14s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +
[jira] [Commented] (YARN-11687) Update CGroupsResourceCalculator to track usages using cgroupv2
[
https://issues.apache.org/jira/browse/YARN-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844240#comment-17844240
]
ASF GitHub Bot commented on YARN-11687:
---
K0K0V0K commented on code in PR #6780:
URL: https://github.com/apache/hadoop/pull/6780#discussion_r1592238394
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsV2ResourceCalculator.java:
##
@@ -0,0 +1,135 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.util.CpuTimeTracker;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+
+/**
+ * A CGroupV2 file-system based Resource calculator without the process tree
features.
+ *
+ * The feature only works if cluster runs in pure V2 version, because when we
read the
+ * /proc/{pid}/cgroup file currently we can not handle multiple lines.
+ */
+public class CGroupsV2ResourceCalculator extends
AbstractCGroupsResourceCalculator {
+ private static final Logger LOG =
LoggerFactory.getLogger(CGroupsV2ResourceCalculator.class);
+ private final Map stats = new ConcurrentHashMap<>();
+
+ @VisibleForTesting
+ String root = "/";
+
+ /**
+ * Create resource calculator for the container that has the specified pid.
+ * @param pid A pid from the cgroup or null for all containers
+ */
+ public CGroupsV2ResourceCalculator(String pid) {
+super(pid);
+ }
+
+ @Override
+ public void initialize() throws YarnException {
+cpuTimeTracker = new CpuTimeTracker(jiffyLengthMs);
Review Comment:
Yes, also i removed that from v1
- i think check only linux can use Cgroup is overkill, and if someone,
somehow, manage to hack a cgroup to other OS it will block.
- Also now i could remove the limitation following limitation
```
* CGroups has its limitations. It can only be enabled, if both CPU and memory
* cgroups are enabled with yarn.nodemanager.resource.cpu.enabled and
* yarn.nodemanager.resource.memory.enabled respectively. This means that
* memory limits are enforced by default. You can turn this off and keep
* memory reporting only with yarn.nodemanager.resource.memory.enforced.
```
> Update CGroupsResourceCalculator to track usages using cgroupv2
> ---
>
> Key: YARN-11687
> URL: https://issues.apache.org/jira/browse/YARN-11687
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Bence Kosztolnik
>Priority: Major
> Labels: pull-request-available
>
> [CGroupsResourceCalculator|https://github.com/apache/hadoop/blob/f609460bda0c2bd87dd3580158e549e2f34f14d5/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsResourceCalculator.java]
> should also be updated to handle the cgroup v2 changes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11694) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/YARN-11694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844201#comment-17844201 ] ASF GitHub Bot commented on YARN-11694: --- kaiyaok2 commented on PR #6793: URL: https://github.com/apache/hadoop/pull/6793#issuecomment-2097723400 > +1 Can we fix the checkstyle issue? @slfan1989 I fixed the 2 "line too long" issues. The remaining 5 are in fact existing issues (as I changed `public static CountDownLatch latch = new CountDownLatch(1);` to `public static CountDownLatch latch;`). I don't think it's necessary to turn `latch` private and add getter methods, since these classes are only used in testing. > 2 tests are non-idempotent (passes in the first run but fails in repeated > runs in the same JVM) > --- > > Key: YARN-11694 > URL: https://issues.apache.org/jira/browse/YARN-11694 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Kaiyao Ke >Priority: Major > Labels: pull-request-available > > ## TestTimelineReaderMetrics#testTimelineReaderMetrics > `org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderMetrics#testTimelineReaderMetrics` > does not perform a source unregistration after test execution, so the > `TimelineReaderMetrics.getInstance()` call in repeated runs will throw an > error since the metrics source `TimelineReaderMetrics` already exists. > Error message in the 2nd run: > ``` > org.apache.hadoop.metrics2.MetricsException: Metrics source > TimelineReaderMetrics already exists! > at > org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152) > at > org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125) > at > org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229) > at > org.apache.hadoop.yarn.server.timelineservice.metrics.TimelineReaderMetrics.getInstance(TimelineReaderMetrics.java:61) > at > org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderMetrics.setup(TestTimelineReaderMetrics.java:52) > at java.base/java.lang.reflect.Method.invoke(Method.java:568) > at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) > at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) > ``` > > ## TestFederationStateStoreClientMetrics#testSuccessfulCalls > `org.apache.hadoop.yarn.server.federation.store.metrics.TestFederationStateStoreClientMetrics#testSuccessfulCalls` > retrieves the historical number of successful calls, but does not retrieve > the historical average latency of those calls. For example, it asserts > `FederationStateStoreClientMetrics.getLatencySucceededCalls()` is 100 after > the `goodStateStore.registerSubCluster(100);` call. However, in the second > execution of the test, 2 historical calls from the first execution (with > latency 100 and 200 respectively) has already been recorded, so > `FederationStateStoreClientMetrics.getLatencySucceededCalls()` will be > 133. (mean of 100, 200 and 100) > > Error message in the 2nd run: > ``` > java.lang.AssertionError: expected:<100.0> but was:<133.34> > at org.junit.Assert.fail(Assert.java:89) > at org.junit.Assert.failNotEquals(Assert.java:835) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java:685) > at > org.apache.hadoop.yarn.server.federation.store.metrics.TestFederationStateStoreClientMetrics.testSuccessfulCalls(TestFederationStateStoreClientMetrics.java:63) > at java.base/java.lang.reflect.Method.invoke(Method.java:568) > ``` -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11694) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[
https://issues.apache.org/jira/browse/YARN-11694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844183#comment-17844183
]
ASF GitHub Bot commented on YARN-11694:
---
hadoop-yetus commented on PR #6793:
URL: https://github.com/apache/hadoop/pull/6793#issuecomment-2097673172
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 48s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 55s | | trunk passed |
| +1 :green_heart: | compile | 9m 36s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 9m 12s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 2m 16s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 19s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 19s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 2m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 41s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 44s | | the patch passed |
| +1 :green_heart: | compile | 9m 48s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 9m 48s | | the patch passed |
| +1 :green_heart: | compile | 8m 59s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 8m 59s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 2m 23s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6793/4/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 5 new + 10 unchanged - 5 fixed = 15 total (was
15) |
| +1 :green_heart: | mvnsite | 1m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 16s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 13s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 2m 22s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 34s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 18s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 1m 7s | |
hadoop-yarn-server-timelineservice in the patch passed. |
| +1 :green_heart: | unit | 11m 42s | | hadoop-sls in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 157m 23s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6793/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6793 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux ffac63be2731 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 48194ae9f729f70d7fa780a21dad2dceadde77c7 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
[jira] [Commented] (YARN-11469) Refactor FederationStateStoreFacade Cache Code
[
https://issues.apache.org/jira/browse/YARN-11469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844090#comment-17844090
]
ASF GitHub Bot commented on YARN-11469:
---
hadoop-yetus commented on PR #6796:
URL: https://github.com/apache/hadoop/pull/6796#issuecomment-2097291421
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 6m 17s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 52s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 39s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 31s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 45s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 40s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 1m 26s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 35m 38s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 18s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common:
The patch generated 0 new + 0 unchanged - 11 fixed = 0 total (was 11) |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 27s | |
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common
generated 0 new + 44 unchanged - 1 fixed = 44 total (was 45) |
| +1 :green_heart: | spotbugs | 1m 22s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 59s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 3m 6s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6796/3/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt)
| hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 139m 0s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.federation.policies.amrmproxy.TestLocalityMulticastAMRMProxyPolicy
|
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6796/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6796 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 30922528e317 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / 8019977624c0553896852554e207af22f5c2d646 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6796/3/testReport/ |
| Max. process+thread count | 552 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6796/3/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Refactor FederationStateStoreFacade Cache Code
> --
>
> Key: YARN-11469
> URL: https://issues.apache.org/jira/browse/YARN-11469
>
[jira] [Commented] (YARN-11694) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/YARN-11694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844065#comment-17844065 ] ASF GitHub Bot commented on YARN-11694: --- slfan1989 commented on PR #6793: URL: https://github.com/apache/hadoop/pull/6793#issuecomment-2097113107 +1 Can we fix the checkstyle issue? > 2 tests are non-idempotent (passes in the first run but fails in repeated > runs in the same JVM) > --- > > Key: YARN-11694 > URL: https://issues.apache.org/jira/browse/YARN-11694 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Kaiyao Ke >Priority: Major > Labels: pull-request-available > > ## TestTimelineReaderMetrics#testTimelineReaderMetrics > `org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderMetrics#testTimelineReaderMetrics` > does not perform a source unregistration after test execution, so the > `TimelineReaderMetrics.getInstance()` call in repeated runs will throw an > error since the metrics source `TimelineReaderMetrics` already exists. > Error message in the 2nd run: > ``` > org.apache.hadoop.metrics2.MetricsException: Metrics source > TimelineReaderMetrics already exists! > at > org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152) > at > org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125) > at > org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229) > at > org.apache.hadoop.yarn.server.timelineservice.metrics.TimelineReaderMetrics.getInstance(TimelineReaderMetrics.java:61) > at > org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderMetrics.setup(TestTimelineReaderMetrics.java:52) > at java.base/java.lang.reflect.Method.invoke(Method.java:568) > at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) > at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) > ``` > > ## TestFederationStateStoreClientMetrics#testSuccessfulCalls > `org.apache.hadoop.yarn.server.federation.store.metrics.TestFederationStateStoreClientMetrics#testSuccessfulCalls` > retrieves the historical number of successful calls, but does not retrieve > the historical average latency of those calls. For example, it asserts > `FederationStateStoreClientMetrics.getLatencySucceededCalls()` is 100 after > the `goodStateStore.registerSubCluster(100);` call. However, in the second > execution of the test, 2 historical calls from the first execution (with > latency 100 and 200 respectively) has already been recorded, so > `FederationStateStoreClientMetrics.getLatencySucceededCalls()` will be > 133. (mean of 100, 200 and 100) > > Error message in the 2nd run: > ``` > java.lang.AssertionError: expected:<100.0> but was:<133.34> > at org.junit.Assert.fail(Assert.java:89) > at org.junit.Assert.failNotEquals(Assert.java:835) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java:685) > at > org.apache.hadoop.yarn.server.federation.store.metrics.TestFederationStateStoreClientMetrics.testSuccessfulCalls(TestFederationStateStoreClientMetrics.java:63) > at java.base/java.lang.reflect.Method.invoke(Method.java:568) > ``` -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11471) FederationStateStoreFacade Cache Support Caffeine
[
https://issues.apache.org/jira/browse/YARN-11471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843618#comment-17843618
]
ASF GitHub Bot commented on YARN-11471:
---
slfan1989 commented on code in PR #6795:
URL: https://github.com/apache/hadoop/pull/6795#discussion_r1590520965
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationCaffeineCache.java:
##
@@ -0,0 +1,125 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+package org.apache.hadoop.yarn.server.federation.cache;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import
org.apache.hadoop.yarn.server.federation.store.records.SubClusterPolicyConfiguration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+public class FederationCaffeineCache extends FederationCache {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(FederationCaffeineCache.class);
+
+ private Cache cache;
+
+ private int cacheTimeToLive;
+ private long cacheEntityNums;
+
+ private String className = this.getClass().getSimpleName();
+
+ private boolean isCachingEnabled = false;
+
+ @Override
+ public boolean isCachingEnabled() {
+return isCachingEnabled;
+ }
+
+ @Override
+ public void initCache(Configuration pConf, FederationStateStore pStateStore)
{
+cacheTimeToLive =
pConf.getInt(YarnConfiguration.FEDERATION_CACHE_TIME_TO_LIVE_SECS,
+YarnConfiguration.DEFAULT_FEDERATION_CACHE_TIME_TO_LIVE_SECS);
+cacheEntityNums =
pConf.getLong(YarnConfiguration.FEDERATION_CACHE_ENTITY_NUMS,
+YarnConfiguration.DEFAULT_FEDERATION_CACHE_ENTITY_NUMS);
+if (cacheTimeToLive <= 0) {
+ isCachingEnabled = false;
Review Comment:
I will add some logs.
> FederationStateStoreFacade Cache Support Caffeine
> -
>
> Key: YARN-11471
> URL: https://issues.apache.org/jira/browse/YARN-11471
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Affects Versions: 3.4.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
> Labels: pull-request-available
>
> FederationStateStoreFacade Cache Support Caffeine
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11471) FederationStateStoreFacade Cache Support Caffeine
[
https://issues.apache.org/jira/browse/YARN-11471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843617#comment-17843617
]
ASF GitHub Bot commented on YARN-11471:
---
slfan1989 commented on code in PR #6795:
URL: https://github.com/apache/hadoop/pull/6795#discussion_r1590520824
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationCaffeineCache.java:
##
@@ -0,0 +1,125 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+package org.apache.hadoop.yarn.server.federation.cache;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import
org.apache.hadoop.yarn.server.federation.store.records.SubClusterPolicyConfiguration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+public class FederationCaffeineCache extends FederationCache {
Review Comment:
@goiri Thank you for helping review the code! I will add some documentation.
> FederationStateStoreFacade Cache Support Caffeine
> -
>
> Key: YARN-11471
> URL: https://issues.apache.org/jira/browse/YARN-11471
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Affects Versions: 3.4.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
> Labels: pull-request-available
>
> FederationStateStoreFacade Cache Support Caffeine
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11694) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[
https://issues.apache.org/jira/browse/YARN-11694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843616#comment-17843616
]
ASF GitHub Bot commented on YARN-11694:
---
hadoop-yetus commented on PR #6793:
URL: https://github.com/apache/hadoop/pull/6793#issuecomment-2095166718
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 26s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 55s | | trunk passed |
| +1 :green_heart: | compile | 8m 46s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 8m 7s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 2m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 38s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 34s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 2m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 46s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 49s | | the patch passed |
| +1 :green_heart: | compile | 8m 27s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 8m 27s | | the patch passed |
| +1 :green_heart: | compile | 8m 3s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 8m 3s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 59s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6793/3/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 7 new + 10 unchanged - 5 fixed = 17 total (was
15) |
| +1 :green_heart: | mvnsite | 1m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 22s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 32s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 2m 46s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 27s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 58s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 1m 25s | |
hadoop-yarn-server-timelineservice in the patch passed. |
| +1 :green_heart: | unit | 11m 50s | | hadoop-sls in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 151m 12s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6793/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6793 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e323692159cd 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ef37ee4f8c1f48bd72be0fb8d4f75ef4f5faff50 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
[jira] [Commented] (YARN-11471) FederationStateStoreFacade Cache Support Caffeine
[
https://issues.apache.org/jira/browse/YARN-11471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843579#comment-17843579
]
ASF GitHub Bot commented on YARN-11471:
---
goiri commented on code in PR #6795:
URL: https://github.com/apache/hadoop/pull/6795#discussion_r1590365458
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationCaffeineCache.java:
##
@@ -0,0 +1,125 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+package org.apache.hadoop.yarn.server.federation.cache;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import
org.apache.hadoop.yarn.server.federation.store.records.SubClusterPolicyConfiguration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+public class FederationCaffeineCache extends FederationCache {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(FederationCaffeineCache.class);
+
+ private Cache cache;
+
+ private int cacheTimeToLive;
+ private long cacheEntityNums;
+
+ private String className = this.getClass().getSimpleName();
+
+ private boolean isCachingEnabled = false;
+
+ @Override
+ public boolean isCachingEnabled() {
+return isCachingEnabled;
+ }
+
+ @Override
+ public void initCache(Configuration pConf, FederationStateStore pStateStore)
{
+cacheTimeToLive =
pConf.getInt(YarnConfiguration.FEDERATION_CACHE_TIME_TO_LIVE_SECS,
+YarnConfiguration.DEFAULT_FEDERATION_CACHE_TIME_TO_LIVE_SECS);
+cacheEntityNums =
pConf.getLong(YarnConfiguration.FEDERATION_CACHE_ENTITY_NUMS,
+YarnConfiguration.DEFAULT_FEDERATION_CACHE_ENTITY_NUMS);
+if (cacheTimeToLive <= 0) {
+ isCachingEnabled = false;
Review Comment:
Log it?
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/cache/FederationCaffeineCache.java:
##
@@ -0,0 +1,125 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+package org.apache.hadoop.yarn.server.federation.cache;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import
org.apache.hadoop.yarn.server.federation.store.records.SubClusterPolicyConfiguration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.
[jira] [Commented] (YARN-11469) Refactor FederationStateStoreFacade Cache Code
[
https://issues.apache.org/jira/browse/YARN-11469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843562#comment-17843562
]
ASF GitHub Bot commented on YARN-11469:
---
hadoop-yetus commented on PR #6796:
URL: https://github.com/apache/hadoop/pull/6796#issuecomment-2094770260
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 49s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 37s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 29s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 42s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 38s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 1m 22s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 36m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 18s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common:
The patch generated 0 new + 0 unchanged - 11 fixed = 0 total (was 11) |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 25s | |
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common
generated 0 new + 44 unchanged - 1 fixed = 44 total (was 45) |
| +1 :green_heart: | spotbugs | 1m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 20s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 2m 57s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6796/2/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt)
| hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 134m 58s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.federation.policies.amrmproxy.TestLocalityMulticastAMRMProxyPolicy
|
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6796/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6796 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 77569b6f3797 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / f41739716669971d640ab5f294b3a14ed84cc8a8 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6796/2/testReport/ |
| Max. process+thread count | 747 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6796/2/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Refactor FederationStateStoreFacade Cache Code
> --
>
> Key: YARN-11469
> URL: https://issues.apache.org/jira/browse/YARN-11469
>
[jira] [Commented] (YARN-11469) Refactor FederationStateStoreFacade Cache Code
[
https://issues.apache.org/jira/browse/YARN-11469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843551#comment-17843551
]
ASF GitHub Bot commented on YARN-11469:
---
hadoop-yetus commented on PR #6796:
URL: https://github.com/apache/hadoop/pull/6796#issuecomment-2094746858
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 6m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 23s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 39s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 31s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 44s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 39s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 1m 26s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 38m 12s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 38m 33s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 17s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common:
The patch generated 0 new + 0 unchanged - 11 fixed = 0 total (was 11) |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | |
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common
generated 0 new + 44 unchanged - 1 fixed = 44 total (was 45) |
| +1 :green_heart: | spotbugs | 1m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 9s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 2m 55s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6796/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt)
| hadoop-yarn-server-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 142m 21s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.federation.policies.amrmproxy.TestLocalityMulticastAMRMProxyPolicy
|
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6796/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6796 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 7c05cf879860 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / edccf15f717b75b7de005aad3f663216b6ace96d |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6796/1/testReport/ |
| Max. process+thread count | 557 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6796/1/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Refactor FederationS
[jira] [Commented] (YARN-11469) Refactor FederationStateStoreFacade Cache Code
[ https://issues.apache.org/jira/browse/YARN-11469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843531#comment-17843531 ] ASF GitHub Bot commented on YARN-11469: --- slfan1989 opened a new pull request, #6796: URL: https://github.com/apache/hadoop/pull/6796 ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Refactor FederationStateStoreFacade Cache Code > -- > > Key: YARN-11469 > URL: https://issues.apache.org/jira/browse/YARN-11469 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > The Cache of FederationStateStoreFacade uses JCache, but considering that > JCache is not a general Cache implementation (the latest version was released > in 2014), this part of the code is refactored to support multiple Cache in > the future. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11694) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[
https://issues.apache.org/jira/browse/YARN-11694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843484#comment-17843484
]
ASF GitHub Bot commented on YARN-11694:
---
hadoop-yetus commented on PR #6793:
URL: https://github.com/apache/hadoop/pull/6793#issuecomment-2094341653
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 28s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 18s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 3s | | trunk passed |
| +1 :green_heart: | compile | 8m 48s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 8m 6s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 2m 10s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 34s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 35s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 2m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 58s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 48s | | the patch passed |
| +1 :green_heart: | compile | 8m 27s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 8m 27s | | the patch passed |
| +1 :green_heart: | compile | 8m 4s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 8m 4s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 59s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6793/2/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 7 new + 10 unchanged - 5 fixed = 17 total (was
15) |
| +1 :green_heart: | mvnsite | 1m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 30s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 33s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 2m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 35s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 55s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 1m 27s | |
hadoop-yarn-server-timelineservice in the patch passed. |
| +1 :green_heart: | unit | 11m 49s | | hadoop-sls in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 152m 5s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6793/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6793 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 6e3842f831ce 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 661559986f690c30cba068943abface26795bb52 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
[jira] [Commented] (YARN-11695) Fixed non-idempotent tests in `TestTaskRunner`
[ https://issues.apache.org/jira/browse/YARN-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843465#comment-17843465 ] ASF GitHub Bot commented on YARN-11695: --- kaiyaok2 closed pull request #6794: YARN-11695. Fixed non-idempotent unit tests in `TestTaskRunner` URL: https://github.com/apache/hadoop/pull/6794 > Fixed non-idempotent tests in `TestTaskRunner` > -- > > Key: YARN-11695 > URL: https://issues.apache.org/jira/browse/YARN-11695 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Kaiyao Ke >Priority: Major > Labels: pull-request-available > > All tests in `org.apache.hadoop.yarn.sls.scheduler.TestTaskRunner` are not > idempotent and fails upon repeated execution within the same JVM instance due > to self-induced state pollution. Specifically, the test runs made changes to > the static fields (e.g. `PreStartTask.first` in the task classes without > restoring them. Therefore, repeated runs throw assertion errors. > Sample error message of `TestTaskRunner#testPreStartQueueing` in repeated > test run: > ``` > java.lang.AssertionError: > at org.junit.Assert.fail(Assert.java:87) > at org.junit.Assert.assertTrue(Assert.java:42) > at org.junit.Assert.assertTrue(Assert.java:53) > at > org.apache.hadoop.yarn.sls.scheduler.TestTaskRunner.testPreStartQueueing(TestTaskRunner.java:244) > at java.base/java.lang.reflect.Method.invoke(Method.java:568) > ``` > The fix is done by explicitly setting (resetting) the static variables > (countdown latches and booleans) at the start of each test. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11695) Fixed non-idempotent tests in `TestTaskRunner`
[ https://issues.apache.org/jira/browse/YARN-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843464#comment-17843464 ] ASF GitHub Bot commented on YARN-11695: --- kaiyaok2 commented on PR #6794: URL: https://github.com/apache/hadoop/pull/6794#issuecomment-2094282705 Closing as code changes merged to #6793 > Fixed non-idempotent tests in `TestTaskRunner` > -- > > Key: YARN-11695 > URL: https://issues.apache.org/jira/browse/YARN-11695 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Kaiyao Ke >Priority: Major > Labels: pull-request-available > > All tests in `org.apache.hadoop.yarn.sls.scheduler.TestTaskRunner` are not > idempotent and fails upon repeated execution within the same JVM instance due > to self-induced state pollution. Specifically, the test runs made changes to > the static fields (e.g. `PreStartTask.first` in the task classes without > restoring them. Therefore, repeated runs throw assertion errors. > Sample error message of `TestTaskRunner#testPreStartQueueing` in repeated > test run: > ``` > java.lang.AssertionError: > at org.junit.Assert.fail(Assert.java:87) > at org.junit.Assert.assertTrue(Assert.java:42) > at org.junit.Assert.assertTrue(Assert.java:53) > at > org.apache.hadoop.yarn.sls.scheduler.TestTaskRunner.testPreStartQueueing(TestTaskRunner.java:244) > at java.base/java.lang.reflect.Method.invoke(Method.java:568) > ``` > The fix is done by explicitly setting (resetting) the static variables > (countdown latches and booleans) at the start of each test. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11695) Fixed non-idempotent tests in `TestTaskRunner`
[ https://issues.apache.org/jira/browse/YARN-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843463#comment-17843463 ] ASF GitHub Bot commented on YARN-11695: --- kaiyaok2 commented on PR #6794: URL: https://github.com/apache/hadoop/pull/6794#issuecomment-2094280302 > @kaiyaok2 Thank you for your contribution! Enhancing unit tests is a good thing. #6793 and #6794 should ideally be fixed together as they both involve the YARN module. It's better not to make it too scattered. Thanks for pointing this out. I've merged changes in this PR to #6793. > Fixed non-idempotent tests in `TestTaskRunner` > -- > > Key: YARN-11695 > URL: https://issues.apache.org/jira/browse/YARN-11695 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Kaiyao Ke >Priority: Major > Labels: pull-request-available > > All tests in `org.apache.hadoop.yarn.sls.scheduler.TestTaskRunner` are not > idempotent and fails upon repeated execution within the same JVM instance due > to self-induced state pollution. Specifically, the test runs made changes to > the static fields (e.g. `PreStartTask.first` in the task classes without > restoring them. Therefore, repeated runs throw assertion errors. > Sample error message of `TestTaskRunner#testPreStartQueueing` in repeated > test run: > ``` > java.lang.AssertionError: > at org.junit.Assert.fail(Assert.java:87) > at org.junit.Assert.assertTrue(Assert.java:42) > at org.junit.Assert.assertTrue(Assert.java:53) > at > org.apache.hadoop.yarn.sls.scheduler.TestTaskRunner.testPreStartQueueing(TestTaskRunner.java:244) > at java.base/java.lang.reflect.Method.invoke(Method.java:568) > ``` > The fix is done by explicitly setting (resetting) the static variables > (countdown latches and booleans) at the start of each test. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11694) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/YARN-11694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843462#comment-17843462 ] ASF GitHub Bot commented on YARN-11694: --- kaiyaok2 commented on PR #6793: URL: https://github.com/apache/hadoop/pull/6793#issuecomment-2094279458 > @kaiyaok2 Thank you for your contribution! but it does indeed introduce some noise (I have already seen at least 4 similar pull requests). Can we find all the relevant issues and fix them together? @slfan1989 Thanks for reviewing! I merged changes in #6794 to this PR (#6793). Now all changes to the MapReduce projects are in #6785 , and all changes to the Yarn projects are in this PR (#6793). Do you want me to further merge them? > 2 tests are non-idempotent (passes in the first run but fails in repeated > runs in the same JVM) > --- > > Key: YARN-11694 > URL: https://issues.apache.org/jira/browse/YARN-11694 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Kaiyao Ke >Priority: Major > Labels: pull-request-available > > ## TestTimelineReaderMetrics#testTimelineReaderMetrics > `org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderMetrics#testTimelineReaderMetrics` > does not perform a source unregistration after test execution, so the > `TimelineReaderMetrics.getInstance()` call in repeated runs will throw an > error since the metrics source `TimelineReaderMetrics` already exists. > Error message in the 2nd run: > ``` > org.apache.hadoop.metrics2.MetricsException: Metrics source > TimelineReaderMetrics already exists! > at > org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152) > at > org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125) > at > org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229) > at > org.apache.hadoop.yarn.server.timelineservice.metrics.TimelineReaderMetrics.getInstance(TimelineReaderMetrics.java:61) > at > org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderMetrics.setup(TestTimelineReaderMetrics.java:52) > at java.base/java.lang.reflect.Method.invoke(Method.java:568) > at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) > at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) > ``` > > ## TestFederationStateStoreClientMetrics#testSuccessfulCalls > `org.apache.hadoop.yarn.server.federation.store.metrics.TestFederationStateStoreClientMetrics#testSuccessfulCalls` > retrieves the historical number of successful calls, but does not retrieve > the historical average latency of those calls. For example, it asserts > `FederationStateStoreClientMetrics.getLatencySucceededCalls()` is 100 after > the `goodStateStore.registerSubCluster(100);` call. However, in the second > execution of the test, 2 historical calls from the first execution (with > latency 100 and 200 respectively) has already been recorded, so > `FederationStateStoreClientMetrics.getLatencySucceededCalls()` will be > 133. (mean of 100, 200 and 100) > > Error message in the 2nd run: > ``` > java.lang.AssertionError: expected:<100.0> but was:<133.34> > at org.junit.Assert.fail(Assert.java:89) > at org.junit.Assert.failNotEquals(Assert.java:835) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java:685) > at > org.apache.hadoop.yarn.server.federation.store.metrics.TestFederationStateStoreClientMetrics.testSuccessfulCalls(TestFederationStateStoreClientMetrics.java:63) > at java.base/java.lang.reflect.Method.invoke(Method.java:568) > ``` -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11471) FederationStateStoreFacade Cache Support Caffeine
[ https://issues.apache.org/jira/browse/YARN-11471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11471: -- Labels: pull-request-available (was: ) > FederationStateStoreFacade Cache Support Caffeine > - > > Key: YARN-11471 > URL: https://issues.apache.org/jira/browse/YARN-11471 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > > FederationStateStoreFacade Cache Support Caffeine -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11471) FederationStateStoreFacade Cache Support Caffeine
[ https://issues.apache.org/jira/browse/YARN-11471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843443#comment-17843443 ] ASF GitHub Bot commented on YARN-11471: --- slfan1989 opened a new pull request, #6795: URL: https://github.com/apache/hadoop/pull/6795 ### Description of PR JIRA: YARN-11471. FederationStateStoreFacade Cache Support Caffeine. ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > FederationStateStoreFacade Cache Support Caffeine > - > > Key: YARN-11471 > URL: https://issues.apache.org/jira/browse/YARN-11471 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > > FederationStateStoreFacade Cache Support Caffeine -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11695) Fixed non-idempotent tests in `TestTaskRunner`
[ https://issues.apache.org/jira/browse/YARN-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843420#comment-17843420 ] ASF GitHub Bot commented on YARN-11695: --- slfan1989 commented on PR #6794: URL: https://github.com/apache/hadoop/pull/6794#issuecomment-2094103774 @kaiyaok2 Thank you for your contribution! Enhancing unit tests is a good thing. #6793 and #6794 should ideally be fixed together as they both involve the YARN module. It's better not to make it too scattered. > Fixed non-idempotent tests in `TestTaskRunner` > -- > > Key: YARN-11695 > URL: https://issues.apache.org/jira/browse/YARN-11695 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Kaiyao Ke >Priority: Major > Labels: pull-request-available > > All tests in `org.apache.hadoop.yarn.sls.scheduler.TestTaskRunner` are not > idempotent and fails upon repeated execution within the same JVM instance due > to self-induced state pollution. Specifically, the test runs made changes to > the static fields (e.g. `PreStartTask.first` in the task classes without > restoring them. Therefore, repeated runs throw assertion errors. > Sample error message of `TestTaskRunner#testPreStartQueueing` in repeated > test run: > ``` > java.lang.AssertionError: > at org.junit.Assert.fail(Assert.java:87) > at org.junit.Assert.assertTrue(Assert.java:42) > at org.junit.Assert.assertTrue(Assert.java:53) > at > org.apache.hadoop.yarn.sls.scheduler.TestTaskRunner.testPreStartQueueing(TestTaskRunner.java:244) > at java.base/java.lang.reflect.Method.invoke(Method.java:568) > ``` > The fix is done by explicitly setting (resetting) the static variables > (countdown latches and booleans) at the start of each test. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11694) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/YARN-11694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843417#comment-17843417 ] ASF GitHub Bot commented on YARN-11694: --- slfan1989 commented on PR #6793: URL: https://github.com/apache/hadoop/pull/6793#issuecomment-2094100213 @kaiyaok2 Thank you for your contribution! but it does indeed introduce some noise (I have already seen at least 4 similar pull requests). Can we find all the relevant issues and fix them together? > 2 tests are non-idempotent (passes in the first run but fails in repeated > runs in the same JVM) > --- > > Key: YARN-11694 > URL: https://issues.apache.org/jira/browse/YARN-11694 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Kaiyao Ke >Priority: Major > Labels: pull-request-available > > ## TestTimelineReaderMetrics#testTimelineReaderMetrics > `org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderMetrics#testTimelineReaderMetrics` > does not perform a source unregistration after test execution, so the > `TimelineReaderMetrics.getInstance()` call in repeated runs will throw an > error since the metrics source `TimelineReaderMetrics` already exists. > Error message in the 2nd run: > ``` > org.apache.hadoop.metrics2.MetricsException: Metrics source > TimelineReaderMetrics already exists! > at > org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152) > at > org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125) > at > org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229) > at > org.apache.hadoop.yarn.server.timelineservice.metrics.TimelineReaderMetrics.getInstance(TimelineReaderMetrics.java:61) > at > org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderMetrics.setup(TestTimelineReaderMetrics.java:52) > at java.base/java.lang.reflect.Method.invoke(Method.java:568) > at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) > at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) > ``` > > ## TestFederationStateStoreClientMetrics#testSuccessfulCalls > `org.apache.hadoop.yarn.server.federation.store.metrics.TestFederationStateStoreClientMetrics#testSuccessfulCalls` > retrieves the historical number of successful calls, but does not retrieve > the historical average latency of those calls. For example, it asserts > `FederationStateStoreClientMetrics.getLatencySucceededCalls()` is 100 after > the `goodStateStore.registerSubCluster(100);` call. However, in the second > execution of the test, 2 historical calls from the first execution (with > latency 100 and 200 respectively) has already been recorded, so > `FederationStateStoreClientMetrics.getLatencySucceededCalls()` will be > 133. (mean of 100, 200 and 100) > > Error message in the 2nd run: > ``` > java.lang.AssertionError: expected:<100.0> but was:<133.34> > at org.junit.Assert.fail(Assert.java:89) > at org.junit.Assert.failNotEquals(Assert.java:835) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java:685) > at > org.apache.hadoop.yarn.server.federation.store.metrics.TestFederationStateStoreClientMetrics.testSuccessfulCalls(TestFederationStateStoreClientMetrics.java:63) > at java.base/java.lang.reflect.Method.invoke(Method.java:568) > ``` -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11695) Fixed non-idempotent tests in `TestTaskRunner`
[
https://issues.apache.org/jira/browse/YARN-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843412#comment-17843412
]
ASF GitHub Bot commented on YARN-11695:
---
hadoop-yetus commented on PR #6794:
URL: https://github.com/apache/hadoop/pull/6794#issuecomment-2094093469
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 24s | | trunk passed |
| +1 :green_heart: | compile | 0m 18s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 17s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 19s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 20s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 26s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 15s | | the patch passed |
| +1 :green_heart: | compile | 0m 14s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 14s | | the patch passed |
| +1 :green_heart: | compile | 0m 12s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 11s |
[/results-checkstyle-hadoop-tools_hadoop-sls.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6794/1/artifact/out/results-checkstyle-hadoop-tools_hadoop-sls.txt)
| hadoop-tools/hadoop-sls: The patch generated 5 new + 10 unchanged - 5 fixed
= 15 total (was 15) |
| +1 :green_heart: | mvnsite | 0m 15s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 15s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 15s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 42s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 11m 47s | | hadoop-sls in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not
generate ASF License warnings. |
| | | 93m 7s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6794/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6794 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b839beb78df7 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ea7f6f94f5ce97ecb4fbe1def221a61e86b58857 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6794/1/testReport/ |
| Max. process+thread count | 552 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-sls U: hadoop-tools/hadoop-sls |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/j
[jira] [Updated] (YARN-11695) Fixed non-idempotent tests in `TestTaskRunner`
[ https://issues.apache.org/jira/browse/YARN-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11695: -- Labels: pull-request-available (was: ) > Fixed non-idempotent tests in `TestTaskRunner` > -- > > Key: YARN-11695 > URL: https://issues.apache.org/jira/browse/YARN-11695 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Kaiyao Ke >Priority: Major > Labels: pull-request-available > > All tests in `org.apache.hadoop.yarn.sls.scheduler.TestTaskRunner` are not > idempotent and fails upon repeated execution within the same JVM instance due > to self-induced state pollution. Specifically, the test runs made changes to > the static fields (e.g. `PreStartTask.first` in the task classes without > restoring them. Therefore, repeated runs throw assertion errors. > Sample error message of `TestTaskRunner#testPreStartQueueing` in repeated > test run: > ``` > java.lang.AssertionError: > at org.junit.Assert.fail(Assert.java:87) > at org.junit.Assert.assertTrue(Assert.java:42) > at org.junit.Assert.assertTrue(Assert.java:53) > at > org.apache.hadoop.yarn.sls.scheduler.TestTaskRunner.testPreStartQueueing(TestTaskRunner.java:244) > at java.base/java.lang.reflect.Method.invoke(Method.java:568) > ``` > The fix is done by explicitly setting (resetting) the static variables > (countdown latches and booleans) at the start of each test. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11695) Fixed non-idempotent tests in `TestTaskRunner`
[ https://issues.apache.org/jira/browse/YARN-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843400#comment-17843400 ] ASF GitHub Bot commented on YARN-11695: --- kaiyaok2 opened a new pull request, #6794: URL: https://github.com/apache/hadoop/pull/6794 ### Description of PR Following up #6793 All tests in `org.apache.hadoop.yarn.sls.scheduler.TestTaskRunner` are not idempotent and fails upon repeated execution within the same JVM instance due to self-induced state pollution. Specifically, the test runs made changes to the static fields (e.g. `PreStartTask.first` in the task classes without restoring them. Therefore, repeated runs throw assertion errors. Sample error message of `TestTaskRunner#testPreStartQueueing` in repeated test run: ``` java.lang.AssertionError: at org.junit.Assert.fail(Assert.java:87) at org.junit.Assert.assertTrue(Assert.java:42) at org.junit.Assert.assertTrue(Assert.java:53) at org.apache.hadoop.yarn.sls.scheduler.TestTaskRunner.testPreStartQueueing(TestTaskRunner.java:244) at java.base/java.lang.reflect.Method.invoke(Method.java:568) ``` The fix is done by explicitly setting (resetting) the static variables (countdown latches and booleans) at the start of each test, so that each test runs on a fresh state. ### How was this patch tested? After the patch, rerunning the tests in the same JVM does not produce any exceptions. > Fixed non-idempotent tests in `TestTaskRunner` > -- > > Key: YARN-11695 > URL: https://issues.apache.org/jira/browse/YARN-11695 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Kaiyao Ke >Priority: Major > > All tests in `org.apache.hadoop.yarn.sls.scheduler.TestTaskRunner` are not > idempotent and fails upon repeated execution within the same JVM instance due > to self-induced state pollution. Specifically, the test runs made changes to > the static fields (e.g. `PreStartTask.first` in the task classes without > restoring them. Therefore, repeated runs throw assertion errors. > Sample error message of `TestTaskRunner#testPreStartQueueing` in repeated > test run: > ``` > java.lang.AssertionError: > at org.junit.Assert.fail(Assert.java:87) > at org.junit.Assert.assertTrue(Assert.java:42) > at org.junit.Assert.assertTrue(Assert.java:53) > at > org.apache.hadoop.yarn.sls.scheduler.TestTaskRunner.testPreStartQueueing(TestTaskRunner.java:244) > at java.base/java.lang.reflect.Method.invoke(Method.java:568) > ``` > The fix is done by explicitly setting (resetting) the static variables > (countdown latches and booleans) at the start of each test. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11694) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[
https://issues.apache.org/jira/browse/YARN-11694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843398#comment-17843398
]
ASF GitHub Bot commented on YARN-11694:
---
hadoop-yetus commented on PR #6793:
URL: https://github.com/apache/hadoop/pull/6793#issuecomment-2094060739
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 32s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 40s | | trunk passed |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 44s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 50s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 46s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 52s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 36s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6793/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt)
| hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 2
new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 37s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 32s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 37s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 44s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 1m 13s | |
hadoop-yarn-server-timelineservice in the patch passed. |
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not
generate ASF License warnings. |
| | | 96m 56s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6793/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6793 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 235fa8be1bd7 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c93cf852b1093bba87195834bfedd092a47d1b88 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Pr
[jira] [Updated] (YARN-11694) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/YARN-11694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11694: -- Labels: pull-request-available (was: ) > 2 tests are non-idempotent (passes in the first run but fails in repeated > runs in the same JVM) > --- > > Key: YARN-11694 > URL: https://issues.apache.org/jira/browse/YARN-11694 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Kaiyao Ke >Priority: Major > Labels: pull-request-available > > ## TestTimelineReaderMetrics#testTimelineReaderMetrics > `org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderMetrics#testTimelineReaderMetrics` > does not perform a source unregistration after test execution, so the > `TimelineReaderMetrics.getInstance()` call in repeated runs will throw an > error since the metrics source `TimelineReaderMetrics` already exists. > Error message in the 2nd run: > ``` > org.apache.hadoop.metrics2.MetricsException: Metrics source > TimelineReaderMetrics already exists! > at > org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152) > at > org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125) > at > org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229) > at > org.apache.hadoop.yarn.server.timelineservice.metrics.TimelineReaderMetrics.getInstance(TimelineReaderMetrics.java:61) > at > org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderMetrics.setup(TestTimelineReaderMetrics.java:52) > at java.base/java.lang.reflect.Method.invoke(Method.java:568) > at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) > at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) > ``` > > ## TestFederationStateStoreClientMetrics#testSuccessfulCalls > `org.apache.hadoop.yarn.server.federation.store.metrics.TestFederationStateStoreClientMetrics#testSuccessfulCalls` > retrieves the historical number of successful calls, but does not retrieve > the historical average latency of those calls. For example, it asserts > `FederationStateStoreClientMetrics.getLatencySucceededCalls()` is 100 after > the `goodStateStore.registerSubCluster(100);` call. However, in the second > execution of the test, 2 historical calls from the first execution (with > latency 100 and 200 respectively) has already been recorded, so > `FederationStateStoreClientMetrics.getLatencySucceededCalls()` will be > 133. (mean of 100, 200 and 100) > > Error message in the 2nd run: > ``` > java.lang.AssertionError: expected:<100.0> but was:<133.34> > at org.junit.Assert.fail(Assert.java:89) > at org.junit.Assert.failNotEquals(Assert.java:835) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java:685) > at > org.apache.hadoop.yarn.server.federation.store.metrics.TestFederationStateStoreClientMetrics.testSuccessfulCalls(TestFederationStateStoreClientMetrics.java:63) > at java.base/java.lang.reflect.Method.invoke(Method.java:568) > ``` -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11694) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/YARN-11694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843390#comment-17843390 ] ASF GitHub Bot commented on YARN-11694: --- kaiyaok2 opened a new pull request, #6793: URL: https://github.com/apache/hadoop/pull/6793 ## Description of PR SImilar as #6785 #6790 , this PR fixes 2 non-idempotent unit tests detected. These tests pass in the first run but fails in the second run in the same JVM. ### `TestTimelineReaderMetrics#testTimelineReaderMetrics` `org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderMetrics#testTimelineReaderMetrics` does not perform a source unregistration after test execution, so the `TimelineReaderMetrics.getInstance()` call in repeated runs will throw an error since the metrics source `TimelineReaderMetrics` already exists. Error message in the 2nd run: ``` org.apache.hadoop.metrics2.MetricsException: Metrics source TimelineReaderMetrics already exists! ``` Fix: Unregister `"TimelineReaderMetrics"` before the test. ### TestFederationStateStoreClientMetrics#testSuccessfulCalls `org.apache.hadoop.yarn.server.federation.store.metrics.TestFederationStateStoreClientMetrics#testSuccessfulCalls` retrieves the historical number of successful calls, but does not retrieve the historical average latency of those calls. For example, it asserts `FederationStateStoreClientMetrics.getLatencySucceededCalls()` is 100 after the `goodStateStore.registerSubCluster(100);` call. However, in the second execution of the test, 2 historical calls from the first execution (with latency 100 and 200 respectively) has already been recorded, so `FederationStateStoreClientMetrics.getLatencySucceededCalls()` will be 133.33... (mean of 100, 200 and 100) Error message in the 2nd run: ``` java.lang.AssertionError: expected:<100.0> but was:<133.34> ``` Fix: Retrieve existing latency data and use them for calculation. ### How was this patch tested? After the patch, rerunning the tests in the same JVM does not produce any exceptions. > 2 tests are non-idempotent (passes in the first run but fails in repeated > runs in the same JVM) > --- > > Key: YARN-11694 > URL: https://issues.apache.org/jira/browse/YARN-11694 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Kaiyao Ke >Priority: Major > > ## TestTimelineReaderMetrics#testTimelineReaderMetrics > `org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderMetrics#testTimelineReaderMetrics` > does not perform a source unregistration after test execution, so the > `TimelineReaderMetrics.getInstance()` call in repeated runs will throw an > error since the metrics source `TimelineReaderMetrics` already exists. > Error message in the 2nd run: > ``` > org.apache.hadoop.metrics2.MetricsException: Metrics source > TimelineReaderMetrics already exists! > at > org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152) > at > org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125) > at > org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229) > at > org.apache.hadoop.yarn.server.timelineservice.metrics.TimelineReaderMetrics.getInstance(TimelineReaderMetrics.java:61) > at > org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderMetrics.setup(TestTimelineReaderMetrics.java:52) > at java.base/java.lang.reflect.Method.invoke(Method.java:568) > at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) > at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) > ``` > > ## TestFederationStateStoreClientMetrics#testSuccessfulCalls > `org.apache.hadoop.yarn.server.federation.store.metrics.TestFederationStateStoreClientMetrics#testSuccessfulCalls` > retrieves the historical number of successful calls, but does not retrieve > the historical average latency of those calls. For example, it asserts > `FederationStateStoreClientMetrics.getLatencySucceededCalls()` is 100 after > the `goodStateStore.registerSubCluster(100);` call. However, in the second > execution of the test, 2 historical calls from the first execution (with > latency 100 and 200 respectively) has already been recorded, so > `FederationStateStoreClientMetrics.getLatencySucceededCalls()` will be > 133. (mean of 100, 200 and 100) > > Error message in the 2nd run: > ``` > java.lang.AssertionError: expected:<100.0> but was:<133.34> > at org.junit.Assert.fail(Assert.java:89) > at org.junit.Assert.failNotEquals(Assert.java:835) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java
[jira] [Commented] (YARN-11687) Update CGroupsResourceCalculator to track usages using cgroupv2
[
https://issues.apache.org/jira/browse/YARN-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843158#comment-17843158
]
ASF GitHub Bot commented on YARN-11687:
---
K0K0V0K commented on code in PR #6780:
URL: https://github.com/apache/hadoop/pull/6780#discussion_r1588929743
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsV2ResourceCalculator.java:
##
@@ -0,0 +1,135 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.util.CpuTimeTracker;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+
+/**
+ * A CGroupV2 file-system based Resource calculator without the process tree
features.
+ *
+ * The feature only works if cluster runs in pure V2 version, because when we
read the
+ * /proc/{pid}/cgroup file currently we can not handle multiple lines.
+ */
+public class CGroupsV2ResourceCalculator extends
AbstractCGroupsResourceCalculator {
+ private static final Logger LOG =
LoggerFactory.getLogger(CGroupsV2ResourceCalculator.class);
+ private final Map stats = new ConcurrentHashMap<>();
+
+ @VisibleForTesting
+ String root = "/";
Review Comment:
Well, good question ... based on
- currently, cgroup V1 and V2 only work on Linux, so we do not have to
prepare for Windows for example
- the previous V1 version used "/proc" dir as procfsDir as a hardcoded
parameter
I would like to keep this "/" in name of the KISS.
This hack is visible for testing cause in unit test we are using a temporal
directory, as "root"
> Update CGroupsResourceCalculator to track usages using cgroupv2
> ---
>
> Key: YARN-11687
> URL: https://issues.apache.org/jira/browse/YARN-11687
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Bence Kosztolnik
>Priority: Major
> Labels: pull-request-available
>
> [CGroupsResourceCalculator|https://github.com/apache/hadoop/blob/f609460bda0c2bd87dd3580158e549e2f34f14d5/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsResourceCalculator.java]
> should also be updated to handle the cgroup v2 changes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11687) Update CGroupsResourceCalculator to track usages using cgroupv2
[
https://issues.apache.org/jira/browse/YARN-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843015#comment-17843015
]
ASF GitHub Bot commented on YARN-11687:
---
brumi1024 commented on code in PR #6780:
URL: https://github.com/apache/hadoop/pull/6780#discussion_r1587900352
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsV2ResourceCalculator.java:
##
@@ -0,0 +1,135 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.util.CpuTimeTracker;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+
+/**
+ * A CGroupV2 file-system based Resource calculator without the process tree
features.
+ *
+ * The feature only works if cluster runs in pure V2 version, because when we
read the
+ * /proc/{pid}/cgroup file currently we can not handle multiple lines.
+ */
+public class CGroupsV2ResourceCalculator extends
AbstractCGroupsResourceCalculator {
+ private static final Logger LOG =
LoggerFactory.getLogger(CGroupsV2ResourceCalculator.class);
+ private final Map stats = new ConcurrentHashMap<>();
+
+ @VisibleForTesting
+ String root = "/";
+
+ /**
+ * Create resource calculator for the container that has the specified pid.
+ * @param pid A pid from the cgroup or null for all containers
+ */
+ public CGroupsV2ResourceCalculator(String pid) {
+super(pid);
+ }
+
+ @Override
+ public void initialize() throws YarnException {
+cpuTimeTracker = new CpuTimeTracker(jiffyLengthMs);
Review Comment:
The v1 calculator has an isAvailable check. Is this intentionally missed
here?
> Update CGroupsResourceCalculator to track usages using cgroupv2
> ---
>
> Key: YARN-11687
> URL: https://issues.apache.org/jira/browse/YARN-11687
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Bence Kosztolnik
>Priority: Major
> Labels: pull-request-available
>
> [CGroupsResourceCalculator|https://github.com/apache/hadoop/blob/f609460bda0c2bd87dd3580158e549e2f34f14d5/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsResourceCalculator.java]
> should also be updated to handle the cgroup v2 changes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11687) Update CGroupsResourceCalculator to track usages using cgroupv2
[
https://issues.apache.org/jira/browse/YARN-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843014#comment-17843014
]
ASF GitHub Bot commented on YARN-11687:
---
brumi1024 commented on code in PR #6780:
URL: https://github.com/apache/hadoop/pull/6780#discussion_r1587889122
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsV2ResourceCalculator.java:
##
@@ -0,0 +1,135 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.util.CpuTimeTracker;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+
+/**
+ * A CGroupV2 file-system based Resource calculator without the process tree
features.
+ *
+ * The feature only works if cluster runs in pure V2 version, because when we
read the
+ * /proc/{pid}/cgroup file currently we can not handle multiple lines.
+ */
+public class CGroupsV2ResourceCalculator extends
AbstractCGroupsResourceCalculator {
+ private static final Logger LOG =
LoggerFactory.getLogger(CGroupsV2ResourceCalculator.class);
+ private final Map stats = new ConcurrentHashMap<>();
+
+ @VisibleForTesting
+ String root = "/";
+
+ /**
+ * Create resource calculator for the container that has the specified pid.
+ * @param pid A pid from the cgroup or null for all containers
+ */
+ public CGroupsV2ResourceCalculator(String pid) {
+super(pid);
+ }
+
+ @Override
+ public void initialize() throws YarnException {
+cpuTimeTracker = new CpuTimeTracker(jiffyLengthMs);
+cGroupsHandler = ResourceHandlerModule.getCGroupsHandler();
+ }
+
+ @Override
+ public long getCumulativeCpuTime() {
+// https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu-interface-files
+return jiffyLengthMs < 0
+? UNAVAILABLE
+: getStat("cpu.stat#usage_usec") * jiffyLengthMs;
+ }
+
+ @Override
+ public long getRssMemorySize(int olderThanAge) {
+//
https://docs.kernel.org/admin-guide/cgroup-v2.html#memory-interface-files
+return 1 < olderThanAge
+? UNAVAILABLE
+: getStat("memory.stat#anon");
+ }
+
+ @Override
+ public long getVirtualMemorySize(int olderThanAge) {
+//
https://docs.kernel.org/admin-guide/cgroup-v2.html#memory-interface-files
+return 1 < olderThanAge
+? UNAVAILABLE
+: getStat("memory.stat#vmalloc");
+ }
+
+ @Override
+ public void updateProcessTree() {
+try (Stream cGroupFiles = Files.list(getCGroupPath())){
+ List statFiles = cGroupFiles
+ .filter(path -> path.toString().endsWith(".stat"))
+ .collect(Collectors.toList());
+ for (Path statFile : statFiles) {
+String[] lines = fileToString(statFile).split(System.lineSeparator());
+for (String line: lines) {
+ String[] parts = line.split(" ");
+ stats.put(statFile.getFileName() + "#" + parts[0], parts[1]);
+}
+ }
+ cpuTimeTracker.updateElapsedJiffies(
+ BigInteger.valueOf(getStat("cpu.stat#usage_usec")), clock.getTime());
Review Comment:
`cpu.stat#usage_usec` could be a final string.
> Update CGroupsResourceCalculator to track usages using cgroupv2
> ---
>
> Key: YARN-11687
> URL: https://issues.apache.org/jira/browse/YARN-11687
> Project: Hadoop YARN
> Issue Type: Sub-task
>Repo
[jira] [Commented] (YARN-11687) Update CGroupsResourceCalculator to track usages using cgroupv2
[
https://issues.apache.org/jira/browse/YARN-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843013#comment-17843013
]
ASF GitHub Bot commented on YARN-11687:
---
brumi1024 commented on code in PR #6780:
URL: https://github.com/apache/hadoop/pull/6780#discussion_r1587889935
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsV2ResourceCalculator.java:
##
@@ -0,0 +1,135 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.util.CpuTimeTracker;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+
+/**
+ * A CGroupV2 file-system based Resource calculator without the process tree
features.
+ *
+ * The feature only works if cluster runs in pure V2 version, because when we
read the
+ * /proc/{pid}/cgroup file currently we can not handle multiple lines.
+ */
+public class CGroupsV2ResourceCalculator extends
AbstractCGroupsResourceCalculator {
+ private static final Logger LOG =
LoggerFactory.getLogger(CGroupsV2ResourceCalculator.class);
+ private final Map stats = new ConcurrentHashMap<>();
+
+ @VisibleForTesting
+ String root = "/";
Review Comment:
Maybe `File.listRoots()`?
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsV2ResourceCalculator.java:
##
@@ -0,0 +1,135 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.util.CpuTimeTracker;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+
+/**
+ * A CGroupV2 file-system based Resource calculator without the process tree
features.
+ *
+ * The feature only works if cluster runs in pure V2 version, because when we
read the
+ * /proc/{pid}/cgroup file currently we can not handle multiple lines.
+ */
+public class CGroupsV2ResourceCalculator extends
AbstractCGroupsResourceCalculator {
+ private static fin
[jira] [Commented] (YARN-11687) Update CGroupsResourceCalculator to track usages using cgroupv2
[
https://issues.apache.org/jira/browse/YARN-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842949#comment-17842949
]
ASF GitHub Bot commented on YARN-11687:
---
hadoop-yetus commented on PR #6780:
URL: https://github.com/apache/hadoop/pull/6780#issuecomment-2090344394
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 16m 1s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 23s | | trunk passed |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 48s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 50s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 59s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 1m 20s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 20s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 28s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/7/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 52s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 25m 23s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 172m 26s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6780 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e0ae52660f3d 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d09276dc2242e8e3ec036576a0e91f21ae48cbba |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/7/testReport/ |
| Max. proce
[jira] [Commented] (YARN-11687) Update CGroupsResourceCalculator to track usages using cgroupv2
[
https://issues.apache.org/jira/browse/YARN-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842905#comment-17842905
]
ASF GitHub Bot commented on YARN-11687:
---
K0K0V0K commented on code in PR #6780:
URL: https://github.com/apache/hadoop/pull/6780#discussion_r1587324661
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsV2ResourceCalculator.java:
##
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.util.CpuTimeTracker;
+import org.apache.hadoop.util.SysInfoLinux;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.util.Clock;
+import org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree;
+import org.apache.hadoop.yarn.util.SystemClock;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * A CGroupV2 file-system based Resource calculator without the process tree
features.
+ *
+ * The feature only works if cluster runs in pure V2 version, because when we
read the
+ * /proc/{pid}/cgroup file currently we can not handle multiple lines.
+ */
+public class CGroupsV2ResourceCalculator extends ResourceCalculatorProcessTree
{
Review Comment:
Thanks @brumi1024 for the review!
I unified the exact same methods. If you think I can try to create other
common methods, but then I will have to refactor the V1 implementation.
Should we do that?
> Update CGroupsResourceCalculator to track usages using cgroupv2
> ---
>
> Key: YARN-11687
> URL: https://issues.apache.org/jira/browse/YARN-11687
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Bence Kosztolnik
>Priority: Major
> Labels: pull-request-available
>
> [CGroupsResourceCalculator|https://github.com/apache/hadoop/blob/f609460bda0c2bd87dd3580158e549e2f34f14d5/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsResourceCalculator.java]
> should also be updated to handle the cgroup v2 changes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11687) Update CGroupsResourceCalculator to track usages using cgroupv2
[
https://issues.apache.org/jira/browse/YARN-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842904#comment-17842904
]
ASF GitHub Bot commented on YARN-11687:
---
K0K0V0K commented on code in PR #6780:
URL: https://github.com/apache/hadoop/pull/6780#discussion_r1587324661
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsV2ResourceCalculator.java:
##
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.util.CpuTimeTracker;
+import org.apache.hadoop.util.SysInfoLinux;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.util.Clock;
+import org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree;
+import org.apache.hadoop.yarn.util.SystemClock;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * A CGroupV2 file-system based Resource calculator without the process tree
features.
+ *
+ * The feature only works if cluster runs in pure V2 version, because when we
read the
+ * /proc/{pid}/cgroup file currently we can not handle multiple lines.
+ */
+public class CGroupsV2ResourceCalculator extends ResourceCalculatorProcessTree
{
Review Comment:
Thanks, @brumi1024 I unified the exact same methods. If you think I can try
to create other common methods, but then I will have to refactor the V1
implementation. Should we do that?
> Update CGroupsResourceCalculator to track usages using cgroupv2
> ---
>
> Key: YARN-11687
> URL: https://issues.apache.org/jira/browse/YARN-11687
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Bence Kosztolnik
>Priority: Major
> Labels: pull-request-available
>
> [CGroupsResourceCalculator|https://github.com/apache/hadoop/blob/f609460bda0c2bd87dd3580158e549e2f34f14d5/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsResourceCalculator.java]
> should also be updated to handle the cgroup v2 changes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11687) Update CGroupsResourceCalculator to track usages using cgroupv2
[
https://issues.apache.org/jira/browse/YARN-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842886#comment-17842886
]
ASF GitHub Bot commented on YARN-11687:
---
brumi1024 commented on code in PR #6780:
URL: https://github.com/apache/hadoop/pull/6780#discussion_r1587268842
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsV2ResourceCalculator.java:
##
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.util.CpuTimeTracker;
+import org.apache.hadoop.util.SysInfoLinux;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.util.Clock;
+import org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree;
+import org.apache.hadoop.yarn.util.SystemClock;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * A CGroupV2 file-system based Resource calculator without the process tree
features.
+ *
+ * The feature only works if cluster runs in pure V2 version, because when we
read the
+ * /proc/{pid}/cgroup file currently we can not handle multiple lines.
+ */
+public class CGroupsV2ResourceCalculator extends ResourceCalculatorProcessTree
{
Review Comment:
Some of the methods are quite similar between this and the original
CgroupResourceCalculator. To keep the implementation simple and make future
changes/fixes easier for both versions I suggest moving the similar methods to
an abstract class and only overriding the different parts, like the new
ResourceHandlers are doing.
> Update CGroupsResourceCalculator to track usages using cgroupv2
> ---
>
> Key: YARN-11687
> URL: https://issues.apache.org/jira/browse/YARN-11687
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Bence Kosztolnik
>Priority: Major
> Labels: pull-request-available
>
> [CGroupsResourceCalculator|https://github.com/apache/hadoop/blob/f609460bda0c2bd87dd3580158e549e2f34f14d5/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsResourceCalculator.java]
> should also be updated to handle the cgroup v2 changes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11687) Update CGroupsResourceCalculator to track usages using cgroupv2
[
https://issues.apache.org/jira/browse/YARN-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842885#comment-17842885
]
ASF GitHub Bot commented on YARN-11687:
---
brumi1024 commented on code in PR #6780:
URL: https://github.com/apache/hadoop/pull/6780#discussion_r1587268842
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsV2ResourceCalculator.java:
##
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.util.CpuTimeTracker;
+import org.apache.hadoop.util.SysInfoLinux;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.util.Clock;
+import org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree;
+import org.apache.hadoop.yarn.util.SystemClock;
+
+import java.io.IOException;
+import java.math.BigInteger;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * A CGroupV2 file-system based Resource calculator without the process tree
features.
+ *
+ * The feature only works if cluster runs in pure V2 version, because when we
read the
+ * /proc/{pid}/cgroup file currently we can not handle multiple lines.
+ */
+public class CGroupsV2ResourceCalculator extends ResourceCalculatorProcessTree
{
Review Comment:
Some of the methods are quite similar between this and the original
CgroupResourceCalculator. To keep the implementation simple and make future
changes/fixes easier I suggest moving the similar methods to an abstract class
and only overriding the different parts, like the new ResourceHandlers are
doing.
> Update CGroupsResourceCalculator to track usages using cgroupv2
> ---
>
> Key: YARN-11687
> URL: https://issues.apache.org/jira/browse/YARN-11687
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Bence Kosztolnik
>Priority: Major
> Labels: pull-request-available
>
> [CGroupsResourceCalculator|https://github.com/apache/hadoop/blob/f609460bda0c2bd87dd3580158e549e2f34f14d5/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsResourceCalculator.java]
> should also be updated to handle the cgroup v2 changes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11693) Refactor Container scheduler
[
https://issues.apache.org/jira/browse/YARN-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842338#comment-17842338
]
ASF GitHub Bot commented on YARN-11693:
---
hadoop-yetus commented on PR #6786:
URL: https://github.com/apache/hadoop/pull/6786#issuecomment-2085033047
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 41s | | trunk passed |
| +1 :green_heart: | compile | 0m 53s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 50s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 22s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 54s | | trunk passed |
| -1 :x: | shadedclient | 24m 2s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 14s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6786/1/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch failed. |
| -1 :x: | compile | 0m 16s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6786/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| hadoop-yarn-server-nodemanager in the patch failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | javac | 0m 16s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6786/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkUbuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1.txt)
| hadoop-yarn-server-nodemanager in the patch failed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1. |
| -1 :x: | compile | 0m 15s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6786/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt)
| hadoop-yarn-server-nodemanager in the patch failed with JDK Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06. |
| -1 :x: | javac | 0m 15s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6786/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkPrivateBuild-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06.txt)
| hadoop-yarn-server-nodemanager in the patch failed with JDK Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 14s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/ha
[jira] [Updated] (YARN-11693) Refactor Container scheduler
[ https://issues.apache.org/jira/browse/YARN-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11693: -- Labels: pull-request-available (was: ) > Refactor Container scheduler > > > Key: YARN-11693 > URL: https://issues.apache.org/jira/browse/YARN-11693 > Project: Hadoop YARN > Issue Type: Task > Components: scheduler, scheduler preemption >Reporter: Mohit Gaggar >Priority: Major > Labels: pull-request-available > > Container Scheduler class, responsible for scheduling containers on nodes > handles multiple smaller responsibilities making it hard to extend the > functionalities. > This PR works on breaking down the class responsibilities into > * ContainerQueueManager : handles all queuing related functions, like > adding/removing to queue > * ContainerStarter : maintains the running queue of containers and starts > new containers > * ContainerPolicyManager : handles the container termination/pausing policy > when enough resources not available > * ContainerScheduler : main class which works with other helper classes to > maintain container queues > !https://msdata.visualstudio.com/25bee5cc-1a60-44a1-904d-a734363b40d4/_apis/git/repositories/719ef898-e962-4b70-a49b-03c67abb2b07/pullRequests/1249358/attachments/Refactoring%20Container%20Scheduler%20%281%29.png|width=710,height=441! > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11693) Refactor Container scheduler
[ https://issues.apache.org/jira/browse/YARN-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842324#comment-17842324 ] ASF GitHub Bot commented on YARN-11693: --- mohitgaggar opened a new pull request, #6786: URL: https://github.com/apache/hadoop/pull/6786 …, running container manager and container queue manager classes ### Description of PR ### How was this patch tested? ### For code changes: - [x ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Refactor Container scheduler > > > Key: YARN-11693 > URL: https://issues.apache.org/jira/browse/YARN-11693 > Project: Hadoop YARN > Issue Type: Task > Components: scheduler, scheduler preemption >Reporter: Mohit Gaggar >Priority: Major > > Container Scheduler class, responsible for scheduling containers on nodes > handles multiple smaller responsibilities making it hard to extend the > functionalities. > This PR works on breaking down the class responsibilities into > * ContainerQueueManager : handles all queuing related functions, like > adding/removing to queue > * ContainerStarter : maintains the running queue of containers and starts > new containers > * ContainerPolicyManager : handles the container termination/pausing policy > when enough resources not available > * ContainerScheduler : main class which works with other helper classes to > maintain container queues > !https://msdata.visualstudio.com/25bee5cc-1a60-44a1-904d-a734363b40d4/_apis/git/repositories/719ef898-e962-4b70-a49b-03c67abb2b07/pullRequests/1249358/attachments/Refactoring%20Container%20Scheduler%20%281%29.png|width=710,height=441! > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11685) Create a config to enable/disable cgroup v2 functionality
[ https://issues.apache.org/jira/browse/YARN-11685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842315#comment-17842315 ] ASF GitHub Bot commented on YARN-11685: --- brumi1024 merged PR #6770: URL: https://github.com/apache/hadoop/pull/6770 > Create a config to enable/disable cgroup v2 functionality > - > > Key: YARN-11685 > URL: https://issues.apache.org/jira/browse/YARN-11685 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Peter Szucs >Priority: Major > Labels: pull-request-available > > Various OS's mount the cgroup v2 differently, some of them mount both the v1 > and v2 structure, others mount a hybrid structure. To avoid initialization > issues the cgroup v1/v2 functionality should be set by a config property. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11685) Create a config to enable/disable cgroup v2 functionality
[ https://issues.apache.org/jira/browse/YARN-11685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842314#comment-17842314 ] ASF GitHub Bot commented on YARN-11685: --- brumi1024 commented on PR #6770: URL: https://github.com/apache/hadoop/pull/6770#issuecomment-2084814001 Thanks @p-szucs for the fixes, the latest state LGTM. Merging to trunk. > Create a config to enable/disable cgroup v2 functionality > - > > Key: YARN-11685 > URL: https://issues.apache.org/jira/browse/YARN-11685 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Peter Szucs >Priority: Major > Labels: pull-request-available > > Various OS's mount the cgroup v2 differently, some of them mount both the v1 > and v2 structure, others mount a hybrid structure. To avoid initialization > issues the cgroup v1/v2 functionality should be set by a config property. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11691) Add HTTP POST method to yarn web proxy servlet
[ https://issues.apache.org/jira/browse/YARN-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842224#comment-17842224 ] ASF GitHub Bot commented on YARN-11691: --- zuston commented on PR #6781: URL: https://github.com/apache/hadoop/pull/6781#issuecomment-2084267643 LGTM. It's a case that flink encounters. Please take a look @slfan1989 . > Add HTTP POST method to yarn web proxy servlet > -- > > Key: YARN-11691 > URL: https://issues.apache.org/jira/browse/YARN-11691 > Project: Hadoop YARN > Issue Type: Bug > Components: webproxy >Affects Versions: 3.2.2, 3.3.6 >Reporter: ude >Priority: Major > Labels: pull-request-available > Fix For: 3.2.2, 3.3.6 > > > When the flink task is running in the YARN environment, the client encounters > an error HTTP ERROR 405 when calling the http proxy rest api by _POST_ Method. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11687) Update CGroupsResourceCalculator to track usages using cgroupv2
[
https://issues.apache.org/jira/browse/YARN-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842142#comment-17842142
]
ASF GitHub Bot commented on YARN-11687:
---
hadoop-yetus commented on PR #6780:
URL: https://github.com/apache/hadoop/pull/6780#issuecomment-2083539073
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 23s | | trunk passed |
| +1 :green_heart: | compile | 1m 33s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 17s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 1m 23s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 28s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/6/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 25m 0s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 173m 18s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6780 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 51b4e5f53a1a 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 72fcc8efc6122e66d26dcf9651aebed7f889e21b |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/6/testReport/ |
| Max. proce
[jira] [Commented] (YARN-11685) Create a config to enable/disable cgroup v2 functionality
[
https://issues.apache.org/jira/browse/YARN-11685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842139#comment-17842139
]
ASF GitHub Bot commented on YARN-11685:
---
hadoop-yetus commented on PR #6770:
URL: https://github.com/apache/hadoop/pull/6770#issuecomment-2083495643
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 59s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 37m 4s | | trunk passed |
| +1 :green_heart: | compile | 8m 7s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 7m 21s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 49s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 2m 34s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 5m 29s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 59s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 45s | | the patch passed |
| +1 :green_heart: | compile | 7m 21s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 7m 21s | | the patch passed |
| +1 :green_heart: | compile | 7m 14s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 7m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 30s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 2m 22s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 5m 50s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 6s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 20s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 25m 5s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 54s | | The patch does not
generate ASF License warnings. |
| | | 235m 13s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6770/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6770 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 71ed1b17070a 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 11c9f8076f96a95e52756d6fe60d5ed06302b9f6 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6770/6/testReport/ |
| Max. process+t
[jira] [Commented] (YARN-11687) Update CGroupsResourceCalculator to track usages using cgroupv2
[
https://issues.apache.org/jira/browse/YARN-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842137#comment-17842137
]
ASF GitHub Bot commented on YARN-11687:
---
hadoop-yetus commented on PR #6780:
URL: https://github.com/apache/hadoop/pull/6780#issuecomment-2083490857
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 2s | | trunk passed |
| +1 :green_heart: | compile | 1m 27s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 39s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 47s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 50s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 5s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 28s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/5/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 25m 12s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 157m 14s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6780 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 6526566d334f 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d822768d6aa4e47280537aa46b9b77f715ba06e5 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/5/testReport/ |
| Max. proce
[jira] [Commented] (YARN-11687) Update CGroupsResourceCalculator to track usages using cgroupv2
[
https://issues.apache.org/jira/browse/YARN-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842044#comment-17842044
]
ASF GitHub Bot commented on YARN-11687:
---
hadoop-yetus commented on PR #6780:
URL: https://github.com/apache/hadoop/pull/6780#issuecomment-2083057386
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 45s | | trunk passed |
| +1 :green_heart: | compile | 1m 30s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 5s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 1m 23s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 28s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 30s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 57s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 174m 2s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6780 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux bfcc9b891f22 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b8d6fb31c828078f20ba29964b80402890f48c58 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.
[jira] [Commented] (YARN-11685) Create a config to enable/disable cgroup v2 functionality
[
https://issues.apache.org/jira/browse/YARN-11685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842040#comment-17842040
]
ASF GitHub Bot commented on YARN-11685:
---
hadoop-yetus commented on PR #6770:
URL: https://github.com/apache/hadoop/pull/6770#issuecomment-2083052404
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 27s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 58s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 5s | | trunk passed |
| +1 :green_heart: | compile | 7m 26s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 7m 3s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 2s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 55s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 2m 49s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 5m 42s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 34s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 47s | | the patch passed |
| +1 :green_heart: | compile | 6m 52s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 6m 52s | | the patch passed |
| +1 :green_heart: | compile | 6m 54s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 6m 54s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 53s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6770/4/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 11 new + 165 unchanged
- 0 fixed = 176 total (was 165) |
| +1 :green_heart: | mvnsite | 2m 43s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 36s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 2m 34s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 6m 2s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 34s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 11s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 35s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 25m 34s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 1m 0s | | The patch does not
generate ASF License warnings. |
| | | 222m 55s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6770/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6770 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 8dfb89689c72 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1a40088f367fd971f35e1c7277b6fb7abb696c2e |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/
[jira] [Commented] (YARN-11685) Create a config to enable/disable cgroup v2 functionality
[
https://issues.apache.org/jira/browse/YARN-11685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842036#comment-17842036
]
ASF GitHub Bot commented on YARN-11685:
---
p-szucs commented on code in PR #6770:
URL: https://github.com/apache/hadoop/pull/6770#discussion_r1583282084
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java:
##
@@ -69,9 +70,9 @@ public class ResourceHandlerModule {
private static volatile CGroupsHandler cGroupsHandler;
private static volatile CGroupsBlkioResourceHandlerImpl
cGroupsBlkioResourceHandler;
- private static volatile CGroupsMemoryResourceHandlerImpl
+ private static volatile AbstractCGroupsMemoryResourceHandler
Review Comment:
Thanks @brumi1024 for the review! Fixed your comment and rebased the patch
to the current trunk.
> Create a config to enable/disable cgroup v2 functionality
> -
>
> Key: YARN-11685
> URL: https://issues.apache.org/jira/browse/YARN-11685
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Peter Szucs
>Priority: Major
> Labels: pull-request-available
>
> Various OS's mount the cgroup v2 differently, some of them mount both the v1
> and v2 structure, others mount a hybrid structure. To avoid initialization
> issues the cgroup v1/v2 functionality should be set by a config property.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11687) Update CGroupsResourceCalculator to track usages using cgroupv2
[
https://issues.apache.org/jira/browse/YARN-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842030#comment-17842030
]
ASF GitHub Bot commented on YARN-11687:
---
hadoop-yetus commented on PR #6780:
URL: https://github.com/apache/hadoop/pull/6780#issuecomment-2083028134
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 11s | | trunk passed |
| +1 :green_heart: | compile | 0m 47s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 47s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 19s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 22s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 51s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 8s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 21s | | the patch passed |
| +1 :green_heart: | compile | 0m 47s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 47s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 15s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/4/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 55s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 38s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 21m 58s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 24s | | The patch does not
generate ASF License warnings. |
| | | 110m 48s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6780 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b7fedb983074 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 39d735158bfbcc38da40d76525ca71cf48b1f6ef |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/4/testReport/ |
| Max. proce
[jira] [Commented] (YARN-11687) Update CGroupsResourceCalculator to track usages using cgroupv2
[
https://issues.apache.org/jira/browse/YARN-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842010#comment-17842010
]
ASF GitHub Bot commented on YARN-11687:
---
hadoop-yetus commented on PR #6780:
URL: https://github.com/apache/hadoop/pull/6780#issuecomment-2082936509
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 38s | | trunk passed |
| +1 :green_heart: | compile | 0m 50s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 47s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 21s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 22s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 53s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 8s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 17s | | the patch passed |
| +1 :green_heart: | compile | 0m 44s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 44s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 12s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/3/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 21s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 49s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 26s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 21m 39s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 22s | | The patch does not
generate ASF License warnings. |
| | | 107m 59s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6780 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e0282c8bbe99 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9b7e44c2af451ed70352eaa2cf600b2bf53cbe08 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/3/testReport/ |
| Max. proce
[jira] [Commented] (YARN-11685) Create a config to enable/disable cgroup v2 functionality
[
https://issues.apache.org/jira/browse/YARN-11685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842007#comment-17842007
]
ASF GitHub Bot commented on YARN-11685:
---
brumi1024 commented on code in PR #6770:
URL: https://github.com/apache/hadoop/pull/6770#discussion_r1583197107
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java:
##
@@ -69,9 +70,9 @@ public class ResourceHandlerModule {
private static volatile CGroupsHandler cGroupsHandler;
private static volatile CGroupsBlkioResourceHandlerImpl
cGroupsBlkioResourceHandler;
- private static volatile CGroupsMemoryResourceHandlerImpl
+ private static volatile AbstractCGroupsMemoryResourceHandler
Review Comment:
Nit: there might not always be an abstract parent class for every
ResourceHandler, instead we could simply return the interface like some of the
getters already do.
AbstractCGroupsMemoryResourceHandler -> MemoryResourceHandler
AbstractCGroupsCpuResourceHandler -> CpuResourceHandler
> Create a config to enable/disable cgroup v2 functionality
> -
>
> Key: YARN-11685
> URL: https://issues.apache.org/jira/browse/YARN-11685
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Peter Szucs
>Priority: Major
> Labels: pull-request-available
>
> Various OS's mount the cgroup v2 differently, some of them mount both the v1
> and v2 structure, others mount a hybrid structure. To avoid initialization
> issues the cgroup v1/v2 functionality should be set by a config property.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842002#comment-17842002
]
ASF GitHub Bot commented on YARN-11675:
---
brumi1024 commented on PR #6760:
URL: https://github.com/apache/hadoop/pull/6760#issuecomment-2082896070
Thanks @p-szucs for the patch, LGTM. Merging to trunk.
> Update MemoryResourceHandler implementation for cgroup v2 support
> -
>
> Key: YARN-11675
> URL: https://issues.apache.org/jira/browse/YARN-11675
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Peter Szucs
>Priority: Major
> Labels: pull-request-available
>
> cgroup v2 has some changes in various controllers (some changed their
> functionality, some were removed). This task is about updating
> MemoryResourceHandler's
> [implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46].
> h3. *Differences in the controls comparing to cgroup v1:*
> h3. Hard limit on memory
> {_}memory{_}.{_}limit_in_bytes{_} control is replaced with _memory.max_
> h3. Soft limit on memory
> {_}memory{_}.soft_{_}limit_in_bytes{_} control is replaced with _memory.low_
> Detailed descriptions about the memory controls can be found in the official
> [cgroup v2 documentation|https://docs.kernel.org/admin-guide/cgroup-v2.html].
> h3. Swappiness
> _memory.swappiness_ has been removed from the available cgroup v2 controls.
> Quoting [redhat documentation|https://access.redhat.com/solutions/103833]:
> {quote}Swappiness is a property for the Linux kernel that changes the balance
> between swapping out runtime memory, as opposed to dropping pages from the
> system page cache. Swappiness can be set to values between 0 and 100,
> inclusive. A low value means the kernel will try to avoid swapping as much as
> possible where a higher value instead will make the kernel aggressively try
> to use swap space.
> {quote}
> Referring [this|https://github.com/opencontainers/runtime-spec/issues/1005]
> case study we found that most of the time swappiness didn't work as expected
> as it mostly depends on the I/O balance of the system, so it is no longer
> available in cgroup v2.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842004#comment-17842004
]
ASF GitHub Bot commented on YARN-11675:
---
brumi1024 merged PR #6760:
URL: https://github.com/apache/hadoop/pull/6760
> Update MemoryResourceHandler implementation for cgroup v2 support
> -
>
> Key: YARN-11675
> URL: https://issues.apache.org/jira/browse/YARN-11675
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Peter Szucs
>Priority: Major
> Labels: pull-request-available
>
> cgroup v2 has some changes in various controllers (some changed their
> functionality, some were removed). This task is about updating
> MemoryResourceHandler's
> [implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46].
> h3. *Differences in the controls comparing to cgroup v1:*
> h3. Hard limit on memory
> {_}memory{_}.{_}limit_in_bytes{_} control is replaced with _memory.max_
> h3. Soft limit on memory
> {_}memory{_}.soft_{_}limit_in_bytes{_} control is replaced with _memory.low_
> Detailed descriptions about the memory controls can be found in the official
> [cgroup v2 documentation|https://docs.kernel.org/admin-guide/cgroup-v2.html].
> h3. Swappiness
> _memory.swappiness_ has been removed from the available cgroup v2 controls.
> Quoting [redhat documentation|https://access.redhat.com/solutions/103833]:
> {quote}Swappiness is a property for the Linux kernel that changes the balance
> between swapping out runtime memory, as opposed to dropping pages from the
> system page cache. Swappiness can be set to values between 0 and 100,
> inclusive. A low value means the kernel will try to avoid swapping as much as
> possible where a higher value instead will make the kernel aggressively try
> to use swap space.
> {quote}
> Referring [this|https://github.com/opencontainers/runtime-spec/issues/1005]
> case study we found that most of the time swappiness didn't work as expected
> as it mostly depends on the I/O balance of the system, so it is no longer
> available in cgroup v2.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11685) Create a config to enable/disable cgroup v2 functionality
[
https://issues.apache.org/jira/browse/YARN-11685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841949#comment-17841949
]
ASF GitHub Bot commented on YARN-11685:
---
hadoop-yetus commented on PR #6770:
URL: https://github.com/apache/hadoop/pull/6770#issuecomment-2082453879
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 32s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 15s | | trunk passed |
| +1 :green_heart: | compile | 7m 31s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 7m 1s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 2s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 58s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 2m 49s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 5m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 36s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 35s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 48s | | the patch passed |
| +1 :green_heart: | compile | 6m 52s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 6m 52s | | the patch passed |
| +1 :green_heart: | compile | 6m 58s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 6m 58s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6770/3/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 1m 54s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6770/3/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 11 new + 165 unchanged
- 0 fixed = 176 total (was 165) |
| +1 :green_heart: | mvnsite | 2m 43s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 37s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 2m 33s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 6m 1s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 33s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 12s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 34s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 25m 33s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 1m 0s | | The patch does not
generate ASF License warnings. |
| | | 218m 50s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6770/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6770 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux bea558bd5ec6 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/ha
[jira] [Commented] (YARN-11691) Add HTTP POST method to yarn web proxy servlet
[
https://issues.apache.org/jira/browse/YARN-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841948#comment-17841948
]
ASF GitHub Bot commented on YARN-11691:
---
hadoop-yetus commented on PR #6781:
URL: https://github.com/apache/hadoop/pull/6781#issuecomment-2082449270
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 32s | | trunk passed |
| +1 :green_heart: | compile | 0m 28s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 21s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 18s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 18s | | the patch passed |
| +1 :green_heart: | compile | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 47s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 9s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 13s | | hadoop-yarn-server-web-proxy in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 125m 55s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6781/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6781 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 72c25e3a8dd8 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4a1758bb786fc76c080e92e40d09ce91d0e53d41 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6781/1/testReport/ |
| Max. process+thread count | 726 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibr
[jira] [Commented] (YARN-11687) Update CGroupsResourceCalculator to track usages using cgroupv2
[
https://issues.apache.org/jira/browse/YARN-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841940#comment-17841940
]
ASF GitHub Bot commented on YARN-11687:
---
hadoop-yetus commented on PR #6780:
URL: https://github.com/apache/hadoop/pull/6780#issuecomment-2082387079
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 40s | | trunk passed |
| +1 :green_heart: | compile | 1m 33s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 31s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 28s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 33s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 49s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 173m 39s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6780/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6780 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d82c6e462a3c 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 507d3202014398eb25291af09c9b54828c15b87e |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.
[jira] [Commented] (YARN-11691) Add HTTP POST method to yarn web proxy servlet
[ https://issues.apache.org/jira/browse/YARN-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841903#comment-17841903 ] ASF GitHub Bot commented on YARN-11691: --- yeezychao opened a new pull request, #6781: URL: https://github.com/apache/hadoop/pull/6781 ### Description of PR The flink front-end tried to use the POST method to call the yarn server rest api, but it returned a 405 error. see [FLINK-35039,I think that flink should not be modified, but it is more reasonable to support the POST method in yarn web proxy. ](https://issues.apache.org/jira/browse/FLINK-35039) ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Add HTTP POST method to yarn web proxy servlet > -- > > Key: YARN-11691 > URL: https://issues.apache.org/jira/browse/YARN-11691 > Project: Hadoop YARN > Issue Type: Bug > Components: webproxy >Affects Versions: 3.2.2, 3.3.6 >Reporter: ude >Priority: Major > Fix For: 3.2.2, 3.3.6 > > > When the flink task is running in the YARN environment, the client encounters > an error HTTP ERROR 405 when calling the http proxy rest api by _POST_ Method. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11691) Add HTTP POST method to yarn web proxy servlet
[ https://issues.apache.org/jira/browse/YARN-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11691: -- Labels: pull-request-available (was: ) > Add HTTP POST method to yarn web proxy servlet > -- > > Key: YARN-11691 > URL: https://issues.apache.org/jira/browse/YARN-11691 > Project: Hadoop YARN > Issue Type: Bug > Components: webproxy >Affects Versions: 3.2.2, 3.3.6 >Reporter: ude >Priority: Major > Labels: pull-request-available > Fix For: 3.2.2, 3.3.6 > > > When the flink task is running in the YARN environment, the client encounters > an error HTTP ERROR 405 when calling the http proxy rest api by _POST_ Method. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11687) Update CGroupsResourceCalculator to track usages using cgroupv2
[ https://issues.apache.org/jira/browse/YARN-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11687: -- Labels: pull-request-available (was: ) > Update CGroupsResourceCalculator to track usages using cgroupv2 > --- > > Key: YARN-11687 > URL: https://issues.apache.org/jira/browse/YARN-11687 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Bence Kosztolnik >Priority: Major > Labels: pull-request-available > > [CGroupsResourceCalculator|https://github.com/apache/hadoop/blob/f609460bda0c2bd87dd3580158e549e2f34f14d5/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsResourceCalculator.java] > should also be updated to handle the cgroup v2 changes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11687) Update CGroupsResourceCalculator to track usages using cgroupv2
[ https://issues.apache.org/jira/browse/YARN-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841881#comment-17841881 ] ASF GitHub Bot commented on YARN-11687: --- K0K0V0K opened a new pull request, #6780: URL: https://github.com/apache/hadoop/pull/6780 ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Update CGroupsResourceCalculator to track usages using cgroupv2 > --- > > Key: YARN-11687 > URL: https://issues.apache.org/jira/browse/YARN-11687 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Bence Kosztolnik >Priority: Major > > [CGroupsResourceCalculator|https://github.com/apache/hadoop/blob/f609460bda0c2bd87dd3580158e549e2f34f14d5/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsResourceCalculator.java] > should also be updated to handle the cgroup v2 changes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11686) Correct traversing indexs when scheduling asynchronously using Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-11686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841845#comment-17841845
]
ASF GitHub Bot commented on YARN-11686:
---
hadoop-yetus commented on PR #6750:
URL: https://github.com/apache/hadoop/pull/6750#issuecomment-2081901457
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 01s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 01s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 91m 20s | | trunk passed |
| +1 :green_heart: | compile | 6m 22s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 47s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 04s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 154m 19s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 39s | | the patch passed |
| +1 :green_heart: | compile | 3m 13s | | the patch passed |
| +1 :green_heart: | javac | 3m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 21s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 00s | | the patch passed |
| +1 :green_heart: | shadedclient | 159m 43s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 22s | | The patch does not
generate ASF License warnings. |
| | | 431m 33s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6750 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 7a39ef331537 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 0e1ae3a46317d9941462ffc5f2aaf3271d0b3390 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6750/2/testReport/
|
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6750/2/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Correct traversing indexs when scheduling asynchronously using Capacity
> Scheduler
> -
>
> Key: YARN-11686
> URL: https://issues.apache.org/jira/browse/YARN-11686
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Yihe Li
>Priority: Minor
> Labels: pull-request-available
>
> When scheduling asynchronously using Capacity Scheduler, the traversing
> indexs in `CapacityScheduler#schedule` will always contains `start` index
> twice. This may not in line with the original intention and needs to be
> corrected.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841783#comment-17841783
]
ASF GitHub Bot commented on YARN-11675:
---
hadoop-yetus commented on PR #6760:
URL: https://github.com/apache/hadoop/pull/6760#issuecomment-2081688017
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 86m 16s | | trunk passed |
| +1 :green_heart: | compile | 5m 06s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 142m 09s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 44s | | the patch passed |
| +1 :green_heart: | compile | 2m 27s | | the patch passed |
| +1 :green_heart: | javac | 2m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 04s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 145m 04s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 09s | | The patch does not
generate ASF License warnings. |
| | | 396m 33s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6760 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 b817df791504 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 18421d720446fa982e620bd77df0b4f3420d533b |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6760/9/testReport/
|
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6760/9/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Update MemoryResourceHandler implementation for cgroup v2 support
> -
>
> Key: YARN-11675
> URL: https://issues.apache.org/jira/browse/YARN-11675
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Peter Szucs
>Priority: Major
> Labels: pull-request-available
>
> cgroup v2 has some changes in various controllers (some changed their
> functionality, some were removed). This task is about updating
> MemoryResourceHandler's
> [implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46].
> h3. *Differences in the controls comparing to cgroup v1:*
> h3. Hard limit on memory
> {_}memory{_}.{_}limit_in_bytes{_} control is replaced with _memory.max_
> h3. Soft limit on memory
> {_}memory{_}.soft_{_}limit_in_bytes{_} control is replaced with _memory.low_
> Detailed descriptions about the memory controls can be found in the official
> [cgroup v2 documentation|https://docs.kernel.org/admin-guide/cgroup-v2.html
[jira] [Commented] (YARN-11685) Create a config to enable/disable cgroup v2 functionality
[ https://issues.apache.org/jira/browse/YARN-11685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841599#comment-17841599 ] ASF GitHub Bot commented on YARN-11685: --- hadoop-yetus commented on PR #6770: URL: https://github.com/apache/hadoop/pull/6770#issuecomment-2081384515 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | -1 :x: | patch | 0m 44s | | https://github.com/apache/hadoop/pull/6770 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/6770 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6770/3/console | | versions | git=2.44.0.windows.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. > Create a config to enable/disable cgroup v2 functionality > - > > Key: YARN-11685 > URL: https://issues.apache.org/jira/browse/YARN-11685 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Peter Szucs >Priority: Major > Labels: pull-request-available > > Various OS's mount the cgroup v2 differently, some of them mount both the v1 > and v2 structure, others mount a hybrid structure. To avoid initialization > issues the cgroup v1/v2 functionality should be set by a config property. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841592#comment-17841592
]
ASF GitHub Bot commented on YARN-11191:
---
hadoop-yetus commented on PR #6768:
URL: https://github.com/apache/hadoop/pull/6768#issuecomment-2081364250
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.4 Compile Tests _ |
| +1 :green_heart: | mvninstall | 88m 19s | | branch-3.4 passed |
| +1 :green_heart: | compile | 5m 43s | | branch-3.4 passed |
| +1 :green_heart: | checkstyle | 4m 35s | | branch-3.4 passed |
| +1 :green_heart: | mvnsite | 5m 36s | | branch-3.4 passed |
| +1 :green_heart: | javadoc | 5m 31s | | branch-3.4 passed |
| +1 :green_heart: | shadedclient | 148m 13s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 26s | | the patch passed |
| +1 :green_heart: | compile | 3m 06s | | the patch passed |
| +1 :green_heart: | javac | 3m 06s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 11s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 55s | | the patch passed |
| +1 :green_heart: | shadedclient | 155m 05s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 323m 16s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6768/3/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| -1 :x: | asflicense | 5m 28s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6768/3/artifact/out/results-asflicense.txt)
| The patch generated 2 ASF License warnings. |
| | | 747m 18s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher |
| | hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore |
| | hadoop.yarn.server.resourcemanager.resource.TestResourceProfiles |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.TestFSSchedulerConfigurationStore
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.TestLeveldbConfigurationStore
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCSAllocateCustomResource
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueConfigurationAutoRefreshPolicy
|
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.converter.TestFSConfigToCSConfigArgumentHandler
|
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.converter.TestFSConfigToCSConfigConverter
|
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.converter.TestFSConfigToCSConfigConverterMain
|
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.converter.TestFSQueueConverter
|
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService
|
| | hadoop.yarn.server.resourcemanager.TestClientRMService |
| | hadoop.yarn.server.resourcemanager.TestSignalContainer |
| | hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
| | hadoop.yarn.server.resourcemanager.volume.csi.TestVolumeProcessor |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication |
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppCustomResourceTypes
|
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsCustomResourceTypes
|
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesContainers |
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication
|
| |
hadoop.yarn.
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841549#comment-17841549
]
ASF GitHub Bot commented on YARN-11191:
---
slfan1989 commented on PR #6768:
URL: https://github.com/apache/hadoop/pull/6768#issuecomment-2081257034
@tomicooler Thanks for the contribution!
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, Lock holding status.png, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841548#comment-17841548
]
ASF GitHub Bot commented on YARN-11191:
---
slfan1989 merged PR #6768:
URL: https://github.com/apache/hadoop/pull/6768
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, Lock holding status.png, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841541#comment-17841541
]
ASF GitHub Bot commented on YARN-11191:
---
hadoop-yetus commented on PR #6768:
URL: https://github.com/apache/hadoop/pull/6768#issuecomment-2081187638
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 6m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.4 Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 44s | | branch-3.4 passed |
| +1 :green_heart: | compile | 0m 36s | | branch-3.4 passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 30s | | branch-3.4 passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 31s | | branch-3.4 passed |
| +1 :green_heart: | mvnsite | 0m 34s | | branch-3.4 passed |
| +1 :green_heart: | javadoc | 0m 37s | | branch-3.4 passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 29s | | branch-3.4 passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 16s | | branch-3.4 passed |
| +1 :green_heart: | shadedclient | 20m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 24s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 28s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 87m 33s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 26s | | The patch does not
generate ASF License warnings. |
| | | 179m 4s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6768/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6768 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 10200f517789 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.4 / 31f1424fcd7011fdd78d507e9482cffee9bcd09b |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6768/3/testReport/ |
| Max. process+thread count | 950 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6768/3/console |
| versions | git=2.25.1 ma
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841525#comment-17841525
]
ASF GitHub Bot commented on YARN-11191:
---
tomicooler commented on PR #6768:
URL: https://github.com/apache/hadoop/pull/6768#issuecomment-2081123115
@slfan1989 Yes, sure. Re-triggered.
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, Lock holding status.png, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841465#comment-17841465
]
ASF GitHub Bot commented on YARN-11191:
---
slfan1989 commented on PR #6768:
URL: https://github.com/apache/hadoop/pull/6768#issuecomment-2080551000
@tomicooler Can we retrigger compilation?
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, Lock holding status.png, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841464#comment-17841464
]
ASF GitHub Bot commented on YARN-11191:
---
slfan1989 commented on PR #6769:
URL: https://github.com/apache/hadoop/pull/6769#issuecomment-2080548182
@tomicooler Thanks for the contribution!
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, Lock holding status.png, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841463#comment-17841463
]
ASF GitHub Bot commented on YARN-11191:
---
slfan1989 merged PR #6769:
URL: https://github.com/apache/hadoop/pull/6769
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, Lock holding status.png, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841296#comment-17841296
]
ASF GitHub Bot commented on YARN-11675:
---
hadoop-yetus commented on PR #6760:
URL: https://github.com/apache/hadoop/pull/6760#issuecomment-2079729101
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 29s | | trunk passed |
| +1 :green_heart: | compile | 0m 48s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 53s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 21s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 0m 44s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 44s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 21s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 53s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 0s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 21m 40s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 24s | | The patch does not
generate ASF License warnings. |
| | | 105m 56s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6760/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6760 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 44c51a75455e 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 18421d720446fa982e620bd77df0b4f3420d533b |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6760/8/testReport/ |
| Max. process+thread count | 551 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6760/8/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14
[jira] [Commented] (YARN-11685) Create a config to enable/disable cgroup v2 functionality
[
https://issues.apache.org/jira/browse/YARN-11685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841286#comment-17841286
]
ASF GitHub Bot commented on YARN-11685:
---
hadoop-yetus commented on PR #6770:
URL: https://github.com/apache/hadoop/pull/6770#issuecomment-2079693838
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 27s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 2s | | trunk passed |
| +1 :green_heart: | compile | 7m 39s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 7m 4s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 56s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 3s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 0s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 2m 51s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 5m 46s | | trunk passed |
| +1 :green_heart: | shadedclient | 35m 35s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 54s | | the patch passed |
| +1 :green_heart: | compile | 7m 55s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 7m 55s | | the patch passed |
| +1 :green_heart: | compile | 7m 15s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 7m 15s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6770/2/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 1m 55s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6770/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 19 new + 174 unchanged
- 0 fixed = 193 total (was 174) |
| +1 :green_heart: | mvnsite | 2m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 37s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 2m 31s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 5m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 14s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 55s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 24m 58s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 58s | | The patch does not
generate ASF License warnings. |
| | | 232m 20s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6770/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6770 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 1251bfb621b7 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/ha
[jira] [Commented] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841274#comment-17841274
]
ASF GitHub Bot commented on YARN-11675:
---
p-szucs commented on PR #6760:
URL: https://github.com/apache/hadoop/pull/6760#issuecomment-2079619303
Thanks @brumi1024 for the review! Rebased and updated the PR with the fix
for your comment.
Added a description on the jira ticket as well summarizing the changes
comparing to cgroupv1: https://issues.apache.org/jira/browse/YARN-11675
> Update MemoryResourceHandler implementation for cgroup v2 support
> -
>
> Key: YARN-11675
> URL: https://issues.apache.org/jira/browse/YARN-11675
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Peter Szucs
>Priority: Major
> Labels: pull-request-available
>
> cgroup v2 has some changes in various controllers (some changed their
> functionality, some were removed). This task is about updating
> MemoryResourceHandler's
> [implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java#L47-L46].
> h3. *Differences in the controls comparing to cgroup v1:*
> h3. Hard limit on memory
> {_}memory{_}.{_}limit_in_bytes{_} control is replaced with _memory.max_
> h3. Soft limit on memory
> {_}memory{_}.soft_{_}limit_in_bytes{_} control is replaced with _memory.low_
> Detailed descriptions about the memory controls can be found in the official
> [cgroup v2 documentation|https://docs.kernel.org/admin-guide/cgroup-v2.html].
> h3. Swappiness
> _memory.swappiness_ has been removed from the available cgroup v2 controls.
> Quoting [redhat documentation|https://access.redhat.com/solutions/103833]:
> {quote}Swappiness is a property for the Linux kernel that changes the balance
> between swapping out runtime memory, as opposed to dropping pages from the
> system page cache. Swappiness can be set to values between 0 and 100,
> inclusive. A low value means the kernel will try to avoid swapping as much as
> possible where a higher value instead will make the kernel aggressively try
> to use swap space.
> {quote}
> Referring [this|https://github.com/opencontainers/runtime-spec/issues/1005]
> case study we found that most of the time swappiness didn't work as expected
> as it mostly depends on the I/O balance of the system, so it is no longer
> available in cgroup v2.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841260#comment-17841260
]
ASF GitHub Bot commented on YARN-11675:
---
hadoop-yetus commented on PR #6760:
URL: https://github.com/apache/hadoop/pull/6760#issuecomment-2079518586
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 6m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 37s | | trunk passed |
| +1 :green_heart: | compile | 0m 47s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 50s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 52s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 5s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 13s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 55s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 58s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 21m 39s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 24s | | The patch does not
generate ASF License warnings. |
| | | 113m 43s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6760/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6760 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux dccf6577322e 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b476d2d6a376f80b2ceeb111a62c66e84d5319fd |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6760/6/testReport/ |
| Max. process+thread count | 560 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6760/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14
[jira] [Commented] (YARN-11675) Update MemoryResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841257#comment-17841257
]
ASF GitHub Bot commented on YARN-11675:
---
brumi1024 commented on code in PR #6760:
URL: https://github.com/apache/hadoop/pull/6760#discussion_r1581108211
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsMemoryResourceHandlerImpl.java:
##
@@ -99,81 +66,31 @@ int getSwappiness() {
}
@Override
- public List reacquireContainer(ContainerId containerId)
- throws ResourceHandlerException {
-return null;
- }
-
- @Override
- public List updateContainer(Container container)
+ protected void updateMemoryHardLimit(String cgroupId, long
containerHardLimit)
throws ResourceHandlerException {
-String cgroupId = container.getContainerId().toString();
-File cgroup = new File(cGroupsHandler.getPathForCGroup(MEMORY, cgroupId));
-if (cgroup.exists()) {
- //memory is in MB
- long containerSoftLimit =
- (long) (container.getResource().getMemorySize() * this.softLimit);
- long containerHardLimit = container.getResource().getMemorySize();
- if (enforce) {
-try {
- cGroupsHandler.updateCGroupParam(MEMORY, cgroupId,
- CGroupsHandler.CGROUP_PARAM_MEMORY_HARD_LIMIT_BYTES,
- String.valueOf(containerHardLimit) + "M");
- ContainerTokenIdentifier id =
container.getContainerTokenIdentifier();
- if (id != null && id.getExecutionType() ==
- ExecutionType.OPPORTUNISTIC) {
-cGroupsHandler.updateCGroupParam(MEMORY, cgroupId,
-CGroupsHandler.CGROUP_PARAM_MEMORY_SOFT_LIMIT_BYTES,
-String.valueOf(OPPORTUNISTIC_SOFT_LIMIT) + "M");
-cGroupsHandler.updateCGroupParam(MEMORY, cgroupId,
-CGroupsHandler.CGROUP_PARAM_MEMORY_SWAPPINESS,
-String.valueOf(OPPORTUNISTIC_SWAPPINESS));
- } else {
-cGroupsHandler.updateCGroupParam(MEMORY, cgroupId,
-CGroupsHandler.CGROUP_PARAM_MEMORY_SOFT_LIMIT_BYTES,
-String.valueOf(containerSoftLimit) + "M");
-cGroupsHandler.updateCGroupParam(MEMORY, cgroupId,
-CGroupsHandler.CGROUP_PARAM_MEMORY_SWAPPINESS,
-String.valueOf(swappiness));
- }
-} catch (ResourceHandlerException re) {
- cGroupsHandler.deleteCGroup(MEMORY, cgroupId);
- LOG.warn("Could not update cgroup for container", re);
- throw re;
-}
- }
-}
-return null;
+getCGroupsHandler().updateCGroupParam(MEMORY, cgroupId,
+CGroupsHandler.CGROUP_PARAM_MEMORY_HARD_LIMIT_BYTES,
+String.valueOf(containerHardLimit) + "M");
}
@Override
- public List preStart(Container container)
- throws ResourceHandlerException {
-String cgroupId = container.getContainerId().toString();
-cGroupsHandler.createCGroup(MEMORY, cgroupId);
-updateContainer(container);
-List ret = new ArrayList<>();
-ret.add(new PrivilegedOperation(
-PrivilegedOperation.OperationType.ADD_PID_TO_CGROUP,
-PrivilegedOperation.CGROUP_ARG_PREFIX
-+ cGroupsHandler.getPathForCGroupTasks(MEMORY, cgroupId)));
-return ret;
+ protected void updateOpportunisticMemoryLimits(String cgroupId) throws
ResourceHandlerException {
+getCGroupsHandler().updateCGroupParam(MEMORY, cgroupId,
+CGroupsHandler.CGROUP_PARAM_MEMORY_SOFT_LIMIT_BYTES,
+String.valueOf(OPPORTUNISTIC_SOFT_LIMIT) + "M");
+getCGroupsHandler().updateCGroupParam(MEMORY, cgroupId,
Review Comment:
Can you please update the description of the jira and maybe explain in short
a comment what the [swappiness](https://access.redhat.com/solutions/103833)
controls, and that it's no longer available in cgroup v2 as most of the times
it didn't behave as expected?
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsV2MemoryResourceHandlerImpl.java:
##
@@ -0,0 +1,52 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "A
[jira] [Commented] (YARN-11643) Skip unnecessary pre-check in Multi Node Placement
[
https://issues.apache.org/jira/browse/YARN-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841253#comment-17841253
]
ASF GitHub Bot commented on YARN-11643:
---
hadoop-yetus commented on PR #6426:
URL: https://github.com/apache/hadoop/pull/6426#issuecomment-2079445379
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 01s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 108m 43s | | trunk passed |
| +1 :green_heart: | compile | 7m 28s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 38s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 06s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 179m 47s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 28s | | the patch passed |
| +1 :green_heart: | compile | 3m 52s | | the patch passed |
| +1 :green_heart: | javac | 3m 52s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 43s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 04s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 38s | | the patch passed |
| +1 :green_heart: | shadedclient | 190m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 37s | | The patch does not
generate ASF License warnings. |
| | | 510m 34s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6426 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 80fcb844f29c 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 4c340e66690d361073ab7a746dbe4a9e7c446f2e |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6426/1/testReport/
|
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6426/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Skip unnecessary pre-check in Multi Node Placement
> --
>
> Key: YARN-11643
> URL: https://issues.apache.org/jira/browse/YARN-11643
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Reporter: Xie YiFan
>Assignee: Xie YiFan
>Priority: Minor
> Labels: pull-request-available
>
> When Multi Node Placement enabled, RegularContainerAllocator do a while loop
> to find one node from candidate set to allocate for a given scheduler key.
> Before do allocate, pre-check be called to check if current node satisfies
> check. If this node does not pass all checks, just continue to next node.
> {code:java}
> if (reservedContainer == null) {
> result = preCheckForNodeCandidateSet(node,
> schedulingMode, resourceLimits, schedulerKey);
> if (null != result) {
> continue;
> }
> } {code}
> But some checks are related to scheduler Key or Application which return
> PRIORITY_SKIPPED or APP_SKIPPED. It means that if first node does not pass
> check, the following nodes also do not pass.
> If cluster have 5000 nodes in default
[jira] [Commented] (YARN-11674) Update CpuResourceHandler implementation for cgroup v2 support
[ https://issues.apache.org/jira/browse/YARN-11674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841207#comment-17841207 ] ASF GitHub Bot commented on YARN-11674: --- brumi1024 commented on PR #6751: URL: https://github.com/apache/hadoop/pull/6751#issuecomment-2079350982 Thanks @tomicooler for the review, merged to trunk. > Update CpuResourceHandler implementation for cgroup v2 support > -- > > Key: YARN-11674 > URL: https://issues.apache.org/jira/browse/YARN-11674 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Labels: pull-request-available > > cgroup v2 has some changes in various controllers (some changed their > functionality, some were removed). This task is about checking if > CpuResourceHandler's > [implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsCpuResourceHandlerImpl.java#L60] > need any updates. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11674) Update CpuResourceHandler implementation for cgroup v2 support
[ https://issues.apache.org/jira/browse/YARN-11674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841206#comment-17841206 ] ASF GitHub Bot commented on YARN-11674: --- brumi1024 merged PR #6751: URL: https://github.com/apache/hadoop/pull/6751 > Update CpuResourceHandler implementation for cgroup v2 support > -- > > Key: YARN-11674 > URL: https://issues.apache.org/jira/browse/YARN-11674 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Labels: pull-request-available > > cgroup v2 has some changes in various controllers (some changed their > functionality, some were removed). This task is about checking if > CpuResourceHandler's > [implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsCpuResourceHandlerImpl.java#L60] > need any updates. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841168#comment-17841168
]
ASF GitHub Bot commented on YARN-11690:
---
brumi1024 merged PR #6771:
URL: https://github.com/apache/hadoop/pull/6771
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file, pid_t pid) {
> int rc = 0;
> uid_t user = geteuid();
> gid_t group = getegid();
> if (change_effective_user(0, 0) != 0) {
> rc = -1;
> goto cleanup;
> }
> // statfs
> struct statfs buf;
> if (statfs(cgroup_file, &buf) == -1) {
> fprintf(LOGFILE, "Can't statfs file %s as node manager - %s\n",
> cgroup_file,
>strerror(errno));
> rc = -1;
> goto cleanup;
> } else if (buf.f_type != CGROUP_SUPER_MAGIC) {
> fprintf(LOGFILE, "Pid file %s is not located on cgroup filesystem\n",
> cgroup_file);
> rc = -1;
> goto cleanup;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841167#comment-17841167
]
ASF GitHub Bot commented on YARN-11690:
---
brumi1024 commented on PR #6771:
URL: https://github.com/apache/hadoop/pull/6771#issuecomment-2079194163
Thanks @tomicooler for the review, merging to trunk.
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file, pid_t pid) {
> int rc = 0;
> uid_t user = geteuid();
> gid_t group = getegid();
> if (change_effective_user(0, 0) != 0) {
> rc = -1;
> goto cleanup;
> }
> // statfs
> struct statfs buf;
> if (statfs(cgroup_file, &buf) == -1) {
> fprintf(LOGFILE, "Can't statfs file %s as node manager - %s\n",
> cgroup_file,
>strerror(errno));
> rc = -1;
> goto cleanup;
> } else if (buf.f_type != CGROUP_SUPER_MAGIC) {
> fprintf(LOGFILE, "Pid file %s is not located on cgroup filesystem\n",
> cgroup_file);
> rc = -1;
> goto cleanup;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841158#comment-17841158
]
ASF GitHub Bot commented on YARN-11690:
---
hadoop-yetus commented on PR #6771:
URL: https://github.com/apache/hadoop/pull/6771#issuecomment-2079174965
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 50m 18s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 91m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed |
| +1 :green_heart: | cc | 1m 14s | | the patch passed |
| +1 :green_heart: | golang | 1m 14s | | the patch passed |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 46s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 162m 52s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6771 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets golang |
| uname | Linux 26c72a68eecc 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ed587f807d2eb5e93978a0faf1c77b4e506c9f21 |
| Default Java | Red Hat, Inc.-1.8.0_402-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/4/testReport/ |
| Max. process+thread count | 534 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6771/4/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file, pid_t pid) {
> int rc = 0;
> uid_t user = geteuid();
> g
[jira] [Commented] (YARN-11674) Update CpuResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841110#comment-17841110
]
ASF GitHub Bot commented on YARN-11674:
---
tomicooler commented on code in PR #6751:
URL: https://github.com/apache/hadoop/pull/6751#discussion_r1580710076
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/AbstractCGroupsCpuResourceHandler.java:
##
@@ -0,0 +1,219 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.records.ContainerId;
+import org.apache.hadoop.yarn.api.records.ExecutionType;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.security.ContainerTokenIdentifier;
+import
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
+import
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperation;
+import org.apache.hadoop.yarn.server.nodemanager.util.NodeManagerHardwareUtils;
+import org.apache.hadoop.yarn.util.ResourceCalculatorPlugin;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+
+@InterfaceStability.Unstable
+@InterfaceAudience.Private
+public abstract class AbstractCGroupsCpuResourceHandler implements
CpuResourceHandler {
+
+ static final Logger LOG =
+ LoggerFactory.getLogger(AbstractCGroupsCpuResourceHandler.class);
+
+ protected CGroupsHandler cGroupsHandler;
+ private boolean strictResourceUsageMode = false;
+ private float yarnProcessors;
+ private int nodeVCores;
+ private static final CGroupsHandler.CGroupController CPU =
+ CGroupsHandler.CGroupController.CPU;
+
+ @VisibleForTesting
+ static final int MAX_QUOTA_US = 1000 * 1000;
+ @VisibleForTesting
+ static final int MIN_PERIOD_US = 1000;
+
+ AbstractCGroupsCpuResourceHandler(CGroupsHandler cGroupsHandler) {
+this.cGroupsHandler = cGroupsHandler;
+ }
+
+ @Override
+ public List bootstrap(Configuration conf)
+ throws ResourceHandlerException {
+return bootstrap(
+ResourceCalculatorPlugin.getResourceCalculatorPlugin(null, conf),
conf);
+ }
+
+ @VisibleForTesting
+ List bootstrap(
+ ResourceCalculatorPlugin plugin, Configuration conf)
+ throws ResourceHandlerException {
+this.strictResourceUsageMode = conf.getBoolean(
+YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_STRICT_RESOURCE_USAGE,
+
YarnConfiguration.DEFAULT_NM_LINUX_CONTAINER_CGROUPS_STRICT_RESOURCE_USAGE);
+this.cGroupsHandler.initializeCGroupController(CPU);
+nodeVCores = NodeManagerHardwareUtils.getVCores(plugin, conf);
+
+// cap overall usage to the number of cores allocated to YARN
+yarnProcessors = NodeManagerHardwareUtils.getContainersCPUs(plugin, conf);
+int systemProcessors = NodeManagerHardwareUtils.getNodeCPUs(plugin, conf);
+boolean existingCpuLimits;
+existingCpuLimits = cpuLimitExists(
+cGroupsHandler.getPathForCGroup(CPU, ""));
+
+if (systemProcessors != (int) yarnProcessors) {
+ LOG.info("YARN containers restricted to " + yarnProcessors + " cores");
+ int[] limits = getOverallLimits(yarnProcessors);
+ updateCgroupMaxCpuLimit("", String.valueOf(limits[1]),
String.valueOf(limits[0]));
+} else if (existingCpuLimits) {
+ LOG.info("Removing CPU constraints for YARN containers.");
+ updateCgroupMaxCpuLimit("", String.valueOf(-1), null);
+}
+return null;
+ }
+
+ protected abstract void updateCgroupMaxCpuLimit(String cgroupId, String
quota, String period)
+ throws ResourceHandlerException;
+ protected abstract boolean c
[jira] [Commented] (YARN-11674) Update CpuResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841104#comment-17841104
]
ASF GitHub Bot commented on YARN-11674:
---
brumi1024 commented on code in PR #6751:
URL: https://github.com/apache/hadoop/pull/6751#discussion_r1580689988
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/AbstractCGroupsCpuResourceHandler.java:
##
@@ -0,0 +1,219 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.api.records.ContainerId;
+import org.apache.hadoop.yarn.api.records.ExecutionType;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.security.ContainerTokenIdentifier;
+import
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
+import
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperation;
+import org.apache.hadoop.yarn.server.nodemanager.util.NodeManagerHardwareUtils;
+import org.apache.hadoop.yarn.util.ResourceCalculatorPlugin;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+
+@InterfaceStability.Unstable
+@InterfaceAudience.Private
+public abstract class AbstractCGroupsCpuResourceHandler implements
CpuResourceHandler {
+
+ static final Logger LOG =
+ LoggerFactory.getLogger(AbstractCGroupsCpuResourceHandler.class);
+
+ protected CGroupsHandler cGroupsHandler;
+ private boolean strictResourceUsageMode = false;
+ private float yarnProcessors;
+ private int nodeVCores;
+ private static final CGroupsHandler.CGroupController CPU =
+ CGroupsHandler.CGroupController.CPU;
+
+ @VisibleForTesting
+ static final int MAX_QUOTA_US = 1000 * 1000;
+ @VisibleForTesting
+ static final int MIN_PERIOD_US = 1000;
+
+ AbstractCGroupsCpuResourceHandler(CGroupsHandler cGroupsHandler) {
+this.cGroupsHandler = cGroupsHandler;
+ }
+
+ @Override
+ public List bootstrap(Configuration conf)
+ throws ResourceHandlerException {
+return bootstrap(
+ResourceCalculatorPlugin.getResourceCalculatorPlugin(null, conf),
conf);
+ }
+
+ @VisibleForTesting
+ List bootstrap(
+ ResourceCalculatorPlugin plugin, Configuration conf)
+ throws ResourceHandlerException {
+this.strictResourceUsageMode = conf.getBoolean(
+YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_STRICT_RESOURCE_USAGE,
+
YarnConfiguration.DEFAULT_NM_LINUX_CONTAINER_CGROUPS_STRICT_RESOURCE_USAGE);
+this.cGroupsHandler.initializeCGroupController(CPU);
+nodeVCores = NodeManagerHardwareUtils.getVCores(plugin, conf);
+
+// cap overall usage to the number of cores allocated to YARN
+yarnProcessors = NodeManagerHardwareUtils.getContainersCPUs(plugin, conf);
+int systemProcessors = NodeManagerHardwareUtils.getNodeCPUs(plugin, conf);
+boolean existingCpuLimits;
+existingCpuLimits = cpuLimitExists(
+cGroupsHandler.getPathForCGroup(CPU, ""));
+
+if (systemProcessors != (int) yarnProcessors) {
+ LOG.info("YARN containers restricted to " + yarnProcessors + " cores");
+ int[] limits = getOverallLimits(yarnProcessors);
+ updateCgroupMaxCpuLimit("", String.valueOf(limits[1]),
String.valueOf(limits[0]));
+} else if (existingCpuLimits) {
+ LOG.info("Removing CPU constraints for YARN containers.");
+ updateCgroupMaxCpuLimit("", String.valueOf(-1), null);
+}
+return null;
+ }
+
+ protected abstract void updateCgroupMaxCpuLimit(String cgroupId, String
quota, String period)
+ throws ResourceHandlerException;
+ protected abstract boolean cp
[jira] [Commented] (YARN-11674) Update CpuResourceHandler implementation for cgroup v2 support
[
https://issues.apache.org/jira/browse/YARN-11674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841102#comment-17841102
]
ASF GitHub Bot commented on YARN-11674:
---
brumi1024 commented on code in PR #6751:
URL: https://github.com/apache/hadoop/pull/6751#discussion_r1580684787
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsV2CpuResourceHandlerImpl.java:
##
@@ -0,0 +1,99 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.classification.VisibleForTesting;
+
+/**
+ * An implementation for using CGroups V2 to restrict CPU usage on Linux. The
+ * implementation supports 3 different controls - restrict usage of all YARN
+ * containers, restrict relative usage of individual YARN containers and
+ * restrict usage of individual YARN containers. Admins can set the overall CPU
+ * to be used by all YARN containers - this is implemented by setting
+ * cpu.max to the value desired. If strict resource usage mode is not enabled,
+ * cpu.weight is set for individual containers - this prevents containers from
+ * exceeding the overall limit for YARN containers but individual containers
+ * can use as much of the CPU as available(under the YARN limit). If strict
+ * resource usage is enabled, then container can only use the percentage of
+ * CPU allocated to them and this is again implemented using cpu.max.
+ */
+@InterfaceStability.Unstable
+@InterfaceAudience.Private
+public class CGroupsV2CpuResourceHandlerImpl extends
AbstractCGroupsCpuResourceHandler {
+ private static final CGroupsHandler.CGroupController CPU =
+ CGroupsHandler.CGroupController.CPU;
+
+ @VisibleForTesting
+ static final int CPU_DEFAULT_WEIGHT = 100; // cgroup v2 default
+ static final int CPU_DEFAULT_WEIGHT_OPPORTUNISTIC = 1;
+ static final int CPU_MAX_WEIGHT = 1;
+ static final String NO_LIMIT = "max";
+
+
+ CGroupsV2CpuResourceHandlerImpl(CGroupsHandler cGroupsHandler) {
+super(cGroupsHandler);
+ }
+
+ @Override
+ protected void updateCgroupMaxCpuLimit(String cgroupId, String max, String
period)
+ throws ResourceHandlerException {
+String cpuMaxLimit = cGroupsHandler.getCGroupParam(CPU, cgroupId,
Review Comment:
Done.
> Update CpuResourceHandler implementation for cgroup v2 support
> --
>
> Key: YARN-11674
> URL: https://issues.apache.org/jira/browse/YARN-11674
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> cgroup v2 has some changes in various controllers (some changed their
> functionality, some were removed). This task is about checking if
> CpuResourceHandler's
> [implementation|https://github.com/apache/hadoop/blob/d336227e5c63a70db06ac26697994c96ed89d230/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsCpuResourceHandlerImpl.java#L60]
> need any updates.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11323) [Federation] Improve Router Handler FinishApps
[ https://issues.apache.org/jira/browse/YARN-11323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841101#comment-17841101 ] ASF GitHub Bot commented on YARN-11323: --- liubin101 commented on PR #4954: URL: https://github.com/apache/hadoop/pull/4954#issuecomment-2078901927 @slfan1989 Hi, I found that newly submitted apps may be mistakenly deleted in Delete triggered condition 1: If an app was submitted to Router, the Router stores this app in state store, but it hasn't been submitted to SubCluster. At this time, the CleanUp thread compares the lists queried in the state store and the SubCluster. This app exists in the state store but not in the SubCluster, so it will be mistakenly deleted. > [Federation] Improve Router Handler FinishApps > -- > > Key: YARN-11323 > URL: https://issues.apache.org/jira/browse/YARN-11323 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation, router, yarn >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841100#comment-17841100
]
ASF GitHub Bot commented on YARN-11690:
---
brumi1024 commented on code in PR #6771:
URL: https://github.com/apache/hadoop/pull/6771#discussion_r1580673804
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h:
##
@@ -80,6 +80,11 @@ enum operations {
#define ROOT_VAR_TMP_DIR "private_var_slash_tmp"
#define COMMAND_FILE_SECTION "command-execution"
+// include cgroup2 super magic number if not defined
+#ifndef CGROUP2_SUPER_MAGIC
Review Comment:
Done.
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file, pid_t pid) {
> int rc = 0;
> uid_t user = geteuid();
> gid_t group = getegid();
> if (change_effective_user(0, 0) != 0) {
> rc = -1;
> goto cleanup;
> }
> // statfs
> struct statfs buf;
> if (statfs(cgroup_file, &buf) == -1) {
> fprintf(LOGFILE, "Can't statfs file %s as node manager - %s\n",
> cgroup_file,
>strerror(errno));
> rc = -1;
> goto cleanup;
> } else if (buf.f_type != CGROUP_SUPER_MAGIC) {
> fprintf(LOGFILE, "Pid file %s is not located on cgroup filesystem\n",
> cgroup_file);
> rc = -1;
> goto cleanup;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11644) LogAggregationService can't upload log in time when application finished
[
https://issues.apache.org/jira/browse/YARN-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841055#comment-17841055
]
ASF GitHub Bot commented on YARN-11644:
---
hadoop-yetus commented on PR #6452:
URL: https://github.com/apache/hadoop/pull/6452#issuecomment-2078653415
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 91m 32s | | trunk passed |
| +1 :green_heart: | compile | 5m 11s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 50s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 149m 06s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 01s | | the patch passed |
| +1 :green_heart: | compile | 2m 39s | | the patch passed |
| +1 :green_heart: | javac | 2m 39s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 01s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 162m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 35s | | The patch does not
generate ASF License warnings. |
| | | 426m 59s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6452 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 8b443c886006 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 19826d01ffafb3bd2d371ef27f8efe86739f666d |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6452/1/testReport/
|
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6452/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> LogAggregationService can't upload log in time when application finished
>
>
> Key: YARN-11644
> URL: https://issues.apache.org/jira/browse/YARN-11644
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: log-aggregation
>Affects Versions: 3.3.6
>Reporter: Xie YiFan
>Assignee: Xie YiFan
>Priority: Minor
> Labels: pull-request-available
> Attachments: image-2024-01-10-11-03-57-553.png
>
>
> LogAggregationService is responsible for uploading log to HDFS. It applies
> thread pool to execute upload task.
> The workflow of upload log as follow:
> # NM construct Applicaiton object when first container of a certain
> application launch, then notify LogAggregationService to init
> AppLogAggregationImpl.
> # LogAggregationService submit AppLogAggregationImpl to task queue
> # The idle worker of thread pool pulls AppLogAggregationImpl from task queue.
> # AppLogAggregationImpl do while loop to check the application state, do
> upload when application finished.
> Suppose the following scenario:
> * LogAggregationService
[jira] [Commented] (YARN-11649) YARN Federation getNewApplication returns different maxresourcecapability
[
https://issues.apache.org/jira/browse/YARN-11649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841015#comment-17841015
]
ASF GitHub Bot commented on YARN-11649:
---
hadoop-yetus commented on PR #6481:
URL: https://github.com/apache/hadoop/pull/6481#issuecomment-2078481513
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 00s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 45s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 90m 10s | | trunk passed |
| +1 :green_heart: | compile | 11m 27s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 16s | | trunk passed |
| +1 :green_heart: | mvnsite | 15m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 15m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 171m 00s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 8m 35s | | the patch passed |
| +1 :green_heart: | compile | 9m 41s | | the patch passed |
| +1 :green_heart: | javac | 9m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 39s | | the patch passed |
| +1 :green_heart: | mvnsite | 10m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 10m 38s | | the patch passed |
| +1 :green_heart: | shadedclient | 178m 22s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 45s | | The patch does not
generate ASF License warnings. |
| | | 488m 11s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6481 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | MINGW64_NT-10.0-17763 e18c298462c5 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 1349aec7840f391e4229fd7a1b0d25159d9aa74d |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6481/1/testReport/
|
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U:
hadoop-yarn-project/hadoop-yarn |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6481/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> YARN Federation getNewApplication returns different maxresourcecapability
> -
>
> Key: YARN-11649
> URL: https://issues.apache.org/jira/browse/YARN-11649
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jeffrey Chang
>Assignee: Jeffrey Chang
>Priority: Minor
> Labels: pull-request-available
>
> When getNewApplication is called against YARN Router with Federation on, its
> possible we get different maxResourceCapabilities in its response on
> different calls. This is because getNewApplication is called against a random
> cluster on each call, which may return different maxResourceCapability based
> on the cluster that the call is executed on. ie in response in HTTP
> [https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_New_Application_API]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (YARN-11690) Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
[
https://issues.apache.org/jira/browse/YARN-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841014#comment-17841014
]
ASF GitHub Bot commented on YARN-11690:
---
tomicooler commented on code in PR #6771:
URL: https://github.com/apache/hadoop/pull/6771#discussion_r1580335818
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h:
##
@@ -80,6 +80,11 @@ enum operations {
#define ROOT_VAR_TMP_DIR "private_var_slash_tmp"
#define COMMAND_FILE_SECTION "command-execution"
+// include cgroup2 super magic number if not defined
+#ifndef CGROUP2_SUPER_MAGIC
Review Comment:
This must be done after the magic.h include. Let's move this to the .c file
somewhere after the include.
> Update container executor to use CGROUP2_SUPER_MAGIC in cgroup 2 scenarios
> --
>
> Key: YARN-11690
> URL: https://issues.apache.org/jira/browse/YARN-11690
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: container-executor
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> The container executor function {{write_pid_to_cgroup_as_root}} writes the
> PID of the newly launched container to the correct cgroup.procs file. However
> it checks if the file is mounted on a cgroup filesystem, and does that check
> using the magic number, which differs for v1 and v2. This should handle v1 or
> v2 filesystems as well.
> {code:java}
> /**
> * Write the pid of the current process to the cgroup file.
> * cgroup_file: Path to cgroup file where pid needs to be written to.
> */
> static int write_pid_to_cgroup_as_root(const char* cgroup_file, pid_t pid) {
> int rc = 0;
> uid_t user = geteuid();
> gid_t group = getegid();
> if (change_effective_user(0, 0) != 0) {
> rc = -1;
> goto cleanup;
> }
> // statfs
> struct statfs buf;
> if (statfs(cgroup_file, &buf) == -1) {
> fprintf(LOGFILE, "Can't statfs file %s as node manager - %s\n",
> cgroup_file,
>strerror(errno));
> rc = -1;
> goto cleanup;
> } else if (buf.f_type != CGROUP_SUPER_MAGIC) {
> fprintf(LOGFILE, "Pid file %s is not located on cgroup filesystem\n",
> cgroup_file);
> rc = -1;
> goto cleanup;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11654) [JDK17] TestLinuxContainerExecutorWithMocks.testStartLocalizer fails
[
https://issues.apache.org/jira/browse/YARN-11654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840902#comment-17840902
]
ASF GitHub Bot commented on YARN-11654:
---
hadoop-yetus commented on PR #6528:
URL: https://github.com/apache/hadoop/pull/6528#issuecomment-2077874147
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 94m 09s | | trunk passed |
| +1 :green_heart: | compile | 5m 02s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 151m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 09s | | the patch passed |
| +1 :green_heart: | compile | 2m 36s | | the patch passed |
| +1 :green_heart: | javac | 2m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 08s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 45s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 32s | | the patch passed |
| +1 :green_heart: | shadedclient | 157m 56s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 44s | | The patch does not
generate ASF License warnings. |
| | | 427m 40s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6528 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 9d60b5d6be65 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / be835e44742a7755c26d1b26d196db8cb2894e6a |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6528/1/testReport/
|
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6528/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> [JDK17] TestLinuxContainerExecutorWithMocks.testStartLocalizer fails
>
>
> Key: YARN-11654
> URL: https://issues.apache.org/jira/browse/YARN-11654
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.4.0
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
> Labels: pull-request-available
>
> [ERROR] TestLinuxContainerExecutorWithMocks.testStartLocalizer:310
> Expected size:<26> but was:<28> in:
> <["nobody",
> "test",
> "0",
> "application_0",
> "12345",
> "/bin/nmPrivateCTokensPath",
>
> "/Users/bilwa/code/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/tmp/nm-local-dir",
> "src/test/resources",
>
> "/opt/homebrew/Cellar/openjdk@17/17.0.8/libexec/openjdk.jdk/Contents/Home/bin/java",
> "-classpath",
>
> "/Users/bilwa/code/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/test-classes:/Users/bilwa/code/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/classes:/Users/bilwa/.m2/repository/org/apache/hadoop/hadoop-common/3.5.0-SNAPSHOT/hadoop-common-3.5.0-SNAPSHOT.jar:/Users/bi
