[jira] [Created] (YARN-11412) Create a Concurrent Users Manager to enable thread safe concurrent resource usage tracking of user
Krishan Goyal created YARN-11412: Summary: Create a Concurrent Users Manager to enable thread safe concurrent resource usage tracking of user Key: YARN-11412 URL: https://issues.apache.org/jira/browse/YARN-11412 Project: Hadoop YARN Issue Type: Sub-task Reporter: Krishan Goyal Assignee: Krishan Goyal We will build ConcurrentUsersManager whose objective is similar to UsersManager but whose APIs provide better concurrent performance with different class invariants Both ConcurrentUsersManager & UsersManager will extend a new class AbstractCSUsersManager which implements AbstractUsersManager The primary behavioural change of ConcurrentUsersManager is that it will provide user resource limits which are eventually consistent wrt containers (allocation or release) / users (active or inactive) - These limits affect scheduling and preemption but at massive scale small variances in individual decisions for a container are fine as long as the scheduler progresses in an acceptable way (i.e - users are unconcerned about these variances) More details of low level details - [https://docs.google.com/document/d/1czUh2XU3_X_eRIJAsSM40hRuHYSHw4ymyfky39dIj4s/edit] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11411) [Umbrella] Build Concurrent Yarn Scheduler
Krishan Goyal created YARN-11411: Summary: [Umbrella] Build Concurrent Yarn Scheduler Key: YARN-11411 URL: https://issues.apache.org/jira/browse/YARN-11411 Project: Hadoop YARN Issue Type: New Feature Reporter: Krishan Goyal Assignee: Krishan Goyal We operate multiple yarn clusters with each cluster capped to ~ 10k nodes which is its scalability limit. We expect multiple benefits with fewer clusters & larger cluster sizes (better elasticity, operational simplicity, larger queues). Thus, we want to scale a single yarn cluster to as much as possible in terms of number of nodes heartbeating to the cluster (& proportionally increase container allocation rate) without degradation in overall quantiles (p50 / p75 / p95) of container allocation delay The scalability limit of a yarn cluster is primarily driven by RM’s processing of node heartbeats & container allocation. The CPU usage of our RM is < 10% & RM is primarily bottlenecked on global queue & user read/write locks for container allocation By removing these locks (through a very naive & incorrect implementation), we were able to scale RM to 25k nodes (& proportional increase in container allocs/sec) with avg RM CPU utilization of 20% (so there is still room for improvement to use more CPU / scale up further). This primarily requires # Async scheduling to decouple scheduling from node heartbeats (existing feature) # Removing global write locks in scheduler path (primarily to maintain queues and users) # Multi threaded event queue dispatcher to process events parallelly Additionally we need to probably scale RPC handling, DT management, preemption flows, Timeline server, RM HA failover. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11217) [Federation] Add dumpSchedulerLogs REST APIs for Router
[
https://issues.apache.org/jira/browse/YARN-11217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655904#comment-17655904
]
ASF GitHub Bot commented on YARN-11217:
---
hadoop-yetus commented on PR #5272:
URL: https://github.com/apache/hadoop/pull/5272#issuecomment-1375097988
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 48s | | trunk passed |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 23s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 22s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | compile | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 53s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 28s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 97m 28s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5272/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5272 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux f5d7686c0c55 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f54114926135e354cfe243d895ec1f84e15b99a9 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5272/4/testReport/ |
| Max. process+thread count | 613 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5272/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
[jira] [Commented] (YARN-11217) [Federation] Add dumpSchedulerLogs REST APIs for Router
[
https://issues.apache.org/jira/browse/YARN-11217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655867#comment-17655867
]
ASF GitHub Bot commented on YARN-11217:
---
slfan1989 commented on code in PR #5272:
URL: https://github.com/apache/hadoop/pull/5272#discussion_r1064239731
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/FederationInterceptorREST.java:
##
@@ -1145,10 +1147,59 @@ public SchedulerTypeInfo getSchedulerInfo() {
throw new NotImplementedException("Code is not implemented");
}
+ /**
+ * This method dumps the scheduler logs for the time got in input, and it is
+ * reachable by using {@link RMWSConsts#SCHEDULER_LOGS}.
+ *
+ * @param time the period of time. It is a FormParam.
+ * @param hsr the servlet request
+ * @return the result of the operation
+ * @throws IOException when it cannot create dump log file
+ */
@Override
public String dumpSchedulerLogs(String time, HttpServletRequest hsr)
throws IOException {
-throw new NotImplementedException("Code is not implemented");
+
+if (StringUtils.isBlank(time)) {
+ routerMetrics.incrDumpSchedulerLogsFailedRetrieved();
+ throw new IllegalArgumentException("Parameter error, the time is empty
or null.");
+}
+
+int period = Integer.parseInt(time);
Review Comment:
Thank you very much for helping to review the code, I will modify the code.
> [Federation] Add dumpSchedulerLogs REST APIs for Router
> ---
>
> Key: YARN-11217
> URL: https://issues.apache.org/jira/browse/YARN-11217
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.4.0, 3.3.4
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11410) Add default methods for StateMachine
[ https://issues.apache.org/jira/browse/YARN-11410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-11410: -- Description: YARN-11395 created a new method in the StateMachine interface, what can break the compatibility with connected softwares, so the method should be converted to default method, what can prevent this break (was: The YARN-11395 created a new method in the StateMachine interface, what can break the compatibility with connected softwares, so the method should be converted to default method, what can prevent this break) > Add default methods for StateMachine > > > Key: YARN-11410 > URL: https://issues.apache.org/jira/browse/YARN-11410 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bence Kosztolnik >Assignee: Bence Kosztolnik >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > YARN-11395 created a new method in the StateMachine interface, what can break > the compatibility with connected softwares, so the method should be converted > to default method, what can prevent this break -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11190) CS Mapping rule bug: User matcher does not work correctly for usernames with dot
[
https://issues.apache.org/jira/browse/YARN-11190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655802#comment-17655802
]
ASF GitHub Bot commented on YARN-11190:
---
hadoop-yetus commented on PR #4471:
URL: https://github.com/apache/hadoop/pull/4471#issuecomment-1374863004
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 12s | | trunk passed |
| +1 :green_heart: | compile | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 56s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 1s | | trunk passed |
| -1 :x: | javadoc | 0m 54s |
[/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4471/2/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-server-resourcemanager in trunk failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 24s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 53s | | the patch passed |
| +1 :green_heart: | compile | 0m 58s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 58s | | the patch passed |
| +1 :green_heart: | compile | 0m 50s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 50s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4471/2/artifact/out/blanks-eol.txt)
| The patch has 4 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 0m 39s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 54s | | the patch passed |
| -1 :x: | javadoc | 0m 39s |
[/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4471/2/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-server-resourcemanager in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 37s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 59s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 0s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 102m 16s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 207m 35s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4471/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4471 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux ee7c5c1307bc 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64
[jira] [Commented] (YARN-11410) Add default methods for StateMachine
[
https://issues.apache.org/jira/browse/YARN-11410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655792#comment-17655792
]
ASF GitHub Bot commented on YARN-11410:
---
hadoop-yetus commented on PR #5286:
URL: https://github.com/apache/hadoop/pull/5286#issuecomment-1374831627
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 13s | | trunk passed |
| +1 :green_heart: | compile | 0m 50s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 46s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 51s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 51s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 42s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 42s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 24s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5286/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt)
| hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: The patch generated 5
new + 0 unchanged - 3 fixed = 5 total (was 3) |
| +1 :green_heart: | mvnsite | 0m 40s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 59s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 5m 31s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 99m 15s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5286/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5286 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 877d72054503 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / de94b2a4674b72c0cccb47d2ed17538246f1c41e |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5286/1/testReport/ |
| Max. process+thread count | 700 (vs. ulimit of 5500) |
[jira] [Commented] (YARN-11410) Add default methods for StateMachine
[ https://issues.apache.org/jira/browse/YARN-11410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655786#comment-17655786 ] ASF GitHub Bot commented on YARN-11410: --- K0K0V0K opened a new pull request, #5286: URL: https://github.com/apache/hadoop/pull/5286 - Fix possible breaks what can be introduced by the YARN-11395. ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Add default methods for StateMachine > > > Key: YARN-11410 > URL: https://issues.apache.org/jira/browse/YARN-11410 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bence Kosztolnik >Assignee: Bence Kosztolnik >Priority: Major > Fix For: 3.4.0 > > > The YARN-11395 created a new method in the StateMachine interface, what can > break the compatibility with connected softwares, so the method should be > converted to default method, what can prevent this break -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11410) Add default methods for StateMachine
[ https://issues.apache.org/jira/browse/YARN-11410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11410: -- Labels: pull-request-available (was: ) > Add default methods for StateMachine > > > Key: YARN-11410 > URL: https://issues.apache.org/jira/browse/YARN-11410 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bence Kosztolnik >Assignee: Bence Kosztolnik >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > The YARN-11395 created a new method in the StateMachine interface, what can > break the compatibility with connected softwares, so the method should be > converted to default method, what can prevent this break -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-11410) Add default methods for StateMachine
[ https://issues.apache.org/jira/browse/YARN-11410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bence Kosztolnik reassigned YARN-11410: --- Assignee: Bence Kosztolnik > Add default methods for StateMachine > > > Key: YARN-11410 > URL: https://issues.apache.org/jira/browse/YARN-11410 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bence Kosztolnik >Assignee: Bence Kosztolnik >Priority: Major > Fix For: 3.4.0 > > > The YARN-11395 created a new method in the StateMachine interface, what can > break the compatibility with connected softwares, so the method should be > converted to default method, what can prevent this break -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11410) Add default methods for StateMachine
Bence Kosztolnik created YARN-11410: --- Summary: Add default methods for StateMachine Key: YARN-11410 URL: https://issues.apache.org/jira/browse/YARN-11410 Project: Hadoop YARN Issue Type: Bug Reporter: Bence Kosztolnik Fix For: 3.4.0 The YARN-11395 created a new method in the StateMachine interface, what can break the compatibility with connected softwares, so the method should be converted to default method, what can prevent this break -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8900) [Router] Federation: routing getContainers REST invocations transparently to multiple RMs
[ https://issues.apache.org/jira/browse/YARN-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655783#comment-17655783 ] ASF GitHub Bot commented on YARN-8900: -- goiri commented on code in PR #5260: URL: https://github.com/apache/hadoop/pull/5260#discussion_r1064113871 ## hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/FederationInterceptorREST.java: ## @@ -2512,36 +2513,41 @@ private Map invokeConcurrent(Collection c getMethod(request.getMethodName(), request.getTypes()); Object retObj = method.invoke(interceptor, request.getParams()); R ret = clazz.cast(retObj); - return Pair.of(ret, null); + return Triple.of(info, ret, null); Review Comment: This starts to get too obfuscated. Let's create some temp class like SubClusterResult > [Router] Federation: routing getContainers REST invocations transparently to > multiple RMs > - > > Key: YARN-8900 > URL: https://issues.apache.org/jira/browse/YARN-8900 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation, router >Reporter: Giovanni Matteo Fumarola >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: YARN-8900.v1.patch, YARN-8900.v2.patch > > Time Spent: 5h 50m > Remaining Estimate: 0h > > This JIRA tracks the design/implementation of the layer for routing > RMWebServicesProtocol requests to the appropriate RM(s) in a federated YARN > cluster. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11217) [Federation] Add dumpSchedulerLogs REST APIs for Router
[
https://issues.apache.org/jira/browse/YARN-11217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655781#comment-17655781
]
ASF GitHub Bot commented on YARN-11217:
---
goiri commented on code in PR #5272:
URL: https://github.com/apache/hadoop/pull/5272#discussion_r1064112228
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/FederationInterceptorREST.java:
##
@@ -1145,10 +1147,59 @@ public SchedulerTypeInfo getSchedulerInfo() {
throw new NotImplementedException("Code is not implemented");
}
+ /**
+ * This method dumps the scheduler logs for the time got in input, and it is
+ * reachable by using {@link RMWSConsts#SCHEDULER_LOGS}.
+ *
+ * @param time the period of time. It is a FormParam.
+ * @param hsr the servlet request
+ * @return the result of the operation
+ * @throws IOException when it cannot create dump log file
+ */
@Override
public String dumpSchedulerLogs(String time, HttpServletRequest hsr)
throws IOException {
-throw new NotImplementedException("Code is not implemented");
+
+if (StringUtils.isBlank(time)) {
+ routerMetrics.incrDumpSchedulerLogsFailedRetrieved();
+ throw new IllegalArgumentException("Parameter error, the time is empty
or null.");
+}
+
+int period = Integer.parseInt(time);
Review Comment:
Can this get a string that is not a number?
> [Federation] Add dumpSchedulerLogs REST APIs for Router
> ---
>
> Key: YARN-11217
> URL: https://issues.apache.org/jira/browse/YARN-11217
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.4.0, 3.3.4
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5607) Document TestContainerResourceUsage#waitForContainerCompletion
[ https://issues.apache.org/jira/browse/YARN-5607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17655777#comment-17655777 ] ASF GitHub Bot commented on YARN-5607: -- szilard-nemeth commented on PR #5119: URL: https://github.com/apache/hadoop/pull/5119#issuecomment-1374763585 Hi @susheel-gupta Thanks for working on this. 1. This should be a fairly simple and concise PR. Now we are having 66 changed files. I don't see a reason to move waitForState from MockRM and MockAM. Especially now that ResourceManagerTesterUtil and MockRM / MockAM has the logic scattered. Before the patch, every related logic were in the MockRM / MockAM classes. The description of the jira ticket says that the waitForContainerCompletion method of TestContainerResourceUsage should be moved to a utility class or to a mock class. In this case, MockRM would be a perfect place to move it as this class has similar methods already so the PR would be far more simple. So I don't think creating ResourceManagerTesterUtil has any added value. 2. In fact, the original method in org.apache.hadoop.yarn.server.resourcemanager.MockRM#waitForState is also kept without any remaining usages so it should be removed. 3. It's not a good practice to mix a library upgrade / update (in this case JUnit) with other code changes. I understand that the tests were failing (as per this comment: https://github.com/apache/hadoop/pull/5119#issuecomment-1330105691). The JUnit stuff has to go in before this patch, with a separate jira. Please file another jira for the JUnit upgrade. You can mention that what was the reason to use the new version of it. > Document TestContainerResourceUsage#waitForContainerCompletion > -- > > Key: YARN-5607 > URL: https://issues.apache.org/jira/browse/YARN-5607 > Project: Hadoop YARN > Issue Type: Test > Components: resourcemanager, test >Affects Versions: 2.9.0 >Reporter: Karthik Kambatla >Assignee: Susheel Gupta >Priority: Major > Labels: newbie, pull-request-available > > The logic in TestContainerResourceUsage#waitForContainerCompletion > (introduced in YARN-5024) is not immediately obvious. It could use some > documentation. Also, this seems like a useful helper method. Should this be > moved to one of the mock classes or to a util class? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
