[jira] [Commented] (YARN-11236) Implement FederationReservationHomeSubClusterStore With MemoryStore
[
https://issues.apache.org/jira/browse/YARN-11236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17576145#comment-17576145
]
ASF GitHub Bot commented on YARN-11236:
---
hadoop-yetus commented on PR #4711:
URL: https://github.com/apache/hadoop/pull/4711#issuecomment-1207149152
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 25s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 49s | | trunk passed |
| +1 :green_heart: | compile | 4m 1s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 29s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 19s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 6s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 56s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 40s | | the patch passed |
| +1 :green_heart: | compile | 3m 49s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | cc | 3m 49s | | the patch passed |
| -1 :x: | javac | 3m 49s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4711/2/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 3 new +
446 unchanged - 0 fixed = 449 total (was 446) |
| +1 :green_heart: | compile | 3m 16s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 3m 16s | | the patch passed |
| -1 :x: | javac | 3m 16s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4711/2/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 3 new
+ 370 unchanged - 0 fixed = 373 total (was 370) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 11s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new +
11 unchanged - 1 fixed = 11 total (was 12) |
| +1 :green_heart: | mvnsite | 1m 49s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 31s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 43s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 42s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 5s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart:
[jira] [Commented] (YARN-10846) Add dispatcher metrics to NM
[ https://issues.apache.org/jira/browse/YARN-10846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17576130#comment-17576130 ] ASF GitHub Bot commented on YARN-10846: --- slfan1989 commented on PR #4687: URL: https://github.com/apache/hadoop/pull/4687#issuecomment-1207143284 @jojochuang Can you help review this PR? I refer to the code of [YARN-9615](https://issues.apache.org/jira/browse/YARN-9615) and the [YARN-10846](https://issues.apache.org/jira/browse/YARN-10846) patch to complete this pr. This pr adds some metrics to NM's dispatcher. > Add dispatcher metrics to NM > > > Key: YARN-10846 > URL: https://issues.apache.org/jira/browse/YARN-10846 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: chaosju >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > Attachments: YARN-10846.001.patch, screenshot-1.png > > Time Spent: 1h 20m > Remaining Estimate: 0h > > base [YARN-9615|https://issues.apache.org/jira/browse/YARN-9615], add > dispatcher metrics to NM -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6539) Create SecureLogin inside Router
[ https://issues.apache.org/jira/browse/YARN-6539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17576129#comment-17576129 ] ASF GitHub Bot commented on YARN-6539: -- slfan1989 commented on PR #4712: URL: https://github.com/apache/hadoop/pull/4712#issuecomment-1207142315 Because the implementation of YarnRouterApi (DelegationToken) requires a router in safe mode, I will continue to follow up on this pr, thanks to those who originally contributed to this jira. > Create SecureLogin inside Router > > > Key: YARN-6539 > URL: https://issues.apache.org/jira/browse/YARN-6539 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Xie YiFan >Priority: Minor > Labels: pull-request-available > Attachments: YARN-6359_1.patch, YARN-6359_2.patch, > YARN-6539-branch-3.1.0.004.patch, YARN-6539-branch-3.1.0.005.patch, > YARN-6539.006.patch, YARN-6539.007.patch, YARN-6539.008.patch, > YARN-6539_3.patch, YARN-6539_4.patch > > Time Spent: 5.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6539) Create SecureLogin inside Router
[ https://issues.apache.org/jira/browse/YARN-6539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17576128#comment-17576128 ] ASF GitHub Bot commented on YARN-6539: -- slfan1989 opened a new pull request, #4712: URL: https://github.com/apache/hadoop/pull/4712 JIRA. YARN-6539. Consider running RM tests against the Router. > Create SecureLogin inside Router > > > Key: YARN-6539 > URL: https://issues.apache.org/jira/browse/YARN-6539 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Xie YiFan >Priority: Minor > Labels: pull-request-available > Attachments: YARN-6359_1.patch, YARN-6359_2.patch, > YARN-6539-branch-3.1.0.004.patch, YARN-6539-branch-3.1.0.005.patch, > YARN-6539.006.patch, YARN-6539.007.patch, YARN-6539.008.patch, > YARN-6539_3.patch, YARN-6539_4.patch > > Time Spent: 5.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9048) Add znode hierarchy in Federation ZK State Store
[ https://issues.apache.org/jira/browse/YARN-9048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17576095#comment-17576095 ] fanshilun commented on YARN-9048: - I will continue to follow up on this pr. > Add znode hierarchy in Federation ZK State Store > > > Key: YARN-9048 > URL: https://issues.apache.org/jira/browse/YARN-9048 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin Chundatt >Assignee: fanshilun >Priority: Major > > Similar to YARN-2962 consider having hierarchy in ZK federation store for > applications -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9048) Add znode hierarchy in Federation ZK State Store
[ https://issues.apache.org/jira/browse/YARN-9048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fanshilun reassigned YARN-9048: --- Assignee: fanshilun > Add znode hierarchy in Federation ZK State Store > > > Key: YARN-9048 > URL: https://issues.apache.org/jira/browse/YARN-9048 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin Chundatt >Assignee: fanshilun >Priority: Major > > Similar to YARN-2962 consider having hierarchy in ZK federation store for > applications -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8972) [Router] Add support to prevent DoS attack over ApplicationSubmissionContext size
[ https://issues.apache.org/jira/browse/YARN-8972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17576094#comment-17576094 ] fanshilun commented on YARN-8972: - I will continue to follow up on this pr. > [Router] Add support to prevent DoS attack over ApplicationSubmissionContext > size > - > > Key: YARN-8972 > URL: https://issues.apache.org/jira/browse/YARN-8972 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola >Priority: Major > Attachments: YARN-8972.v1.patch, YARN-8972.v2.patch, > YARN-8972.v3.patch, YARN-8972.v4.patch, YARN-8972.v5.patch > > > This jira tracks the effort to add a new interceptor in the Router to prevent > user to submit applications with oversized ASC. > This avoid YARN cluster to failover. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6476) Advanced Federation UI based on YARN UI v2
[ https://issues.apache.org/jira/browse/YARN-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17576093#comment-17576093 ] fanshilun commented on YARN-6476: - I will continue to follow up on this pr. > Advanced Federation UI based on YARN UI v2 > -- > > Key: YARN-6476 > URL: https://issues.apache.org/jira/browse/YARN-6476 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Tanuj Nayak >Priority: Major > Attachments: YARN-6476-screenshots.zip, YARN-6476.001.patch > > > Extension of YARN-3661 based on new YARN UI YARN-3368. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11236) Implement FederationReservationHomeSubClusterStore With MemoryStore
[
https://issues.apache.org/jira/browse/YARN-11236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17576087#comment-17576087
]
ASF GitHub Bot commented on YARN-11236:
---
slfan1989 commented on code in PR #4711:
URL: https://github.com/apache/hadoop/pull/4711#discussion_r939449057
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/AbstractRouterPolicy.java:
##
@@ -98,7 +99,16 @@ protected Map
prefilterSubClusters(
// if a reservation exists limit scope to the sub-cluster this
// reservation is mapped to
-// TODO: Implemented in YARN-11236
+if (reservationId != null) {
+ // note this might throw YarnException if the reservation is
+ // unknown. This is to be expected, and should be handled by
+ // policy invoker.
+ SubClusterId resSubCluster =
getPolicyContext().getFederationStateStoreFacade().
Review Comment:
Thank you very much for your help reviewing the code, I will fix it.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/AbstractRouterPolicy.java:
##
@@ -98,7 +99,16 @@ protected Map
prefilterSubClusters(
// if a reservation exists limit scope to the sub-cluster this
// reservation is mapped to
-// TODO: Implemented in YARN-11236
+if (reservationId != null) {
+ // note this might throw YarnException if the reservation is
+ // unknown. This is to be expected, and should be handled by
+ // policy invoker.
+ SubClusterId resSubCluster =
getPolicyContext().getFederationStateStoreFacade().
+ getReservationHomeSubCluster(reservationId);
+
+ return Collections.singletonMap(resSubCluster,
activeSubClusters.get(resSubCluster));
Review Comment:
I will fix it.
> Implement FederationReservationHomeSubClusterStore With MemoryStore
> ---
>
> Key: YARN-11236
> URL: https://issues.apache.org/jira/browse/YARN-11236
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11236) Implement FederationReservationHomeSubClusterStore With MemoryStore
[
https://issues.apache.org/jira/browse/YARN-11236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17576068#comment-17576068
]
ASF GitHub Bot commented on YARN-11236:
---
goiri commented on code in PR #4711:
URL: https://github.com/apache/hadoop/pull/4711#discussion_r939396343
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/MemoryFederationStateStore.java:
##
@@ -312,4 +324,41 @@ public Version loadVersion() {
return null;
}
+ @Override
+ public AddReservationHomeSubClusterResponse addReservationHomeSubCluster(
+ AddReservationHomeSubClusterRequest request) throws YarnException {
+FederationReservationHomeSubClusterStoreInputValidator
+.validateAddReservationHomeSubClusterRequest(request);
Review Comment:
These names are pretty verose.
If we are in FederationReservationHomeSubClusterStoreInputValidator isn't
enough to say validateRequest? or validate as the type is in the arg?
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/utils/FederationReservationHomeSubClusterStoreInputValidator.java:
##
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.federation.store.utils;
+
+import org.apache.hadoop.yarn.api.records.ReservationId;
+import
org.apache.hadoop.yarn.server.federation.store.exception.FederationStateStoreInvalidInputException;
+import
org.apache.hadoop.yarn.server.federation.store.records.AddReservationHomeSubClusterRequest;
+import
org.apache.hadoop.yarn.server.federation.store.records.ReservationHomeSubCluster;
+import
org.apache.hadoop.yarn.server.federation.store.records.GetReservationHomeSubClusterRequest;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Utility class to validate the inputs to
+ * {@code FederationReservationHomeSubClusterStore}, allows a fail fast
+ * mechanism for invalid user inputs.
+ *
+ */
+public final class FederationReservationHomeSubClusterStoreInputValidator {
+
+ private static final Logger LOG = LoggerFactory
+ .getLogger(FederationReservationHomeSubClusterStoreInputValidator.class);
+
+ private FederationReservationHomeSubClusterStoreInputValidator() {
+ }
+
+ /**
+ * Quick validation on the input to check some obvious fail conditions (fail
+ * fast). Check if the provided {@link AddReservationHomeSubClusterRequest}
+ * for adding a new reservation is valid or not.
+ *
+ * @param request the {@link AddReservationHomeSubClusterRequest} to validate
+ * against
+ * @throws FederationStateStoreInvalidInputException if the request is
invalid
+ */
+ public static void validateAddReservationHomeSubClusterRequest(
+ AddReservationHomeSubClusterRequest request)
+ throws FederationStateStoreInvalidInputException {
+if (request == null) {
+ String message = "Missing AddReservationHomeSubCluster Request."
+ + " Please try again by specifying"
+ + " an AddReservationHomeSubCluster information.";
+ LOG.warn(message);
+ throw new FederationStateStoreInvalidInputException(message);
+}
+
+// validate ReservationHomeSubCluster info
+checkReservationHomeSubCluster(request.getReservationHomeSubCluster());
+ }
+
+ /**
+ * Quick validation on the input to check some obvious fail conditions (fail
+ * fast). Check if the provided {@link GetReservationHomeSubClusterRequest}
+ * for querying reservation's information is valid or not.
+ *
+ * @param request the {@link GetReservationHomeSubClusterRequest} to validate
+ * against
+ * @throws FederationStateStoreInvalidInputException if the request is
invalid
+ */
+ public static void validateGetReservationHomeSubClusterRequest(
Review Comment:
validate() should be enough as the type is there.
##
[jira] [Commented] (YARN-11236) Implement FederationReservationHomeSubClusterStore With MemoryStore
[ https://issues.apache.org/jira/browse/YARN-11236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17576060#comment-17576060 ] ASF GitHub Bot commented on YARN-11236: --- slfan1989 commented on PR #4711: URL: https://github.com/apache/hadoop/pull/4711#issuecomment-1206930345 @goiri Could you please help to review the code? Thank you very much! > Implement FederationReservationHomeSubClusterStore With MemoryStore > --- > > Key: YARN-11236 > URL: https://issues.apache.org/jira/browse/YARN-11236 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11236) Implement FederationReservationHomeSubClusterStore With MemoryStore
[
https://issues.apache.org/jira/browse/YARN-11236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17576040#comment-17576040
]
ASF GitHub Bot commented on YARN-11236:
---
hadoop-yetus commented on PR #4711:
URL: https://github.com/apache/hadoop/pull/4711#issuecomment-1206819328
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 1s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 14s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 1s | | trunk passed |
| +1 :green_heart: | compile | 4m 25s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 38s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 18s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 2s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 52s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 28s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 44s | | the patch passed |
| +1 :green_heart: | compile | 4m 13s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | cc | 4m 13s | | the patch passed |
| -1 :x: | javac | 4m 13s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4711/1/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 3 new +
446 unchanged - 0 fixed = 449 total (was 446) |
| +1 :green_heart: | compile | 3m 36s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 3m 36s | | the patch passed |
| -1 :x: | javac | 3m 36s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4711/1/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 3 new
+ 370 unchanged - 0 fixed = 373 total (was 370) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 14s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4711/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt)
| hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 1
new + 11 unchanged - 1 fixed = 12 total (was 12) |
| +1 :green_heart: | mvnsite | 1m 52s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 35s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 26s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 56s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m
[jira] [Commented] (YARN-10885) Make FederationStateStoreFacade#getApplicationHomeSubCluster use JCache
[
https://issues.apache.org/jira/browse/YARN-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575997#comment-17575997
]
ASF GitHub Bot commented on YARN-10885:
---
hadoop-yetus commented on PR #4701:
URL: https://github.com/apache/hadoop/pull/4701#issuecomment-1206720683
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 39s | | trunk passed |
| +1 :green_heart: | compile | 0m 52s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 52s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 54s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 3s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 21s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common:
The patch generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 25s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 42s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 47s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 105m 4s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4701/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4701 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux df77139aa86b 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f6096e23c15ee26061410ab3f012c59b3f0777b6 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4701/7/testReport/ |
| Max. process+thread count | 627 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output |
[jira] [Commented] (YARN-10885) Make FederationStateStoreFacade#getApplicationHomeSubCluster use JCache
[ https://issues.apache.org/jira/browse/YARN-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575949#comment-17575949 ] ASF GitHub Bot commented on YARN-10885: --- slfan1989 commented on PR #4701: URL: https://github.com/apache/hadoop/pull/4701#issuecomment-1206638972 @goiri Please help to review the code again, thank you very much! > Make FederationStateStoreFacade#getApplicationHomeSubCluster use JCache > --- > > Key: YARN-10885 > URL: https://issues.apache.org/jira/browse/YARN-10885 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: chaosju >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > Yarn Client getApplicationReport function may produce lots of zookeeper ops, > Its import to use the JCache that cache the mapping of application and > subcluster id. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10885) Make FederationStateStoreFacade#getApplicationHomeSubCluster use JCache
[
https://issues.apache.org/jira/browse/YARN-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575946#comment-17575946
]
ASF GitHub Bot commented on YARN-10885:
---
hadoop-yetus commented on PR #4701:
URL: https://github.com/apache/hadoop/pull/4701#issuecomment-1206638096
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 0s | | trunk passed |
| +1 :green_heart: | compile | 0m 51s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 46s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 51s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 40s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 2s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 40s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 22s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common:
The patch generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | mvnsite | 0m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 31s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 51s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 46s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 105m 30s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4701/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4701 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 28cfbcc832ba 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0eefb19e003b934c7314abbfced5203f959148d0 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4701/6/testReport/ |
| Max. process+thread count | 534 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output |
[jira] [Commented] (YARN-10885) Make FederationStateStoreFacade#getApplicationHomeSubCluster use JCache
[
https://issues.apache.org/jira/browse/YARN-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575947#comment-17575947
]
ASF GitHub Bot commented on YARN-10885:
---
slfan1989 commented on code in PR #4701:
URL: https://github.com/apache/hadoop/pull/4701#discussion_r938980459
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/utils/TestFederationStateStoreFacade.java:
##
@@ -206,4 +206,19 @@ public void testAddApplicationHomeSubCluster() throws
YarnException {
Assert.assertEquals(subClusterId1, result);
}
+ @Test
+ public void testGetApplicationHomeSubClusterCache() throws YarnException {
+ApplicationId appId = ApplicationId.newInstance(clusterTs, numApps + 1);
+SubClusterId subClusterId1 = SubClusterId.newInstance("Home1");
+
+ApplicationHomeSubCluster appHomeSubCluster =
+ApplicationHomeSubCluster.newInstance(appId, subClusterId1);
+SubClusterId subClusterIdAdd =
+facade.addApplicationHomeSubCluster(appHomeSubCluster);
+
+SubClusterId subClusterIdCache =
facade.getApplicationHomeSubCluster(appId);
+Assert.assertEquals(subClusterIdCache, subClusterIdAdd);
Review Comment:
I refactored the test for getting objects from Cache. If the test case
enables Cache mode, the comparison will be as follows:
Whether the object obtained directly from the Cache and the object obtained
from the Facade are the same.
> Make FederationStateStoreFacade#getApplicationHomeSubCluster use JCache
> ---
>
> Key: YARN-10885
> URL: https://issues.apache.org/jira/browse/YARN-10885
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: chaosju
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Yarn Client getApplicationReport function may produce lots of zookeeper ops,
> Its import to use the JCache that cache the mapping of application and
> subcluster id.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11228) [Federation] Add getAppAttempts, getAppAttempt REST APIs for Router
[ https://issues.apache.org/jira/browse/YARN-11228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575938#comment-17575938 ] ASF GitHub Bot commented on YARN-11228: --- slfan1989 commented on PR #4695: URL: https://github.com/apache/hadoop/pull/4695#issuecomment-1206623737 @goiri Thank you very much for your help reviewing the code, I will follow up with YARN-11227. > [Federation] Add getAppAttempts, getAppAttempt REST APIs for Router > --- > > Key: YARN-11228 > URL: https://issues.apache.org/jira/browse/YARN-11228 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0, 3.3.4 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11236) Implement FederationReservationHomeSubClusterStore With MemoryStore
[ https://issues.apache.org/jira/browse/YARN-11236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575926#comment-17575926 ] ASF GitHub Bot commented on YARN-11236: --- slfan1989 opened a new pull request, #4711: URL: https://github.com/apache/hadoop/pull/4711 JIRA:YARN-11236. Implement FederationReservationHomeSubClusterStore With MemoryStore. > Implement FederationReservationHomeSubClusterStore With MemoryStore > --- > > Key: YARN-11236 > URL: https://issues.apache.org/jira/browse/YARN-11236 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11236) Implement FederationReservationHomeSubClusterStore With MemoryStore
[ https://issues.apache.org/jira/browse/YARN-11236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11236: -- Labels: pull-request-available (was: ) > Implement FederationReservationHomeSubClusterStore With MemoryStore > --- > > Key: YARN-11236 > URL: https://issues.apache.org/jira/browse/YARN-11236 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11241) Add uncleaning option for local app log file with log-aggregation enabled
[ https://issues.apache.org/jira/browse/YARN-11241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575923#comment-17575923 ] ASF GitHub Bot commented on YARN-11241: --- ashutoshcipher commented on PR #4703: URL: https://github.com/apache/hadoop/pull/4703#issuecomment-1206601744 FYI: Checkstyle -0 showing here is not due changes in this PR. > Add uncleaning option for local app log file with log-aggregation enabled > - > > Key: YARN-11241 > URL: https://issues.apache.org/jira/browse/YARN-11241 > Project: Hadoop YARN > Issue Type: New Feature > Components: log-aggregation >Reporter: groot >Assignee: groot >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Add uncleaning option for local app log file with log-aggregation enabled > This will be helpful for debugging purpose. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10846) Add dispatcher metrics to NM
[ https://issues.apache.org/jira/browse/YARN-10846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575920#comment-17575920 ] ASF GitHub Bot commented on YARN-10846: --- slfan1989 commented on PR #4687: URL: https://github.com/apache/hadoop/pull/4687#issuecomment-1206593455 @ayushtkn Can you help review this PR? I refer to the code of [YARN-9615](https://issues.apache.org/jira/browse/YARN-9615) and the [YARN-10846](https://issues.apache.org/jira/browse/YARN-10846) patch to complete this pr. This pr adds some metrics to NM's dispatcher. > Add dispatcher metrics to NM > > > Key: YARN-10846 > URL: https://issues.apache.org/jira/browse/YARN-10846 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: chaosju >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > Attachments: YARN-10846.001.patch, screenshot-1.png > > Time Spent: 1h 20m > Remaining Estimate: 0h > > base [YARN-9615|https://issues.apache.org/jira/browse/YARN-9615], add > dispatcher metrics to NM -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7833) [PERF/TEST] Extend SLS to support simulation of a Federated Environment
[ https://issues.apache.org/jira/browse/YARN-7833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575905#comment-17575905 ] fanshilun commented on YARN-7833: - I will continue to follow up on this pr. > [PERF/TEST] Extend SLS to support simulation of a Federated Environment > --- > > Key: YARN-7833 > URL: https://issues.apache.org/jira/browse/YARN-7833 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Tanuj Nayak >Priority: Major > Attachments: YARN-7833.v1.patch, YARN-7833.v2.patch, > YARN-7833.v3.patch, YARN-7833.v4.patch, YARN-7833.v5.patch, > YARN-7833.v6.patch, YARN-7833.v7.patch > > > To develop algorithms for federation, it would be of great help to have a > version of SLS that supports multi RMs and GPG. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11241) Add uncleaning option for local app log file with log-aggregation enabled
[
https://issues.apache.org/jira/browse/YARN-11241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575816#comment-17575816
]
ASF GitHub Bot commented on YARN-11241:
---
hadoop-yetus commented on PR #4703:
URL: https://github.com/apache/hadoop/pull/4703#issuecomment-1206390918
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 25s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 19s | | trunk passed |
| +1 :green_heart: | compile | 11m 4s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 9m 37s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 51s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 46s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 36s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 5s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 7s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 58s | | the patch passed |
| +1 :green_heart: | compile | 9m 44s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 9m 44s | | the patch passed |
| +1 :green_heart: | compile | 9m 12s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 9m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 44s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4703/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 286 unchanged
- 2 fixed = 287 total (was 288) |
| +1 :green_heart: | mvnsite | 2m 49s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 22s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 16s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 41s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 6s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 4m 58s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 24m 7s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not
generate ASF License warnings. |
| | | 197m 35s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4703/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4703 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 10de55906fe3 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3
10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / dad54d34a12f75a1dff228b4bd9881ca154dc46f |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
[jira] [Commented] (YARN-10900) Yarn nodes missing from router web app
[
https://issues.apache.org/jira/browse/YARN-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575598#comment-17575598
]
zhangjunj commented on YARN-10900:
--
yarn.router.webapp.interceptor-class.pipeline=org.apache.hadoop.yarn.server.router.webapp.FederationInterceptorREST
Yes， this method can solve your problem.
> Yarn nodes missing from router web app
> --
>
> Key: YARN-10900
> URL: https://issues.apache.org/jira/browse/YARN-10900
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation, router
>Affects Versions: 3.2.1
>Reporter: Babble Shack
>Priority: Major
>
> {color:#172b4d}Hi,
> I am trying to configure YARN Federation mode.I seem to be able to schedule
> to all nodes in my federation across each of my subclusters.
>
> However my federation router shows both of my subclusters, but nodes from
> only a single cluster.{color}
>
>
> {color:#ff}[!https://preview.redd.it/hdjwtn43ptj71.png?width=1437=png=webp=2d55343688c0de7a6f3da629e334cd318219c392!|https://preview.redd.it/hdjwtn43ptj71.png?width=1437=png=webp=2d55343688c0de7a6f3da629e334cd318219c392]{color}
> {color:#172b4d}Federation Page – Showing both clusters and both nodes{color}
> {color:#172b4d}
> This page is showing both of my clusters, configured with a single <8 CPU,
> 7GB> node.{color}
> {color:#172b4d}However the "Nodes" and "About" pages are invalid.{color}
>
>
> [!https://preview.redd.it/lawgst2yotj71.png?width=1373=png=webp=d06663904538bc993418c6184c3686cc2b02ea6e!|https://preview.redd.it/lawgst2yotj71.png?width=1373=png=webp=d06663904538bc993418c6184c3686cc2b02ea6e]
> {color:#172b4d}Nodes Page – showing nodes from only one cluster{color}
>
>
> [!https://preview.redd.it/dtuqblquotj71.png?width=482=png=webp=df740ff49df8b8de5015bccc936c9accee65aee0!|https://preview.redd.it/dtuqblquotj71.png?width=482=png=webp=df740ff49df8b8de5015bccc936c9accee65aee0]
> {color:#172b4d}About Page – showing nodes from only one cluster{color}
> {color:#172b4d}Each node is configured as follows:{color}
> {color:#172b4d}*Minimum Memory Allocation:* 512MB{color}
> {color:#172b4d}*Minimum CPU Allocation:* 1{color}
> {color:#172b4d}*Maximum Memory Allocation:* 7168{color}
> {color:#172b4d}*Maximum CPU Allocation:* 7{color}
> {color:#172b4d}Federation configuration can be found at this
> [link|https://drive.google.com/file/d/16xc2V7CvJLVQgsaDEOHhIaDrz5dnKxB_/view?usp=sharing]{color}
> {color:#172b4d}Has anyone had an issue like this before, does anyone have any
> solutions?{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
