[hadoop] branch trunk updated: Add documentation for YARN-10623 auto refresh queue conf in CS (#3279)
This is an automated email from the ASF dual-hosted git repository. snemeth pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 3024a47 Add documentation for YARN-10623 auto refresh queue conf in CS (#3279) 3024a47 is described below commit 3024a4702676632f3de71196fe5c2714eae1905f Author: zhuqi <821684...@qq.com> AuthorDate: Wed Sep 8 22:03:15 2021 +0800 Add documentation for YARN-10623 auto refresh queue conf in CS (#3279) --- .../hadoop-yarn-site/src/site/markdown/CapacityScheduler.md | 8 1 file changed, 8 insertions(+) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md index 254c51a..d35869c 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md @@ -831,6 +831,14 @@ Changing queue/scheduler properties and adding/removing queues can be done in tw Remove the queue configurations from the file and run refresh as described above +### Enabling periodic configuration refresh +Enabling queue configuration periodic refresh allows reloading and applying the configuration by editing the *conf/capacity-scheduler.xml* without the necessicity of calling yarn rmadmin -refreshQueues. + +| Property | Description | +|: |: | +| `yarn.resourcemanager.scheduler.monitor.enable` | Enabling monitoring is necessary for the periodic refresh. Default value is false. | +| `yarn.resourcemanager.scheduler.monitor.policies` | This is a configuration property that holds a list of classes. Adding more classes means more monitor tasks will be launched, Add `org.apache.hadoop.yarn.server.resourcemanager.capacity.QueueConfigurationAutoRefreshPolicy` to the policies list to enable the periodic refresh. Default value of this property is `org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy`, it means the preemption f [...] +| `yarn.resourcemanager.queue.auto.refresh.monitoring-interval` | Adjusting the auto-refresh monitoring interval is possible with this configuration property. The value is in milliseconds. The default value is 5000 (5 seconds). | ### Changing queue configuration via API Editing by API uses a backing store for the scheduler configuration. To enable this, the following parameters can be configured in yarn-site.xml. - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10522. Document for Flexible Auto Queue Creation in Capacity Scheduler
This is an automated email from the ASF dual-hosted git repository. snemeth pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new d9cb698 YARN-10522. Document for Flexible Auto Queue Creation in Capacity Scheduler d9cb698 is described below commit d9cb69853b51e33d379e29df616689ce9e20 Author: Benjamin Teke AuthorDate: Wed Sep 8 15:43:57 2021 +0200 YARN-10522. Document for Flexible Auto Queue Creation in Capacity Scheduler --- .../src/site/markdown/CapacityScheduler.md | 72 +++--- 1 file changed, 65 insertions(+), 7 deletions(-) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md index 7e4c3bd..254c51a 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md @@ -123,17 +123,25 @@ Configuration | Property | Description | |: |: | -| `yarn.scheduler.capacity..capacity` | Queue *capacity* in percentage (%) as a float (e.g. 12.5) OR as absolute resource queue minimum capacity. The sum of capacities for all queues, at each level, must be equal to 100. However if absolute resource is configured, sum of absolute resources of child queues could be less than it's parent absolute resource capacity. Applications in the queue may consume more resources than the queue's capacity if there are free resources, provid [...] -| `yarn.scheduler.capacity..maximum-capacity` | Maximum queue capacity in percentage (%) as a float OR as absolute resource queue maximum capacity. This limits the *elasticity* for applications in the queue. 1) Value is between 0 and 100. 2) Admin needs to make sure absolute maximum capacity >= absolute capacity for each queue. Also, setting this value to -1 sets maximum capacity to 100%. | +| `yarn.scheduler.capacity..capacity` | Queue *capacity* in percentage (%) as a float (e.g. 12.5), weight as a float with the postfix *w* (e.g. 2.0w) or as absolute resource queue minimum capacity. When using percentage values the sum of capacities for all queues, at each level, must be equal to 100. If absolute resource is configured, sum of absolute resources of child queues could be less than its parent absolute resource capacity. Applications in the queue may consume more [...] +| `yarn.scheduler.capacity..maximum-capacity` | Maximum queue capacity in percentage (%) as a float (when the *capacity* property is defined with either percentages or weights) or as absolute resource queue maximum capacity. This limits the *elasticity* for applications in the queue. 1) Value is between 0 and 100. 2) Admin needs to make sure absolute maximum capacity >= absolute capacity for each queue. Also, setting this value to -1 sets maximum capacity to 100%. | | `yarn.scheduler.capacity..minimum-user-limit-percent` | Each queue enforces a limit on the percentage of resources allocated to a user at any given time, if there is demand for resources. The user limit can vary between a minimum and maximum value. The former (the minimum value) is set to this property value and the latter (the maximum value) depends on the number of users who have submitted applications. For e.g., suppose the value of this property is 25. If two users have [...] | `yarn.scheduler.capacity..user-limit-factor` | User limit factor provides a way to control the max amount of resources that a single user can consume. It is the multiple of the queue's capacity. By default this is set to 1 which ensures that a single user can never take more than the queue's configured capacity irrespective of how idle the cluster is. Increasing it means a single user can use more than the minimum capacity of the cluster, while decreasing it results in lowe [...] | `yarn.scheduler.capacity..maximum-allocation-mb` | The per queue maximum limit of memory to allocate to each container request at the Resource Manager. This setting overrides the cluster configuration `yarn.scheduler.maximum-allocation-mb`. This value must be smaller than or equal to the cluster maximum. | | `yarn.scheduler.capacity..maximum-allocation-vcores` | The per queue maximum limit of virtual cores to allocate to each container request at the Resource Manager. This setting overrides the cluster configuration `yarn.scheduler.maximum-allocation-vcores`. This value must be smaller than or equal to the cluster maximum. | | `yarn.scheduler.capacity..user-settings..weight` | This floating point value is used when calculating the user limit resource values for users in a queue. This value will weight each user more or less than the other users in the queue. For example, if user A should receiv
[hadoop] branch trunk updated: YARN-10576. Update Capacity Scheduler documentation with JSON-based placement mapping. Contributed by Benjamin Teke
This is an automated email from the ASF dual-hosted git repository. snemeth pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 9c8fe1e YARN-10576. Update Capacity Scheduler documentation with JSON-based placement mapping. Contributed by Benjamin Teke 9c8fe1e is described below commit 9c8fe1e512df62be5dc994f07951c5c6d03690f3 Author: Szilard Nemeth AuthorDate: Wed Sep 8 15:17:27 2021 +0200 YARN-10576. Update Capacity Scheduler documentation with JSON-based placement mapping. Contributed by Benjamin Teke --- .../src/site/markdown/CapacityScheduler.md | 51 +- 1 file changed, 40 insertions(+), 11 deletions(-) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md index ebec79f..7e4c3bd 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md @@ -261,7 +261,7 @@ Below example covers single mapping separately. In case of multiple mappings wit In order to make the queue mapping feature more versatile, a new format and evaluation engine has been added to Capacity Scheduler. The new engine is fully backwards compatible with the old one and adds several new features. Note that it can also parse the old format, but the new features are only available if you specify the mappings in JSON. - * Syntax +Syntax Based on the current JSON schema, users can define mapping rules the following way: @@ -288,7 +288,27 @@ Below example covers single mapping separately. In case of multiple mappings wit Rules are evaluated from top to bottom. Compared to the legacy mapping rule evaluator, it can be adjusted more flexibly what happens when the evaluation stops and a given rule does not match. - * Rules +How to enable JSON-based queue mapping + +The following properties control how the new placement engine expects rules. + +| Setting | Description | +|: |: | +| `yarn.scheduler.capacity.mapping-rule-format` | Allowed values are `legacy` or `json`. If it is not set, then the engine assumes that the old format might be in use so it also checks the value of `yarn.scheduler.capacity.queue-mappings`. Therefore, this must be set to `json` and cannot be left empty. | +| `yarn.scheduler.capacity.mapping-rule-json` | The value of this property should contain the entire chain of rules inline. This is the preferred way of configuring Capacity Scheduler if you use the Mutation API, ie. modify configuration real-time via the REST interface. | +| `yarn.scheduler.capacity.mapping-rule-json-file` | Defines an absolute path to a JSON file which contains the rules. For example, `/opt/hadoop/config/mapping-rules.json`. | + +The property `yarn.scheduler.capacity.mapping-rule-json` takes precedence over `yarn.scheduler.capacity.mapping-rule-json-file`. If the format is set to `json` but you don't define either of these, then you'll get a warning but the initialization of Capacity Scheduler will not fail. + +Differences between legacy and flexible queue auto-creation modes + +To use the flexible Queue Auto-Creation under a parent the queue capacities must be configured with weights. The flexible mode gives the user much more freedom to automatically create new leaf queues or entire queue hierarchies based on mapping rules. "Legacy" mode refers to either percentage-based configuration or where capacities are defined with absolute resources. + +In flexible Queue Auto-Creation mode, every parent queue can have dynamically created parent or leaf queues (if the `yarn.scheduler.capacity..auto-queue-creation-v2.enabled` property is set to true), even if it already has static child queues. This also means that certain settings influence the outcome of the queue placement depending on how the scheduler is configured. + +When the mode is relevant, the document explains how certain settings or flags affect the overall logic. + +Rules Each mapping rule can have the following settings: @@ -298,21 +318,28 @@ Rules are evaluated from top to bottom. Compared to the legacy mapping rule eval | `matches` | The string to match, or an asterisk "*" which means "all". For example, if the type is `user` and this string is "hadoop" then the rule will only be evaluated if the submitter user is "hadoop". The "*" does not work with groups. | | `policy` | Selects a list of pre-defined policies which defines where the application should be placed. This will be explained later in the "Policies" section. | | `parentQueue` | In case of `user`, `primaryGroup`, `primaryGroupUser`, `
[hadoop] branch trunk updated: YARN-10891. Extend QueueInfo with max-parallel-apps in CS. (#3314)
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 16e6030 YARN-10891. Extend QueueInfo with max-parallel-apps in CS.
(#3314)
16e6030 is described below
commit 16e6030e25a6607b9886d01f9174984b1e71fb38
Author: Tamas Domok
AuthorDate: Fri Aug 27 23:09:54 2021 +0200
YARN-10891. Extend QueueInfo with max-parallel-apps in CS. (#3314)
Co-authored-by: Tamas Domok
---
.../apache/hadoop/yarn/api/records/QueueInfo.java | 24 +-
.../src/main/proto/yarn_protos.proto | 1 +
.../apache/hadoop/yarn/client/cli/QueueCLI.java| 2 ++
.../hadoop/yarn/client/ProtocolHATestBase.java | 2 +-
.../apache/hadoop/yarn/client/cli/TestYarnCLI.java | 8 +---
.../yarn/api/records/impl/pb/QueueInfoPBImpl.java | 12 +++
.../apache/hadoop/yarn/api/TestPBImplRecords.java | 2 +-
.../scheduler/capacity/AbstractCSQueue.java| 2 ++
.../scheduler/capacity/CSQueue.java| 6 ++
.../webapp/dao/CapacitySchedulerQueueInfo.java | 6 ++
.../TestConfigurationMutationACLPolicies.java | 2 +-
.../scheduler/TestSchedulerApplicationAttempt.java | 2 +-
.../scheduler/capacity/TestLeafQueue.java | 2 +-
.../webapp/TestRMWebServicesCapacitySched.java | 14 ++---
.../src/site/markdown/ResourceManagerRest.md | 19 +
15 files changed, 88 insertions(+), 16 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java
index eb703cd..707554c 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java
@@ -60,7 +60,8 @@ public abstract class QueueInfo {
List childQueues, List applications,
QueueState queueState, Set accessibleNodeLabels,
String defaultNodeLabelExpression, QueueStatistics queueStatistics,
- boolean preemptionDisabled, float weight) {
+ boolean preemptionDisabled, float weight,
+ int maxParallelApps) {
QueueInfo queueInfo = Records.newRecord(QueueInfo.class);
queueInfo.setQueueName(queueName);
queueInfo.setQueuePath(queuePath);
@@ -75,6 +76,7 @@ public abstract class QueueInfo {
queueInfo.setQueueStatistics(queueStatistics);
queueInfo.setPreemptionDisabled(preemptionDisabled);
queueInfo.setWeight(weight);
+queueInfo.setMaxParallelApps(maxParallelApps);
return queueInfo;
}
@@ -86,14 +88,14 @@ public abstract class QueueInfo {
List childQueues, List applications,
QueueState queueState, Set accessibleNodeLabels,
String defaultNodeLabelExpression, QueueStatistics queueStatistics,
- boolean preemptionDisabled, float weight,
+ boolean preemptionDisabled, float weight, int maxParallelApps,
Map queueConfigurations) {
QueueInfo queueInfo = QueueInfo.newInstance(queueName, queuePath, capacity,
maximumCapacity, currentCapacity,
childQueues, applications,
queueState, accessibleNodeLabels,
defaultNodeLabelExpression, queueStatistics,
-preemptionDisabled, weight);
+preemptionDisabled, weight, maxParallelApps);
queueInfo.setQueueConfigurations(queueConfigurations);
return queueInfo;
}
@@ -106,7 +108,7 @@ public abstract class QueueInfo {
List childQueues, List applications,
QueueState queueState, Set accessibleNodeLabels,
String defaultNodeLabelExpression, QueueStatistics queueStatistics,
- boolean preemptionDisabled, float weight,
+ boolean preemptionDisabled, float weight, int maxParallelApps,
Map queueConfigurations,
boolean intraQueuePreemptionDisabled) {
QueueInfo queueInfo = QueueInfo.newInstance(queueName, queuePath, capacity,
@@ -114,7 +116,7 @@ public abstract class QueueInfo {
childQueues, applications,
queueState, accessibleNodeLabels,
defaultNodeLabelExpression, queueStatistics,
-preemptionDisabled, weight, queueConfigurations);
+preemptionDisabled, weight, maxParallelApps, queueConfigurations);
queueInfo.setIntraQueuePreemptionDisabled(intraQueuePreemptionDisabled);
return queueInfo;
}
@@ -166,6 +168,18 @@ public abstract class QueueInfo {
@Private
@Unstable
public abstract void setWeight(float weight);
+
+ /**
+ * Get the configured max parallel apps of the queue.
+ * @return configured max parallel apps of the queue
+ */
+ @Public
+ @Stable
+ public abstract int getMaxParallelApps();
+
+ @Private
+ @Unstable
[hadoop] branch trunk updated: YARN-10838. Implement an optimised version of Configuration getPropsWithPrefix. Contributed by Andras Gyori, Benjamin Teke
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new e06a5cb YARN-10838. Implement an optimised version of Configuration
getPropsWithPrefix. Contributed by Andras Gyori, Benjamin Teke
e06a5cb is described below
commit e06a5cb197829ea821b490d02334d95d91bcb6d3
Author: Szilard Nemeth
AuthorDate: Tue Aug 24 15:27:34 2021 +0200
YARN-10838. Implement an optimised version of Configuration
getPropsWithPrefix. Contributed by Andras Gyori, Benjamin Teke
---
.../capacity/CapacitySchedulerConfiguration.java | 24 +++
.../capacity/ConfigurationProperties.java | 214 +
.../capacity/TestConfigurationProperties.java | 114 +++
3 files changed, 352 insertions(+)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
index 5a5304e..84302e5 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
@@ -406,6 +406,7 @@ public class CapacitySchedulerConfiguration extends
ReservationSchedulerConfigur
public static final String MAPPING_RULE_FORMAT_DEFAULT =
MAPPING_RULE_FORMAT_LEGACY;
+ private ConfigurationProperties configurationProperties;
/**
* Different resource types supported.
@@ -1048,6 +1049,29 @@ public class CapacitySchedulerConfiguration extends
ReservationSchedulerConfigur
set(queuePrefix + MAXIMUM_ALLOCATION, maximumAllocation);
}
+ /**
+ * Get all configuration properties parsed in a
+ * {@code ConfigurationProperties} object.
+ * @return configuration properties
+ */
+ public ConfigurationProperties getConfigurationProperties() {
+if (configurationProperties == null) {
+ reinitializeConfigurationProperties();
+}
+
+return configurationProperties;
+ }
+
+ /**
+ * Reinitializes the cached {@code ConfigurationProperties} object.
+ */
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ public void reinitializeConfigurationProperties() {
+// Props are always Strings, therefore this cast is safe
+Map props = (Map) getProps();
+configurationProperties = new ConfigurationProperties(props);
+ }
+
public long getQueueMaximumAllocationMb(String queue) {
String queuePrefix = getQueuePrefix(queue);
return getInt(queuePrefix + MAXIMUM_ALLOCATION_MB, (int)UNDEFINED);
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ConfigurationProperties.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ConfigurationProperties.java
new file mode 100644
index 000..fa05b9f
--- /dev/null
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ConfigurationProperties.java
@@ -0,0 +1,214 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * A trie stora
[hadoop] branch branch-3.3 updated: YARN-10814. Fallback to RandomSecretProvider if the secret file is empty. Contributed by Tamas Domok
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new 224b421 YARN-10814. Fallback to RandomSecretProvider if the secret
file is empty. Contributed by Tamas Domok
224b421 is described below
commit 224b42108dc168a896e53a04cfc1bff7282ffe9f
Author: Szilard Nemeth
AuthorDate: Tue Aug 24 14:16:15 2021 +0200
YARN-10814. Fallback to RandomSecretProvider if the secret file is empty.
Contributed by Tamas Domok
---
.../server/AuthenticationFilter.java | 4 ++--
.../util/FileSignerSecretProvider.java | 27 +
.../server/TestAuthenticationFilter.java | 28 ++
.../util/TestFileSignerSecretProvider.java | 27 +
4 files changed, 68 insertions(+), 18 deletions(-)
diff --git
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java
index b339a5d..3658bd8 100644
---
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java
+++
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java
@@ -237,8 +237,8 @@ public class AuthenticationFilter implements Filter {
provider.init(config, ctx, validity);
} catch (Exception e) {
if (!disallowFallbackToRandomSecretProvider) {
- LOG.info("Unable to initialize FileSignerSecretProvider, " +
- "falling back to use random secrets.");
+ LOG.warn("Unable to initialize FileSignerSecretProvider, " +
+ "falling back to use random secrets. Reason: " + e.getMessage());
provider = new RandomSignerSecretProvider();
provider.init(config, ctx, validity);
} else {
diff --git
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java
index c037037..2a8a712 100644
---
a/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java
+++
b/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java
@@ -13,15 +13,15 @@
*/
package org.apache.hadoop.security.authentication.util;
-import org.apache.hadoop.thirdparty.com.google.common.base.Charsets;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
import org.apache.hadoop.security.authentication.server.AuthenticationFilter;
-import org.apache.hadoop.security.authentication.util.SignerSecretProvider;
import javax.servlet.ServletContext;
import java.io.*;
-import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
import java.util.Properties;
/**
@@ -43,29 +43,24 @@ public class FileSignerSecretProvider extends
SignerSecretProvider {
String signatureSecretFile = config.getProperty(
AuthenticationFilter.SIGNATURE_SECRET_FILE, null);
-Reader reader = null;
if (signatureSecretFile != null) {
- try {
+ try (Reader reader = new InputStreamReader(Files.newInputStream(
+ Paths.get(signatureSecretFile)), StandardCharsets.UTF_8)) {
StringBuilder sb = new StringBuilder();
-reader = new InputStreamReader(
-new FileInputStream(signatureSecretFile), Charsets.UTF_8);
int c = reader.read();
while (c > -1) {
sb.append((char) c);
c = reader.read();
}
-secret = sb.toString().getBytes(Charset.forName("UTF-8"));
+
+secret = sb.toString().getBytes(StandardCharsets.UTF_8);
+if (secret.length == 0) {
+ throw new RuntimeException("No secret in signature secret file: "
+ + signatureSecretFile);
+}
} catch (IOException ex) {
throw new RuntimeException("Could not read signature secret file: " +
signatureSecretFile);
- } finally {
-if (reader != null) {
- try {
-reader.close();
- } catch (IOException e) {
-// nothing to do
- }
-}
}
}
diff --git
a/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestAuthenticationFilter.java
b/hadoop-common-project/h
[hadoop] branch branch-3.1 updated: YARN-10882. Fix branch-3.1 build: zstd library is missing from the Dockerfile (#3297)
This is an automated email from the ASF dual-hosted git repository. snemeth pushed a commit to branch branch-3.1 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.1 by this push: new f7ac747 YARN-10882. Fix branch-3.1 build: zstd library is missing from the Dockerfile (#3297) f7ac747 is described below commit f7ac747b6a78c93c8a97dfadc55cbda86680aff4 Author: Tamas Domok AuthorDate: Wed Aug 11 18:29:00 2021 +0200 YARN-10882. Fix branch-3.1 build: zstd library is missing from the Dockerfile (#3297) Co-authored-by: Tamas Domok --- dev-support/docker/Dockerfile | 1 + 1 file changed, 1 insertion(+) diff --git a/dev-support/docker/Dockerfile b/dev-support/docker/Dockerfile index 14f6324..e6ebeb1 100644 --- a/dev-support/docker/Dockerfile +++ b/dev-support/docker/Dockerfile @@ -66,6 +66,7 @@ RUN apt-get -q update \ libssl-dev \ libsnappy-dev \ libtool \ +libzstd1-dev \ locales \ make \ pinentry-curses \ - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10874. Refactor NM ContainerLaunch#getEnvDependencies's unit tests (#3248)
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new a5811dd YARN-10874. Refactor NM ContainerLaunch#getEnvDependencies's
unit tests (#3248)
a5811dd is described below
commit a5811dda7bce9df41d967386cedde421da7af236
Author: Tamas Domok
AuthorDate: Tue Aug 3 16:19:04 2021 +0200
YARN-10874. Refactor NM ContainerLaunch#getEnvDependencies's unit tests
(#3248)
Co-authored-by: Tamas Domok
---
hadoop-project/pom.xml | 6 +
.../hadoop-yarn-server-nodemanager/pom.xml | 5 +
.../launcher/TestContainerLaunch.java | 139 ---
.../launcher/TestContainerLaunchParameterized.java | 149 +
4 files changed, 160 insertions(+), 139 deletions(-)
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 9ce429e..48abf2f 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -1047,6 +1047,12 @@
test
+org.junit.jupiter
+junit-jupiter-params
+${junit.jupiter.version}
+test
+
+
org.junit.vintage
junit-vintage-engine
${junit.vintage.version}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
index 34741e4..6f84bfc 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
@@ -128,6 +128,11 @@
test
+ org.junit.jupiter
+ junit-jupiter-params
+ test
+
+
org.mockito
mockito-core
test
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
index 365e2bb..6b0732b 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
@@ -2201,145 +2201,6 @@ public class TestContainerLaunch extends
BaseContainerManagerTest {
icC<0 || icA<0 || icA expected = new HashSet<>();
-final ContainerLaunch.ShellScriptBuilder bash =
-ContainerLaunch.ShellScriptBuilder.create(Shell.OSType.OS_TYPE_LINUX);
-String s;
-
-s = null;
-Assert.assertEquals("failed to parse " + s, expected,
-bash.getEnvDependencies(s));
-s = "";
-Assert.assertEquals("failed to parse " + s, expected,
-bash.getEnvDependencies(s));
-s = "A";
-Assert.assertEquals("failed to parse " + s, expected,
-bash.getEnvDependencies(s));
-s = "\\$A";
-Assert.assertEquals("failed to parse " + s, expected,
-bash.getEnvDependencies(s));
-s = "$$";
-Assert.assertEquals("failed to parse " + s, expected,
-bash.getEnvDependencies(s));
-s = "$1";
-Assert.assertEquals("failed to parse " + s, expected,
-bash.getEnvDependencies(s));
-s = "handle \"'$A'\" simple quotes";
-Assert.assertEquals("failed to parse " + s, expected,
-bash.getEnvDependencies(s));
-s = "$ crash test for StringArrayOutOfBoundException";
-Assert.assertEquals("failed to parse " + s, expected,
-bash.getEnvDependencies(s));
-s = "${ crash test for StringArrayOutOfBoundException";
-Assert.assertEquals("failed to parse " + s, expected,
-bash.getEnvDependencies(s));
-s = "${# crash test for StringArrayOutOfBoundException";
-Assert.assertEquals("failed to parse " + s, expected,
-bash.getEnvDependencies(s));
-s = "crash test for StringArrayOutOfBoundException $";
-Assert.assertEquals("failed to parse " + s, expected,
-bash.getEnvDependencies(s));
-s = "crash test for StringArrayOutOfBoundException ${";
-Assert.assertEquals("failed to parse " + s, expected,
-bash.getEnvDependencies(s));
-s = "crash test for StringAr
[hadoop] branch branch-3.1 updated: YARN-6221. Entities missing from ATS when summary log file info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new ac30549 YARN-6221. Entities missing from ATS when summary log file
info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang
ac30549 is described below
commit ac30549b06866dfab61eb86169841d4cdbf50d66
Author: Szilard Nemeth
AuthorDate: Sat Jul 31 14:55:43 2021 +0200
YARN-6221. Entities missing from ATS when summary log file info got
returned to the ATS before the domain log. Contributed by Xiaomin Zhang
---
.../timeline/EntityGroupFSTimelineStore.java | 3 ++-
.../timeline/TestEntityGroupFSTimelineStore.java | 25 ++
2 files changed, 27 insertions(+), 1 deletion(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java
index 80baf89..fd8c57f 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java
@@ -780,10 +780,11 @@ public class EntityGroupFSTimelineStore extends
CompositeService
LogInfo log;
if (isDomainLog) {
log = new DomainLogInfo(attemptDirName, filename, owner);
+summaryLogs.add(0, log);
} else {
log = new EntityLogInfo(attemptDirName, filename, owner);
+summaryLogs.add(log);
}
- summaryLogs.add(log);
}
private synchronized void addDetailLog(String attemptDirName,
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/TestEntityGroupFSTimelineStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/TestEntityGroupFSTimelineStore.java
index 61da3c8..13224d1 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/TestEntityGroupFSTimelineStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/TestEntityGroupFSTimelineStore.java
@@ -225,6 +225,31 @@ public class TestEntityGroupFSTimelineStore extends
TimelineStoreTestUtils {
}
@Test
+ public void testAppLogsDomainLogLastlyScanned() throws Exception {
+EntityGroupFSTimelineStore.AppLogs appLogs =
+store.new AppLogs(mainTestAppId, mainTestAppDirPath,
+AppState.COMPLETED);
+Path attemptDirPath = new Path(new Path(testActiveDirPath,
+mainTestAppId.toString()),
+getAttemptDirName(mainTestAppId));
+//Delete the domain log from AppDirPath so first scan won't find it
+fs.delete(new Path(attemptDirPath, TEST_DOMAIN_LOG_FILE_NAME), false);
+appLogs.scanForLogs();
+List summaryLogs = appLogs.getSummaryLogs();
+assertEquals(1, summaryLogs.size());
+assertEquals(TEST_SUMMARY_LOG_FILE_NAME, summaryLogs.get(0).getFilename());
+
+//Generate the domain log
+FSDataOutputStream out = fs.create(
+new Path(attemptDirPath, TEST_DOMAIN_LOG_FILE_NAME));
+out.close();
+
+appLogs.scanForLogs();
+assertEquals(2, summaryLogs.size());
+assertEquals(TEST_DOMAIN_LOG_FILE_NAME, summaryLogs.get(0).getFilename());
+ }
+
+ @Test
public void testMoveToDone() throws Exception {
EntityGroupFSTimelineStore.AppLogs appLogs =
store.new AppLogs(mainTestAppId, mainTestAppDirPath,
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.2 updated: YARN-6221. Entities missing from ATS when summary log file info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new e4df745 YARN-6221. Entities missing from ATS when summary log file
info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang
e4df745 is described below
commit e4df745c139ba4540c73ffcf78c86387951c9d95
Author: Szilard Nemeth
AuthorDate: Sat Jul 31 14:54:51 2021 +0200
YARN-6221. Entities missing from ATS when summary log file info got
returned to the ATS before the domain log. Contributed by Xiaomin Zhang
---
.../timeline/EntityGroupFSTimelineStore.java | 3 ++-
.../timeline/TestEntityGroupFSTimelineStore.java | 25 ++
2 files changed, 27 insertions(+), 1 deletion(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java
index f8f0aed..5c2d399 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java
@@ -774,10 +774,11 @@ public class EntityGroupFSTimelineStore extends
CompositeService
LogInfo log;
if (isDomainLog) {
log = new DomainLogInfo(attemptDirName, filename, owner);
+summaryLogs.add(0, log);
} else {
log = new EntityLogInfo(attemptDirName, filename, owner);
+summaryLogs.add(log);
}
- summaryLogs.add(log);
}
private synchronized void addDetailLog(String attemptDirName,
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/TestEntityGroupFSTimelineStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/TestEntityGroupFSTimelineStore.java
index 61da3c8..13224d1 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/TestEntityGroupFSTimelineStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/TestEntityGroupFSTimelineStore.java
@@ -225,6 +225,31 @@ public class TestEntityGroupFSTimelineStore extends
TimelineStoreTestUtils {
}
@Test
+ public void testAppLogsDomainLogLastlyScanned() throws Exception {
+EntityGroupFSTimelineStore.AppLogs appLogs =
+store.new AppLogs(mainTestAppId, mainTestAppDirPath,
+AppState.COMPLETED);
+Path attemptDirPath = new Path(new Path(testActiveDirPath,
+mainTestAppId.toString()),
+getAttemptDirName(mainTestAppId));
+//Delete the domain log from AppDirPath so first scan won't find it
+fs.delete(new Path(attemptDirPath, TEST_DOMAIN_LOG_FILE_NAME), false);
+appLogs.scanForLogs();
+List summaryLogs = appLogs.getSummaryLogs();
+assertEquals(1, summaryLogs.size());
+assertEquals(TEST_SUMMARY_LOG_FILE_NAME, summaryLogs.get(0).getFilename());
+
+//Generate the domain log
+FSDataOutputStream out = fs.create(
+new Path(attemptDirPath, TEST_DOMAIN_LOG_FILE_NAME));
+out.close();
+
+appLogs.scanForLogs();
+assertEquals(2, summaryLogs.size());
+assertEquals(TEST_DOMAIN_LOG_FILE_NAME, summaryLogs.get(0).getFilename());
+ }
+
+ @Test
public void testMoveToDone() throws Exception {
EntityGroupFSTimelineStore.AppLogs appLogs =
store.new AppLogs(mainTestAppId, mainTestAppDirPath,
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.3 updated: YARN-6221. Entities missing from ATS when summary log file info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new 0a0628b YARN-6221. Entities missing from ATS when summary log file
info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang
0a0628b is described below
commit 0a0628b1cbaab3fb0a1c71781b77dd2c91ed0c73
Author: Szilard Nemeth
AuthorDate: Sat Jul 31 14:53:30 2021 +0200
YARN-6221. Entities missing from ATS when summary log file info got
returned to the ATS before the domain log. Contributed by Xiaomin Zhang
---
.../timeline/EntityGroupFSTimelineStore.java | 3 ++-
.../timeline/TestEntityGroupFSTimelineStore.java | 25 ++
2 files changed, 27 insertions(+), 1 deletion(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java
index e9957c8..c2992b4 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java
@@ -808,10 +808,11 @@ public class EntityGroupFSTimelineStore extends
CompositeService
LogInfo log;
if (isDomainLog) {
log = new DomainLogInfo(attemptDirName, filename, owner);
+summaryLogs.add(0, log);
} else {
log = new EntityLogInfo(attemptDirName, filename, owner);
+summaryLogs.add(log);
}
- summaryLogs.add(log);
}
private synchronized void addDetailLog(String attemptDirName,
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/TestEntityGroupFSTimelineStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/TestEntityGroupFSTimelineStore.java
index 984e157..7302ae1 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/TestEntityGroupFSTimelineStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/TestEntityGroupFSTimelineStore.java
@@ -227,6 +227,31 @@ public class TestEntityGroupFSTimelineStore extends
TimelineStoreTestUtils {
}
@Test
+ public void testAppLogsDomainLogLastlyScanned() throws Exception {
+EntityGroupFSTimelineStore.AppLogs appLogs =
+store.new AppLogs(mainTestAppId, mainTestAppDirPath,
+AppState.COMPLETED);
+Path attemptDirPath = new Path(new Path(testActiveDirPath,
+mainTestAppId.toString()),
+getAttemptDirName(mainTestAppId));
+//Delete the domain log from AppDirPath so first scan won't find it
+fs.delete(new Path(attemptDirPath, TEST_DOMAIN_LOG_FILE_NAME), false);
+appLogs.scanForLogs();
+List summaryLogs = appLogs.getSummaryLogs();
+assertEquals(1, summaryLogs.size());
+assertEquals(TEST_SUMMARY_LOG_FILE_NAME, summaryLogs.get(0).getFilename());
+
+//Generate the domain log
+FSDataOutputStream out = fs.create(
+new Path(attemptDirPath, TEST_DOMAIN_LOG_FILE_NAME));
+out.close();
+
+appLogs.scanForLogs();
+assertEquals(2, summaryLogs.size());
+assertEquals(TEST_DOMAIN_LOG_FILE_NAME, summaryLogs.get(0).getFilename());
+ }
+
+ @Test
public void testMoveToDone() throws Exception {
EntityGroupFSTimelineStore.AppLogs appLogs =
store.new AppLogs(mainTestAppId, mainTestAppDirPath,
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.2 updated: YARN-10789. RM HA startup can fail due to race conditions in ZKConfigurationStore. Contributed by Tarun Parimi
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 7a8b626 YARN-10789. RM HA startup can fail due to race conditions in
ZKConfigurationStore. Contributed by Tarun Parimi
7a8b626 is described below
commit 7a8b6265c626c45d3af099723f0b1b9b76dd5cb4
Author: Szilard Nemeth
AuthorDate: Thu Jul 29 19:22:57 2021 +0200
YARN-10789. RM HA startup can fail due to race conditions in
ZKConfigurationStore. Contributed by Tarun Parimi
---
.../capacity/conf/ZKConfigurationStore.java| 25 +-
1 file changed, 20 insertions(+), 5 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java
index 3742c36..6f3612c 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java
@@ -19,6 +19,7 @@
package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf;
import com.google.common.annotations.VisibleForTesting;
+import org.apache.zookeeper.KeeperException.NodeExistsException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.hadoop.conf.Configuration;
@@ -62,7 +63,8 @@ public class ZKConfigurationStore extends
YarnConfigurationStore {
private static final String CONF_STORE_PATH = "CONF_STORE";
private static final String FENCING_PATH = "FENCING";
private static final String CONF_VERSION_PATH = "CONF_VERSION";
-
+ private static final String NODEEXISTS_MSG = "Encountered NodeExists error."
+ + " Skipping znode creation since another RM has already created it";
private String zkVersionPath;
private String logsPath;
private String confStorePath;
@@ -93,7 +95,11 @@ public class ZKConfigurationStore extends
YarnConfigurationStore {
this.fencingNodePath = getNodePath(znodeParentPath, FENCING_PATH);
this.confVersionPath = getNodePath(znodeParentPath, CONF_VERSION_PATH);
-zkManager.createRootDirRecursively(znodeParentPath, zkAcl);
+try {
+ zkManager.createRootDirRecursively(znodeParentPath, zkAcl);
+} catch(NodeExistsException e) {
+ LOG.warn(NODEEXISTS_MSG, e);
+}
zkManager.delete(fencingNodePath);
if (createNewZkPath(logsPath)) {
@@ -247,7 +253,12 @@ public class ZKConfigurationStore extends
YarnConfigurationStore {
*/
private boolean createNewZkPath(String path) throws Exception {
if (!zkManager.exists(path)) {
- zkManager.create(path);
+ try {
+zkManager.create(path);
+ } catch(NodeExistsException e) {
+LOG.warn(NODEEXISTS_MSG, e);
+return false;
+ }
return true;
} else {
return false;
@@ -282,8 +293,12 @@ public class ZKConfigurationStore extends
YarnConfigurationStore {
@VisibleForTesting
protected void safeCreateZkData(String path, byte[] data) throws Exception {
-zkManager.safeCreate(path, data, zkAcl, CreateMode.PERSISTENT,
-zkAcl, fencingNodePath);
+try {
+ zkManager.safeCreate(path, data, zkAcl, CreateMode.PERSISTENT,
+ zkAcl, fencingNodePath);
+} catch(NodeExistsException e) {
+ LOG.warn(NODEEXISTS_MSG, e);
+}
}
private static String getNodePath(String root, String nodeName) {
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.3 updated: YARN-10789. RM HA startup can fail due to race conditions in ZKConfigurationStore. Contributed by Tarun Parimi
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new a272adc YARN-10789. RM HA startup can fail due to race conditions in
ZKConfigurationStore. Contributed by Tarun Parimi
a272adc is described below
commit a272adc5fad175157914f3aeb2d2cf6dceb8fa51
Author: Szilard Nemeth
AuthorDate: Thu Jul 29 19:21:58 2021 +0200
YARN-10789. RM HA startup can fail due to race conditions in
ZKConfigurationStore. Contributed by Tarun Parimi
---
.../capacity/conf/ZKConfigurationStore.java| 25 +-
1 file changed, 20 insertions(+), 5 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java
index 7276d3d..703193b 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java
@@ -19,6 +19,7 @@
package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf;
import
org.apache.hadoop.thirdparty.com.google.common.annotations.VisibleForTesting;
+import org.apache.zookeeper.KeeperException.NodeExistsException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.hadoop.conf.Configuration;
@@ -61,7 +62,8 @@ public class ZKConfigurationStore extends
YarnConfigurationStore {
private static final String CONF_STORE_PATH = "CONF_STORE";
private static final String FENCING_PATH = "FENCING";
private static final String CONF_VERSION_PATH = "CONF_VERSION";
-
+ private static final String NODEEXISTS_MSG = "Encountered NodeExists error."
+ + " Skipping znode creation since another RM has already created it";
private String zkVersionPath;
private String logsPath;
private String confStorePath;
@@ -92,7 +94,11 @@ public class ZKConfigurationStore extends
YarnConfigurationStore {
this.fencingNodePath = getNodePath(znodeParentPath, FENCING_PATH);
this.confVersionPath = getNodePath(znodeParentPath, CONF_VERSION_PATH);
-zkManager.createRootDirRecursively(znodeParentPath, zkAcl);
+try {
+ zkManager.createRootDirRecursively(znodeParentPath, zkAcl);
+} catch(NodeExistsException e) {
+ LOG.warn(NODEEXISTS_MSG, e);
+}
zkManager.delete(fencingNodePath);
if (createNewZkPath(logsPath)) {
@@ -244,7 +250,12 @@ public class ZKConfigurationStore extends
YarnConfigurationStore {
*/
private boolean createNewZkPath(String path) throws Exception {
if (!zkManager.exists(path)) {
- zkManager.create(path);
+ try {
+zkManager.create(path);
+ } catch(NodeExistsException e) {
+LOG.warn(NODEEXISTS_MSG, e);
+return false;
+ }
return true;
} else {
return false;
@@ -279,8 +290,12 @@ public class ZKConfigurationStore extends
YarnConfigurationStore {
@VisibleForTesting
protected void safeCreateZkData(String path, byte[] data) throws Exception {
-zkManager.safeCreate(path, data, zkAcl, CreateMode.PERSISTENT,
-zkAcl, fencingNodePath);
+try {
+ zkManager.safeCreate(path, data, zkAcl, CreateMode.PERSISTENT,
+ zkAcl, fencingNodePath);
+} catch(NodeExistsException e) {
+ LOG.warn(NODEEXISTS_MSG, e);
+}
}
private static String getNodePath(String root, String nodeName) {
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10856. Prevent ATS v2 health check REST API call if the ATS service itself is disabled. (#3236)
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 8f750c5 YARN-10856. Prevent ATS v2 health check REST API call if the
ATS service itself is disabled. (#3236)
8f750c5 is described below
commit 8f750c54aaa8b07e958b204b372f591f47cad2fa
Author: Benjamin Teke
AuthorDate: Thu Jul 29 19:15:27 2021 +0200
YARN-10856. Prevent ATS v2 health check REST API call if the ATS service
itself is disabled. (#3236)
Co-authored-by: Benjamin Teke
---
.../src/main/webapp/app/initializers/loader.js | 8
.../src/main/webapp/app/routes/application.js | 18 --
.../src/main/webapp/app/routes/yarn-app/logs.js| 14 +-
3 files changed, 29 insertions(+), 11 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/initializers/loader.js
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/initializers/loader.js
index bf544ec..5a697ea 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/initializers/loader.js
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/initializers/loader.js
@@ -62,6 +62,10 @@ function getYarnHttpProtocolScheme(rmhost, application) {
return getConfigFromYarn(rmhost, application, 'yarn.http.policy');
}
+function getYarnTimelineEnabled(rmhost, application) {
+ return getConfigFromYarn(rmhost, application,
'yarn.timeline-service.enabled');
+}
+
function getTimeLineURL(rmhost, isHttpsSchemeEnabled) {
var url = window.location.protocol + '//' +
(ENV.hosts.localBaseAddress? ENV.hosts.localBaseAddress + '/' : '') +
rmhost;
@@ -192,6 +196,10 @@ function updateConfigs(application) {
Ember.Logger.log("NodeMananger port: " + nodeManagerPort);
ENV.nodeManagerPort = nodeManagerPort;
+ var timelineServiceEnabled = getYarnTimelineEnabled(rmhost, application);
+ Ember.Logger.log("Timeline Service enabled: " + timelineServiceEnabled);
+ ENV.timelineServiceEnabled = (timelineServiceEnabled.toLowerCase() ===
'true');
+
if (!ENV.hosts.jhsAddress) {
var jhsAddress = getJHSURL(rmhost, application, isHttpsSchemeEnabled);
jhsAddress = transformURL(jhsAddress, hostname);
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/routes/application.js
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/routes/application.js
index 93631e0..7c3b311 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/routes/application.js
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/routes/application.js
@@ -21,7 +21,7 @@ import AbstractRoute from './abstract';
export default AbstractRoute.extend({
model() {
-return Ember.RSVP.hash({
+let promises = {
clusterInfo: this.store.findAll('ClusterInfo', {reload:
true}).catch(function() {
return null;
}),
@@ -30,11 +30,15 @@ export default AbstractRoute.extend({
}),
jhsHealth: this.store.queryRecord('jhs-health', {}).catch(function() {
return null;
- }),
- timelineHealth: this.store.queryRecord('timeline-health',
{}).catch(function() {
-return null;
})
-});
+};
+
+if (ENV.timelineServiceEnabled) {
+ promises.timelineHealth = this.store.queryRecord('timeline-health',
{}).catch(function() {
+return null;
+ });
+}
+return Ember.RSVP.hash(promises);
},
actions: {
@@ -62,7 +66,9 @@ export default AbstractRoute.extend({
unloadAll: function() {
this.store.unloadAll('ClusterInfo');
this.store.unloadAll('cluster-user-info');
-this.store.unloadAll('timeline-health');
+if (ENV.timelineServiceEnabled) {
+ this.store.unloadAll('timeline-health');
+}
this.store.unloadAll('jhs-health');
},
});
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/routes/yarn-app/logs.js
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/routes/yarn-app/logs.js
index 55c52cd..888ab7c 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/routes/yarn-app/logs.js
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/routes/yarn-app/logs.js
@@ -25,7 +25,7 @@ export default AbstractRoute.extend(AppAttemptMixin, {
const { app_id } = this.paramsFor('yarn-app');
const { service } = param;
transition.send('updateBreadcrumbs', app_id, service, [{text: 'Logs'}]);
-return Ember.RSVP.hash({
+let promises = {
appId: app_id,
service
[hadoop] branch trunk updated (74770c8 -> ac0a4e7)
This is an automated email from the ASF dual-hosted git repository. snemeth pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git. from 74770c8 YARN-10663. Add runningApps stats in SLS. Contributed by Vadaga Ananyo Rao add ac0a4e7 YARN-10869. CS considers only the default maximum-allocation-mb/vcore property as a maximum when it creates dynamic queues (#3225) No new revisions were added by this update. Summary of changes: .../scheduler/capacity/AbstractCSQueue.java| 8 +- .../capacity/CapacitySchedulerConfiguration.java | 14 + .../TestCapacitySchedulerAutoCreatedQueueBase.java | 32 +++-- .../TestCapacitySchedulerAutoQueueCreation.java| 33 -- 4 files changed, 81 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10663. Add runningApps stats in SLS. Contributed by Vadaga Ananyo Rao
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 74770c8 YARN-10663. Add runningApps stats in SLS. Contributed by
Vadaga Ananyo Rao
74770c8 is described below
commit 74770c8a16ecd427d7d4e15b797bcb3ad068ff40
Author: Szilard Nemeth
AuthorDate: Thu Jul 29 17:37:40 2021 +0200
YARN-10663. Add runningApps stats in SLS. Contributed by Vadaga Ananyo Rao
---
hadoop-tools/hadoop-sls/pom.xml| 5 ++
.../hadoop/yarn/sls/appmaster/AMSimulator.java | 14 +++
.../hadoop/yarn/sls/appmaster/DAGAMSimulator.java | 3 +-
.../hadoop/yarn/sls/appmaster/MRAMSimulator.java | 6 +-
.../yarn/sls/appmaster/StreamAMSimulator.java | 3 +-
.../hadoop/yarn/sls/nodemanager/NMSimulator.java | 21 -
.../yarn/sls/resourcemanager/MockAMLauncher.java | 3 +-
.../hadoop/yarn/sls/appmaster/TestAMSimulator.java | 29 ++
.../yarn/sls/nodemanager/TestNMSimulator.java | 100 -
9 files changed, 176 insertions(+), 8 deletions(-)
diff --git a/hadoop-tools/hadoop-sls/pom.xml b/hadoop-tools/hadoop-sls/pom.xml
index 5bb5a41..208cbdf 100644
--- a/hadoop-tools/hadoop-sls/pom.xml
+++ b/hadoop-tools/hadoop-sls/pom.xml
@@ -73,6 +73,11 @@
com.fasterxml.jackson.core
jackson-databind
+
+ org.mockito
+ mockito-core
+ test
+
diff --git
a/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java
b/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java
index 1330e4d..5315eaa 100644
---
a/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java
+++
b/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java
@@ -25,7 +25,9 @@ import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Set;
import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ConcurrentSkipListSet;
import java.util.concurrent.LinkedBlockingQueue;
import org.apache.hadoop.classification.InterfaceAudience.Private;
@@ -50,6 +52,7 @@ import
org.apache.hadoop.yarn.api.records.ContainerLaunchContext;
import org.apache.hadoop.yarn.api.records.ExecutionType;
import org.apache.hadoop.yarn.api.records.ExecutionTypeRequest;
import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
+import org.apache.hadoop.yarn.api.records.NodeId;
import org.apache.hadoop.yarn.api.records.Priority;
import org.apache.hadoop.yarn.api.records.ReservationId;
import org.apache.hadoop.yarn.api.records.Resource;
@@ -118,6 +121,8 @@ public abstract class AMSimulator extends TaskRunner.Task {
private Map appIdToAMSim;
+ private Set ranNodes = new ConcurrentSkipListSet();
+
public AMSimulator() {
this.responseQueue = new LinkedBlockingQueue<>();
}
@@ -236,6 +241,11 @@ public abstract class AMSimulator extends TaskRunner.Task {
LOG.info("AM container is null");
}
+// Clear runningApps for ranNodes of this app
+for (NodeId nodeId : ranNodes) {
+ se.getNmMap().get(nodeId).finishApplication(getApplicationId());
+}
+
if (null == appAttemptId) {
// If appAttemptId == null, AM is not launched from RM's perspective, so
// it's unnecessary to finish am as well
@@ -497,4 +507,8 @@ public abstract class AMSimulator extends TaskRunner.Task {
public ApplicationAttemptId getApplicationAttemptId() {
return appAttemptId;
}
+
+ public Set getRanNodes() {
+return this.ranNodes;
+ }
}
diff --git
a/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/DAGAMSimulator.java
b/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/DAGAMSimulator.java
index f886a69..c67544e 100644
---
a/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/DAGAMSimulator.java
+++
b/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/DAGAMSimulator.java
@@ -189,7 +189,8 @@ public class DAGAMSimulator extends AMSimulator {
appId, container.getId());
assignedContainers.put(container.getId(), cs);
se.getNmMap().get(container.getNodeId())
- .addNewContainer(container, cs.getLifeTime());
+ .addNewContainer(container, cs.getLifeTime(), appId);
+ getRanNodes().add(container.getNodeId());
}
}
}
diff --git
a/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/MRAMSimulator.java
b/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/MRAMSimulator.java
index 586c671..184fdca 100644
---
a/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/MRAMSimulato
[hadoop] branch trunk updated: YARN-10628. Add node usage metrics in SLS. Contributed by Vadaga Ananyo Rao
This is an automated email from the ASF dual-hosted git repository. snemeth pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 54f9fff YARN-10628. Add node usage metrics in SLS. Contributed by Vadaga Ananyo Rao 54f9fff is described below commit 54f9fff218e548cca3a8f08d56df00ed59ae4262 Author: Szilard Nemeth AuthorDate: Thu Jul 29 13:43:40 2021 +0200 YARN-10628. Add node usage metrics in SLS. Contributed by Vadaga Ananyo Rao --- .../src/main/html/showSimulationTrace.html | 21 +++ .../src/main/html/simulate.html.template | 28 - .../yarn/sls/scheduler/SchedulerMetrics.java | 47 +++ .../hadoop/yarn/sls/utils/NodeUsageRanges.java | 68 ++ .../apache/hadoop/yarn/sls/utils/package-info.java | 21 +++ .../org/apache/hadoop/yarn/sls/web/SLSWebApp.java | 33 +++ 6 files changed, 215 insertions(+), 3 deletions(-) diff --git a/hadoop-tools/hadoop-sls/src/main/html/showSimulationTrace.html b/hadoop-tools/hadoop-sls/src/main/html/showSimulationTrace.html index 102fdcf..e5dbc03 100644 --- a/hadoop-tools/hadoop-sls/src/main/html/showSimulationTrace.html +++ b/hadoop-tools/hadoop-sls/src/main/html/showSimulationTrace.html @@ -100,6 +100,10 @@ + + + + Â
[hadoop] branch branch-3.1 updated: YARN-9551. TestTimelineClientV2Impl.testSyncCall fails intermittent (#3212)
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 3f14a9b YARN-9551. TestTimelineClientV2Impl.testSyncCall fails
intermittent (#3212)
3f14a9b is described below
commit 3f14a9b21f9aead9a2d14a3bcd966a375301ba2f
Author: 9uapaw
AuthorDate: Wed Jul 28 04:35:20 2021 +0200
YARN-9551. TestTimelineClientV2Impl.testSyncCall fails intermittent (#3212)
Co-authored-by: Andras Gyori
(cherry picked from commit b4a524722a2466ede935aefd7b24a1e08fdbf076)
(cherry picked from commit f001595b47ae9f60cc2ca13676d9daf3baff9963)
(cherry picked from commit 1dc343726c0910141cc156b5e3f58d248a8a67c5)
---
.../client/api/impl/TestTimelineClientV2Impl.java | 36 ++
1 file changed, 24 insertions(+), 12 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClientV2Impl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClientV2Impl.java
index 95595a9..95381a8 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClientV2Impl.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClientV2Impl.java
@@ -248,24 +248,36 @@ public class TestTimelineClientV2Impl {
Thread.sleep(TIME_TO_SLEEP);
}
printReceivedEntities();
-Assert.assertEquals("TimelineEntities not published as desired", 3,
-client.getNumOfTimelineEntitiesPublished());
+
+boolean asyncPushesMerged = client.getNumOfTimelineEntitiesPublished() ==
3;
+int lastPublishIndex = asyncPushesMerged ? 2 : 3;
+
TimelineEntities firstPublishedEntities = client.getPublishedEntities(0);
Assert.assertEquals("sync entities should not be merged with async", 1,
firstPublishedEntities.getEntities().size());
-// test before pushing the sync entities asyncs are merged and pushed
-TimelineEntities secondPublishedEntities = client.getPublishedEntities(1);
-Assert.assertEquals(
-"async entities should be merged before publishing sync", 2,
-secondPublishedEntities.getEntities().size());
-Assert.assertEquals("Order of Async Events Needs to be FIFO", "2",
-secondPublishedEntities.getEntities().get(0).getId());
-Assert.assertEquals("Order of Async Events Needs to be FIFO", "3",
-secondPublishedEntities.getEntities().get(1).getId());
+// async push does not guarantee a merge but is FIFO
+if (asyncPushesMerged) {
+ TimelineEntities secondPublishedEntities =
client.getPublishedEntities(1);
+ Assert.assertEquals(
+ "async entities should be merged before publishing sync", 2,
+ secondPublishedEntities.getEntities().size());
+ Assert.assertEquals("Order of Async Events Needs to be FIFO", "2",
+ secondPublishedEntities.getEntities().get(0).getId());
+ Assert.assertEquals("Order of Async Events Needs to be FIFO", "3",
+ secondPublishedEntities.getEntities().get(1).getId());
+} else {
+ TimelineEntities secondAsyncPublish = client.getPublishedEntities(1);
+ Assert.assertEquals("Order of Async Events Needs to be FIFO", "2",
+ secondAsyncPublish.getEntities().get(0).getId());
+ TimelineEntities thirdAsyncPublish = client.getPublishedEntities(2);
+ Assert.assertEquals("Order of Async Events Needs to be FIFO", "3",
+ thirdAsyncPublish.getEntities().get(0).getId());
+}
// test the last entity published is sync put
-TimelineEntities thirdPublishedEntities = client.getPublishedEntities(2);
+TimelineEntities thirdPublishedEntities =
+client.getPublishedEntities(lastPublishIndex);
Assert.assertEquals("sync entities had to be published at the last", 1,
thirdPublishedEntities.getEntities().size());
Assert.assertEquals("Expected last sync Event is not proper", "4",
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.2 updated: YARN-9551. TestTimelineClientV2Impl.testSyncCall fails intermittent (#3212)
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 1dc3437 YARN-9551. TestTimelineClientV2Impl.testSyncCall fails
intermittent (#3212)
1dc3437 is described below
commit 1dc343726c0910141cc156b5e3f58d248a8a67c5
Author: 9uapaw
AuthorDate: Wed Jul 28 04:35:20 2021 +0200
YARN-9551. TestTimelineClientV2Impl.testSyncCall fails intermittent (#3212)
Co-authored-by: Andras Gyori
(cherry picked from commit b4a524722a2466ede935aefd7b24a1e08fdbf076)
(cherry picked from commit f001595b47ae9f60cc2ca13676d9daf3baff9963)
---
.../client/api/impl/TestTimelineClientV2Impl.java | 36 ++
1 file changed, 24 insertions(+), 12 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClientV2Impl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClientV2Impl.java
index 95595a9..95381a8 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClientV2Impl.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClientV2Impl.java
@@ -248,24 +248,36 @@ public class TestTimelineClientV2Impl {
Thread.sleep(TIME_TO_SLEEP);
}
printReceivedEntities();
-Assert.assertEquals("TimelineEntities not published as desired", 3,
-client.getNumOfTimelineEntitiesPublished());
+
+boolean asyncPushesMerged = client.getNumOfTimelineEntitiesPublished() ==
3;
+int lastPublishIndex = asyncPushesMerged ? 2 : 3;
+
TimelineEntities firstPublishedEntities = client.getPublishedEntities(0);
Assert.assertEquals("sync entities should not be merged with async", 1,
firstPublishedEntities.getEntities().size());
-// test before pushing the sync entities asyncs are merged and pushed
-TimelineEntities secondPublishedEntities = client.getPublishedEntities(1);
-Assert.assertEquals(
-"async entities should be merged before publishing sync", 2,
-secondPublishedEntities.getEntities().size());
-Assert.assertEquals("Order of Async Events Needs to be FIFO", "2",
-secondPublishedEntities.getEntities().get(0).getId());
-Assert.assertEquals("Order of Async Events Needs to be FIFO", "3",
-secondPublishedEntities.getEntities().get(1).getId());
+// async push does not guarantee a merge but is FIFO
+if (asyncPushesMerged) {
+ TimelineEntities secondPublishedEntities =
client.getPublishedEntities(1);
+ Assert.assertEquals(
+ "async entities should be merged before publishing sync", 2,
+ secondPublishedEntities.getEntities().size());
+ Assert.assertEquals("Order of Async Events Needs to be FIFO", "2",
+ secondPublishedEntities.getEntities().get(0).getId());
+ Assert.assertEquals("Order of Async Events Needs to be FIFO", "3",
+ secondPublishedEntities.getEntities().get(1).getId());
+} else {
+ TimelineEntities secondAsyncPublish = client.getPublishedEntities(1);
+ Assert.assertEquals("Order of Async Events Needs to be FIFO", "2",
+ secondAsyncPublish.getEntities().get(0).getId());
+ TimelineEntities thirdAsyncPublish = client.getPublishedEntities(2);
+ Assert.assertEquals("Order of Async Events Needs to be FIFO", "3",
+ thirdAsyncPublish.getEntities().get(0).getId());
+}
// test the last entity published is sync put
-TimelineEntities thirdPublishedEntities = client.getPublishedEntities(2);
+TimelineEntities thirdPublishedEntities =
+client.getPublishedEntities(lastPublishIndex);
Assert.assertEquals("sync entities had to be published at the last", 1,
thirdPublishedEntities.getEntities().size());
Assert.assertEquals("Expected last sync Event is not proper", "4",
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.3 updated: YARN-9551. TestTimelineClientV2Impl.testSyncCall fails intermittent (#3212)
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new f001595 YARN-9551. TestTimelineClientV2Impl.testSyncCall fails
intermittent (#3212)
f001595 is described below
commit f001595b47ae9f60cc2ca13676d9daf3baff9963
Author: 9uapaw
AuthorDate: Wed Jul 28 04:35:20 2021 +0200
YARN-9551. TestTimelineClientV2Impl.testSyncCall fails intermittent (#3212)
Co-authored-by: Andras Gyori
(cherry picked from commit b4a524722a2466ede935aefd7b24a1e08fdbf076)
---
.../client/api/impl/TestTimelineClientV2Impl.java | 36 ++
1 file changed, 24 insertions(+), 12 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClientV2Impl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClientV2Impl.java
index 6770c07..8d437af 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClientV2Impl.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClientV2Impl.java
@@ -248,24 +248,36 @@ public class TestTimelineClientV2Impl {
Thread.sleep(TIME_TO_SLEEP);
}
printReceivedEntities();
-Assert.assertEquals("TimelineEntities not published as desired", 3,
-client.getNumOfTimelineEntitiesPublished());
+
+boolean asyncPushesMerged = client.getNumOfTimelineEntitiesPublished() ==
3;
+int lastPublishIndex = asyncPushesMerged ? 2 : 3;
+
TimelineEntities firstPublishedEntities = client.getPublishedEntities(0);
Assert.assertEquals("sync entities should not be merged with async", 1,
firstPublishedEntities.getEntities().size());
-// test before pushing the sync entities asyncs are merged and pushed
-TimelineEntities secondPublishedEntities = client.getPublishedEntities(1);
-Assert.assertEquals(
-"async entities should be merged before publishing sync", 2,
-secondPublishedEntities.getEntities().size());
-Assert.assertEquals("Order of Async Events Needs to be FIFO", "2",
-secondPublishedEntities.getEntities().get(0).getId());
-Assert.assertEquals("Order of Async Events Needs to be FIFO", "3",
-secondPublishedEntities.getEntities().get(1).getId());
+// async push does not guarantee a merge but is FIFO
+if (asyncPushesMerged) {
+ TimelineEntities secondPublishedEntities =
client.getPublishedEntities(1);
+ Assert.assertEquals(
+ "async entities should be merged before publishing sync", 2,
+ secondPublishedEntities.getEntities().size());
+ Assert.assertEquals("Order of Async Events Needs to be FIFO", "2",
+ secondPublishedEntities.getEntities().get(0).getId());
+ Assert.assertEquals("Order of Async Events Needs to be FIFO", "3",
+ secondPublishedEntities.getEntities().get(1).getId());
+} else {
+ TimelineEntities secondAsyncPublish = client.getPublishedEntities(1);
+ Assert.assertEquals("Order of Async Events Needs to be FIFO", "2",
+ secondAsyncPublish.getEntities().get(0).getId());
+ TimelineEntities thirdAsyncPublish = client.getPublishedEntities(2);
+ Assert.assertEquals("Order of Async Events Needs to be FIFO", "3",
+ thirdAsyncPublish.getEntities().get(0).getId());
+}
// test the last entity published is sync put
-TimelineEntities thirdPublishedEntities = client.getPublishedEntities(2);
+TimelineEntities thirdPublishedEntities =
+client.getPublishedEntities(lastPublishIndex);
Assert.assertEquals("sync entities had to be published at the last", 1,
thirdPublishedEntities.getEntities().size());
Assert.assertEquals("Expected last sync Event is not proper", "4",
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-6272. TestAMRMClient#testAMRMClientWithContainerResourceChange fails intermittently. Contributed by Andras Gyory & Prabhu Joseph
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new f2b6c03 YARN-6272.
TestAMRMClient#testAMRMClientWithContainerResourceChange fails intermittently.
Contributed by Andras Gyory & Prabhu Joseph
f2b6c03 is described below
commit f2b6c03fc1929679170dfec04891f202181d88e8
Author: Szilard Nemeth
AuthorDate: Wed Jul 28 17:02:15 2021 +0200
YARN-6272. TestAMRMClient#testAMRMClientWithContainerResourceChange fails
intermittently. Contributed by Andras Gyory & Prabhu Joseph
---
.../yarn/client/api/impl/TestAMRMClient.java | 55 +-
1 file changed, 33 insertions(+), 22 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java
index 24666f0..fff27bd 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java
@@ -956,7 +956,6 @@ public class TestAMRMClient extends BaseAMRMClientTest{
return containers;
}
-
private void doContainerResourceChange(
final AMRMClient amClient, List containers)
throws YarnException, IOException {
@@ -986,38 +985,50 @@ public class TestAMRMClient extends BaseAMRMClientTest{
Resource.newInstance(512, 1), null));
assertEquals(Resource.newInstance(512, 1),
amClientImpl.change.get(container1.getId()).getValue().getCapability());
-// request resource increase for container2
-amClientImpl.requestContainerUpdate(container2,
-UpdateContainerRequest.newInstance(container2.getVersion(),
-container2.getId(), ContainerUpdateType.INCREASE_RESOURCE,
-Resource.newInstance(2048, 1), null));
-assertEquals(Resource.newInstance(2048, 1),
-
amClientImpl.change.get(container2.getId()).getValue().getCapability());
// verify release request will cancel pending change requests for the same
// container
amClientImpl.requestContainerUpdate(container3,
UpdateContainerRequest.newInstance(container3.getVersion(),
container3.getId(), ContainerUpdateType.INCREASE_RESOURCE,
Resource.newInstance(2048, 1), null));
-assertEquals(3, amClientImpl.pendingChange.size());
-amClientImpl.releaseAssignedContainer(container3.getId());
assertEquals(2, amClientImpl.pendingChange.size());
+amClientImpl.releaseAssignedContainer(container3.getId());
+assertEquals(1, amClientImpl.pendingChange.size());
// as of now: container1 asks to decrease to (512, 1)
//container2 asks to increase to (2048, 1)
// send allocation requests
-AllocateResponse allocResponse = amClient.allocate(0.1f);
-assertEquals(0, amClientImpl.change.size());
-// we should get decrease confirmation right away
-List updatedContainers =
-allocResponse.getUpdatedContainers();
-assertEquals(1, updatedContainers.size());
// we should get increase allocation after the next NM's heartbeat to RM
-triggerSchedulingWithNMHeartBeat();
-// get allocations
-allocResponse = amClient.allocate(0.1f);
-updatedContainers =
-allocResponse.getUpdatedContainers();
-assertEquals(1, updatedContainers.size());
+assertUpdatedContainers(amClient, container1);
+// request resource increase for container2
+amClientImpl.requestContainerUpdate(container2,
+UpdateContainerRequest.newInstance(container2.getVersion(),
+container2.getId(), ContainerUpdateType.INCREASE_RESOURCE,
+Resource.newInstance(2048, 1), null));
+assertEquals(Resource.newInstance(2048, 1),
+
amClientImpl.change.get(container2.getId()).getValue().getCapability());
+assertUpdatedContainers(amClient, container2);
+ }
+
+ private void assertUpdatedContainers(AMRMClient amClient,
Container container) {
+RMContext context = yarnCluster.getResourceManager().getRMContext();
+RMNode rmNode = context.getRMNodes().get(container.getNodeId());
+List updateResponse = new ArrayList<>();
+int allocationAttempts = 0;
+while (allocationAttempts < 1000) {
+ context.getScheduler().handle(new NodeUpdateSchedulerEvent(rmNode));
+ try {
+updateResponse = amClient.allocate(0.1f).getUpdatedContainers();
+if (updateResponse.size() == 1) {
+ break;
+} else {
+ allocationAttempts++;
+ sleep(20);
+}
+ } ca
[hadoop] branch trunk updated: YARN-10790. CS Flexible AQC: Add separate parent and leaf template property. Contributed by Andras Gyori
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 1b9efe5 YARN-10790. CS Flexible AQC: Add separate parent and leaf
template property. Contributed by Andras Gyori
1b9efe5 is described below
commit 1b9efe58c90a4485308387fe34d8b50756b717a3
Author: Szilard Nemeth
AuthorDate: Wed Jul 28 16:50:14 2021 +0200
YARN-10790. CS Flexible AQC: Add separate parent and leaf template
property. Contributed by Andras Gyori
---
.../capacity/AutoCreatedQueueTemplate.java | 102 ++---
.../webapp/dao/CapacitySchedulerInfo.java | 14 ++-
.../webapp/dao/CapacitySchedulerQueueInfo.java | 16 +++-
.../dao/helper/CapacitySchedulerInfoHelper.java| 5 +-
.../capacity/TestAutoCreatedQueueTemplate.java | 41 +
.../webapp/TestRMWebServicesCapacitySched.java | 6 +-
.../TestRMWebServicesForCSWithPartitions.java | 2 +-
7 files changed, 164 insertions(+), 22 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AutoCreatedQueueTemplate.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AutoCreatedQueueTemplate.java
index 203ec4d..cb0f789 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AutoCreatedQueueTemplate.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AutoCreatedQueueTemplate.java
@@ -30,6 +30,7 @@ import java.util.Set;
import static
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.AUTO_QUEUE_CREATION_V2_PREFIX;
import static
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.ROOT;
+import static
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueuePrefix;
/**
* A handler for storing and setting auto created queue template settings.
@@ -37,10 +38,17 @@ import static
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.C
public class AutoCreatedQueueTemplate {
public static final String AUTO_QUEUE_TEMPLATE_PREFIX =
AUTO_QUEUE_CREATION_V2_PREFIX + "template.";
+ public static final String AUTO_QUEUE_LEAF_TEMPLATE_PREFIX =
+ AUTO_QUEUE_CREATION_V2_PREFIX + "leaf-template.";
+ public static final String AUTO_QUEUE_PARENT_TEMPLATE_PREFIX =
+ AUTO_QUEUE_CREATION_V2_PREFIX + "parent-template.";
+
private static final String WILDCARD_QUEUE = "*";
private static final int MAX_WILDCARD_LEVEL = 1;
private final Map templateProperties = new HashMap<>();
+ private final Map leafOnlyProperties = new HashMap<>();
+ private final Map parentOnlyProperties = new HashMap<>();
public AutoCreatedQueueTemplate(Configuration configuration,
String queuePath) {
@@ -54,7 +62,7 @@ public class AutoCreatedQueueTemplate {
}
/**
- * Get the template properties attached to a parent queue.
+ * Get the common template properties specified for a parent queue.
* @return template property names and values
*/
public Map getTemplateProperties() {
@@ -62,13 +70,44 @@ public class AutoCreatedQueueTemplate {
}
/**
- * Sets the configuration properties of a child queue based on its parent
+ * Get the leaf specific template properties specified for a parent queue.
+ * @return template property names and values
+ */
+ public Map getLeafOnlyProperties() {
+return leafOnlyProperties;
+ }
+
+ /**
+ * Get the parent specific template properties specified for a parent queue.
+ * @return template property names and values
+ */
+ public Map getParentOnlyProperties() {
+return parentOnlyProperties;
+ }
+
+ /**
+ * Sets the common template properties and parent specific template
+ * properties of a child queue based on its parent template settings.
+ * @param conf configuration to set
+ * @param childQueuePath child queue path used for prefixing the properties
+ */
+ public void setTemplateEntriesForChild(Configuration conf,
+ String childQueuePath) {
+setTemplateEntriesForChild(conf, childQueuePath, false);
+ }
+
+ /**
+ * Sets the common template properties and leaf or parent specific template
+ * properties of a child queue based on its parent template settings.
* template set
[hadoop] branch trunk updated: YARN-10790. CS Flexible AQC: Add separate parent and leaf template property. Contributed by Andras Gyori
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new baea191 YARN-10790. CS Flexible AQC: Add separate parent and leaf
template property. Contributed by Andras Gyori
baea191 is described below
commit baea191a91cd881e2df987aa7de8553627e8e109
Author: Szilard Nemeth
AuthorDate: Wed Jul 28 16:40:23 2021 +0200
YARN-10790. CS Flexible AQC: Add separate parent and leaf template
property. Contributed by Andras Gyori
---
.../capacity/AutoCreatedQueueTemplate.java | 102 ++---
.../webapp/dao/CapacitySchedulerInfo.java | 14 ++-
.../webapp/dao/CapacitySchedulerQueueInfo.java | 16 +++-
.../dao/helper/CapacitySchedulerInfoHelper.java| 5 +-
.../capacity/TestAutoCreatedQueueTemplate.java | 40
.../webapp/TestRMWebServicesCapacitySched.java | 6 +-
.../TestRMWebServicesForCSWithPartitions.java | 2 +-
7 files changed, 162 insertions(+), 23 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AutoCreatedQueueTemplate.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AutoCreatedQueueTemplate.java
index 203ec4d..cf78fed 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AutoCreatedQueueTemplate.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AutoCreatedQueueTemplate.java
@@ -30,6 +30,7 @@ import java.util.Set;
import static
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.AUTO_QUEUE_CREATION_V2_PREFIX;
import static
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.ROOT;
+import static
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueuePrefix;
/**
* A handler for storing and setting auto created queue template settings.
@@ -37,10 +38,17 @@ import static
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.C
public class AutoCreatedQueueTemplate {
public static final String AUTO_QUEUE_TEMPLATE_PREFIX =
AUTO_QUEUE_CREATION_V2_PREFIX + "template.";
+ public static final String AUTO_QUEUE_LEAF_TEMPLATE_PREFIX =
+ AUTO_QUEUE_CREATION_V2_PREFIX + "leaf-template.";
+ public static final String AUTO_QUEUE_PARENT_TEMPLATE_PREFIX =
+ AUTO_QUEUE_CREATION_V2_PREFIX + "parent-template.";
+
private static final String WILDCARD_QUEUE = "*";
private static final int MAX_WILDCARD_LEVEL = 1;
private final Map templateProperties = new HashMap<>();
+ private final Map leafOnlyProperties = new HashMap<>();
+ private final Map parentOnlyProperties = new HashMap<>();
public AutoCreatedQueueTemplate(Configuration configuration,
String queuePath) {
@@ -54,7 +62,7 @@ public class AutoCreatedQueueTemplate {
}
/**
- * Get the template properties attached to a parent queue.
+ * Get the common template properties specified for a parent queue.
* @return template property names and values
*/
public Map getTemplateProperties() {
@@ -62,13 +70,43 @@ public class AutoCreatedQueueTemplate {
}
/**
- * Sets the configuration properties of a child queue based on its parent
- * template settings.
- * @param conf configuration to set
+ * Get the leaf specific template properties specified for a parent queue.
+ * @return template property names and values
+ */
+ public Map getLeafOnlyProperties() {
+return leafOnlyProperties;
+ }
+
+ /**
+ * Get the parent specific template properties specified for a parent queue.
+ * @return template property names and values
+ */
+ public Map getParentOnlyProperties() {
+return parentOnlyProperties;
+ }
+
+ /**
+ * Sets the common template properties and parent specific template
+ * properties of a child queue based on its parent template settings.
+ * @param conf configuration to set
* @param childQueuePath child queue path used for prefixing the properties
*/
public void setTemplateEntriesForChild(Configuration conf,
String childQueuePath) {
+setTemplateEntriesForChild(conf, childQueuePath, false);
+ }
+
+ /**
+ * Sets the common template properties and leaf or parent specific template
+ * properties of a child
[hadoop] branch branch-3.3 updated: YARN-10813. Set default capacity of root for node labels. Contributed by Andras Gyori
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new 72801be YARN-10813. Set default capacity of root for node labels.
Contributed by Andras Gyori
72801be is described below
commit 72801be13af9938ae67a675fe212d38df65ba23b
Author: Szilard Nemeth
AuthorDate: Wed Jul 28 14:55:19 2021 +0200
YARN-10813. Set default capacity of root for node labels. Contributed by
Andras Gyori
---
.../scheduler/capacity/CapacitySchedulerConfiguration.java | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
index 047beeb..9d4b6e2 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
@@ -712,10 +712,11 @@ public class CapacitySchedulerConfiguration extends
ReservationSchedulerConfigur
// root.From AbstractCSQueue, absolute resource will be parsed and
// updated. Once nodes are added/removed in cluster, capacity in
// percentage will also be re-calculated.
- return defaultValue;
+ return queue.equals("root") ? 100.0f : defaultValue;
}
-float capacity = getFloat(capacityPropertyName, defaultValue);
+float capacity = queue.equals("root") ? 100.0f
+: getFloat(capacityPropertyName, defaultValue);
if (capacity < MINIMUM_CAPACITY_VALUE
|| capacity > MAXIMUM_CAPACITY_VALUE) {
throw new IllegalArgumentException("Illegal capacity of " + capacity
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.2 updated: YARN-10813. Set default capacity of root for node labels. Contributed by Andras Gyori
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 6bf02a6 YARN-10813. Set default capacity of root for node labels.
Contributed by Andras Gyori
6bf02a6 is described below
commit 6bf02a63e78703bced947500d874c827e74bf08b
Author: Szilard Nemeth
AuthorDate: Wed Jul 28 14:54:36 2021 +0200
YARN-10813. Set default capacity of root for node labels. Contributed by
Andras Gyori
---
.../scheduler/capacity/CapacitySchedulerConfiguration.java | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
index 2feecd9..06b56b1 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
@@ -697,10 +697,11 @@ public class CapacitySchedulerConfiguration extends
ReservationSchedulerConfigur
// root.From AbstractCSQueue, absolute resource will be parsed and
// updated. Once nodes are added/removed in cluster, capacity in
// percentage will also be re-calculated.
- return defaultValue;
+ return queue.equals("root") ? 100.0f : defaultValue;
}
-float capacity = getFloat(capacityPropertyName, defaultValue);
+float capacity = queue.equals("root") ? 100.0f
+: getFloat(capacityPropertyName, defaultValue);
if (capacity < MINIMUM_CAPACITY_VALUE
|| capacity > MAXIMUM_CAPACITY_VALUE) {
throw new IllegalArgumentException("Illegal capacity of " + capacity
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10727. ParentQueue does not validate the queue on removal. Contributed by Andras Gyori
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 8d0297c YARN-10727. ParentQueue does not validate the queue on
removal. Contributed by Andras Gyori
8d0297c is described below
commit 8d0297c2136dbe7fb7fa729ed840b2e8433947ea
Author: Szilard Nemeth
AuthorDate: Wed Jul 28 14:49:10 2021 +0200
YARN-10727. ParentQueue does not validate the queue on removal. Contributed
by Andras Gyori
---
.../scheduler/capacity/ParentQueue.java| 15
.../TestCapacitySchedulerNewQueueAutoCreation.java | 45 ++
2 files changed, 60 insertions(+)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
index 8ff4d91..e1307d7 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
@@ -581,6 +581,21 @@ public class ParentQueue extends AbstractCSQueue {
throws SchedulerDynamicEditException {
writeLock.lock();
try {
+ if (!(queue instanceof AbstractCSQueue) ||
+ !((AbstractCSQueue) queue).isDynamicQueue()) {
+throw new SchedulerDynamicEditException("Queue " + getQueuePath()
++ " can not remove " + queue.getQueuePath() +
+" because it is not a dynamic queue");
+ }
+
+ // We need to check if the parent of the child queue is exactly this
+ // ParentQueue object
+ if (queue.getParent() != this) {
+throw new SchedulerDynamicEditException("Queue " + getQueuePath()
++ " can not remove " + queue.getQueuePath() +
+" because it has a different parent queue");
+ }
+
// Now we can do remove and update
this.childQueues.remove(queue);
this.scheduler.getCapacitySchedulerQueueManager()
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerNewQueueAutoCreation.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerNewQueueAutoCreation.java
index 4229676..053a135 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerNewQueueAutoCreation.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerNewQueueAutoCreation.java
@@ -1162,6 +1162,51 @@ public class TestCapacitySchedulerNewQueueAutoCreation
"when its dynamic parent is removed", bAutoLeaf);
}
+ @Test
+ public void testParentQueueDynamicChildRemoval() throws Exception {
+startScheduler();
+
+createQueue("root.a.a-auto");
+createQueue("root.a.a-auto");
+AbstractCSQueue aAuto = (AbstractCSQueue) cs.
+getQueue("root.a.a-auto");
+Assert.assertTrue(aAuto.isDynamicQueue());
+ParentQueue a = (ParentQueue) cs.
+getQueue("root.a");
+createQueue("root.e.e1-auto");
+AbstractCSQueue eAuto = (AbstractCSQueue) cs.
+getQueue("root.e.e1-auto");
+Assert.assertTrue(eAuto.isDynamicQueue());
+ParentQueue e = (ParentQueue) cs.
+getQueue("root.e");
+
+// Try to remove a static child queue
+try {
+ a.removeChildQueue(cs.getQueue("root.a.a1"));
+ Assert.fail("root.a.a1 is a static queue and should not be removed at " +
+ "runtime");
+} catch (SchedulerDynamicEditException ignored) {
+}
+
+// Try to remove a dynamic queue with a different parent
+try {
+ a.removeChildQueue(eAuto);
+ Assert.fail("root.a should not be able to remove root.e.e1-auto");
+} catch (SchedulerDynamicEditException ignored) {
+}
+
+a.removeChildQueue(aAuto);
+e.removeChildQueue(eAuto);
+
+aAuto = (AbstractCSQueue) cs.
[hadoop] branch trunk updated: YARN-6221. Entities missing from ATS when summary log file info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new fa0289b YARN-6221. Entities missing from ATS when summary log file
info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang
fa0289b is described below
commit fa0289b0226f40fa5c7f898de586b8fd8b763374
Author: Szilard Nemeth
AuthorDate: Tue Jul 27 16:47:41 2021 +0200
YARN-6221. Entities missing from ATS when summary log file info got
returned to the ATS before the domain log. Contributed by Xiaomin Zhang
---
.../timeline/EntityGroupFSTimelineStore.java | 3 ++-
.../timeline/TestEntityGroupFSTimelineStore.java | 25 ++
2 files changed, 27 insertions(+), 1 deletion(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java
index e9957c8..c2992b4 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java
@@ -808,10 +808,11 @@ public class EntityGroupFSTimelineStore extends
CompositeService
LogInfo log;
if (isDomainLog) {
log = new DomainLogInfo(attemptDirName, filename, owner);
+summaryLogs.add(0, log);
} else {
log = new EntityLogInfo(attemptDirName, filename, owner);
+summaryLogs.add(log);
}
- summaryLogs.add(log);
}
private synchronized void addDetailLog(String attemptDirName,
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/TestEntityGroupFSTimelineStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/TestEntityGroupFSTimelineStore.java
index 984e157..7302ae1 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/TestEntityGroupFSTimelineStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/test/java/org/apache/hadoop/yarn/server/timeline/TestEntityGroupFSTimelineStore.java
@@ -227,6 +227,31 @@ public class TestEntityGroupFSTimelineStore extends
TimelineStoreTestUtils {
}
@Test
+ public void testAppLogsDomainLogLastlyScanned() throws Exception {
+EntityGroupFSTimelineStore.AppLogs appLogs =
+store.new AppLogs(mainTestAppId, mainTestAppDirPath,
+AppState.COMPLETED);
+Path attemptDirPath = new Path(new Path(testActiveDirPath,
+mainTestAppId.toString()),
+getAttemptDirName(mainTestAppId));
+//Delete the domain log from AppDirPath so first scan won't find it
+fs.delete(new Path(attemptDirPath, TEST_DOMAIN_LOG_FILE_NAME), false);
+appLogs.scanForLogs();
+List summaryLogs = appLogs.getSummaryLogs();
+assertEquals(1, summaryLogs.size());
+assertEquals(TEST_SUMMARY_LOG_FILE_NAME, summaryLogs.get(0).getFilename());
+
+//Generate the domain log
+FSDataOutputStream out = fs.create(
+new Path(attemptDirPath, TEST_DOMAIN_LOG_FILE_NAME));
+out.close();
+
+appLogs.scanForLogs();
+assertEquals(2, summaryLogs.size());
+assertEquals(TEST_DOMAIN_LOG_FILE_NAME, summaryLogs.get(0).getFilename());
+ }
+
+ @Test
public void testMoveToDone() throws Exception {
EntityGroupFSTimelineStore.AppLogs appLogs =
store.new AppLogs(mainTestAppId, mainTestAppDirPath,
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.2 updated: YARN-10828. Backport YARN-9789 to branch-3.2. Contributed by Tarun Parimi
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new c8b090d YARN-10828. Backport YARN-9789 to branch-3.2. Contributed by
Tarun Parimi
c8b090d is described below
commit c8b090d9a492211e3be55e6adb79da57c6db304e
Author: Szilard Nemeth
AuthorDate: Thu Jun 24 20:20:23 2021 +0200
YARN-10828. Backport YARN-9789 to branch-3.2. Contributed by Tarun Parimi
---
.../capacity/conf/LeveldbConfigurationStore.java | 17 +
.../capacity/conf/ZKConfigurationStore.java| 20 ---
.../conf/TestLeveldbConfigurationStore.java| 29 ++
3 files changed, 52 insertions(+), 14 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/LeveldbConfigurationStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/LeveldbConfigurationStore.java
index b347aba..1ff53a6 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/LeveldbConfigurationStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/LeveldbConfigurationStore.java
@@ -193,12 +193,14 @@ public class LeveldbConfigurationStore extends
YarnConfigurationStore {
@Override
public void logMutation(LogMutation logMutation) throws IOException {
-LinkedList logs = deserLogMutations(db.get(bytes(LOG_KEY)));
-logs.add(logMutation);
-if (logs.size() > maxLogs) {
- logs.removeFirst();
+if(maxLogs > 0) {
+ LinkedList logs = deserLogMutations(db.get(bytes(LOG_KEY)));
+ logs.add(logMutation);
+ if (logs.size() > maxLogs) {
+logs.removeFirst();
+ }
+ db.put(bytes(LOG_KEY), serLogMutations(logs));
}
-db.put(bytes(LOG_KEY), serLogMutations(logs));
pendingMutation = logMutation;
}
@@ -293,6 +295,11 @@ public class LeveldbConfigurationStore extends
YarnConfigurationStore {
return deserLogMutations(db.get(bytes(LOG_KEY)));
}
+ @VisibleForTesting
+ protected DB getDB() {
+return db;
+ }
+
@Override
public void storeVersion() throws Exception {
try {
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java
index 229f680..3742c36 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java
@@ -156,16 +156,18 @@ public class ZKConfigurationStore extends
YarnConfigurationStore {
@Override
public void logMutation(LogMutation logMutation) throws Exception {
-byte[] storedLogs = getZkData(logsPath);
-LinkedList logs = new LinkedList<>();
-if (storedLogs != null) {
- logs = unsafeCast(deserializeObject(storedLogs));
-}
-logs.add(logMutation);
-if (logs.size() > maxLogs) {
- logs.remove(logs.removeFirst());
+if(maxLogs > 0) {
+ byte[] storedLogs = getZkData(logsPath);
+ LinkedList logs = new LinkedList<>();
+ if (storedLogs != null) {
+logs = unsafeCast(deserializeObject(storedLogs));
+ }
+ logs.add(logMutation);
+ if (logs.size() > maxLogs) {
+logs.remove(logs.removeFirst());
+ }
+ safeSetZkData(logsPath, logs);
}
-safeSetZkData(logsPath, logs);
pendingMutation = logMutation;
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestLeveldbConfigurationStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestLeveldbConfigurationStore.java
index 4c92120..e88e7de 100644
---
a/hadoop-yarn-project/hadoop-yarn/
[hadoop] branch trunk updated: YARN-10813. Set default capacity of root for node labels. Contributed by Andras Gyori
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 6562391 YARN-10813. Set default capacity of root for node labels.
Contributed by Andras Gyori
6562391 is described below
commit 65623917377db409fa5ff813b9794f279511ce6e
Author: Szilard Nemeth
AuthorDate: Wed Jun 16 18:55:09 2021 +0200
YARN-10813. Set default capacity of root for node labels. Contributed by
Andras Gyori
---
.../scheduler/capacity/CapacitySchedulerConfiguration.java | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
index 58a8002..622fef6 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
@@ -800,10 +800,11 @@ public class CapacitySchedulerConfiguration extends
ReservationSchedulerConfigur
// root.From AbstractCSQueue, absolute resource, and weight will be
parsed
// and updated separately. Once nodes are added/removed in cluster,
// capacity is percentage will also be re-calculated.
- return defaultValue;
+ return queue.equals("root") ? 100.0f : defaultValue;
}
-float capacity = getFloat(capacityPropertyName, defaultValue);
+float capacity = queue.equals("root") ? 100.0f
+: getFloat(capacityPropertyName, defaultValue);
if (capacity < MINIMUM_CAPACITY_VALUE
|| capacity > MAXIMUM_CAPACITY_VALUE) {
throw new IllegalArgumentException(
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10801. Fix Auto Queue template to properly set all configuration properties. Contributed by Andras Gyori
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 428478b YARN-10801. Fix Auto Queue template to properly set all
configuration properties. Contributed by Andras Gyori
428478b is described below
commit 428478bbe284ebd4cdec967fc082f31841a96c1b
Author: Szilard Nemeth
AuthorDate: Wed Jun 16 18:26:58 2021 +0200
YARN-10801. Fix Auto Queue template to properly set all configuration
properties. Contributed by Andras Gyori
---
.../scheduler/capacity/AbstractCSQueue.java| 18 --
.../scheduler/capacity/LeafQueue.java | 22 ++--
.../scheduler/capacity/ParentQueue.java| 41 +++---
.../TestCapacitySchedulerNewQueueAutoCreation.java | 10 ++
4 files changed, 58 insertions(+), 33 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
index be9a0e3..94ade5e 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
@@ -360,9 +360,8 @@ public abstract class AbstractCSQueue implements CSQueue {
writeLock.lock();
try {
- if (isDynamicQueue() && getParent() instanceof ParentQueue) {
-((ParentQueue) getParent()).getAutoCreatedQueueTemplate()
-.setTemplateEntriesForChild(configuration, getQueuePath());
+ if (isDynamicQueue()) {
+setDynamicQueueProperties(configuration);
}
// get labels
this.accessibleLabels =
@@ -478,6 +477,19 @@ public abstract class AbstractCSQueue implements CSQueue {
}
}
+ /**
+ * Set properties specific to dynamic queues.
+ * @param configuration configuration on which the properties are set
+ */
+ protected void setDynamicQueueProperties(
+ CapacitySchedulerConfiguration configuration) {
+// Set properties from parent template
+if (getParent() instanceof ParentQueue) {
+ ((ParentQueue) getParent()).getAutoCreatedQueueTemplate()
+ .setTemplateEntriesForChild(configuration, getQueuePath());
+}
+ }
+
private void setupMaximumAllocation(CapacitySchedulerConfiguration csConf) {
String myQueuePath = getQueuePath();
Resource clusterMax = ResourceUtils
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
index c76bf98..19de6db 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
@@ -147,14 +147,21 @@ public class LeafQueue extends AbstractCSQueue {
@SuppressWarnings({ "unchecked", "rawtypes" })
public LeafQueue(CapacitySchedulerContext cs,
String queueName, CSQueue parent, CSQueue old) throws IOException {
- this(cs, cs.getConfiguration(), queueName, parent, old);
+this(cs, cs.getConfiguration(), queueName, parent, old, false);
}
public LeafQueue(CapacitySchedulerContext cs,
CapacitySchedulerConfiguration configuration,
- String queueName, CSQueue parent, CSQueue old) throws
+ String queueName, CSQueue parent, CSQueue old) throws IOException {
+this(cs, configuration, queueName, parent, old, false);
+ }
+
+ public LeafQueue(CapacitySchedulerContext cs,
+ CapacitySchedulerConfiguration configuration,
+ String queueName, CSQueue parent, CSQueue old, boolean isDynamic) throws
IOException {
super(cs, configuration, queueName, parent, old);
+setDynamicQueue(isDynamic);
this.scheduler = cs;
this.usersManager = new UsersManager(metrics, this, labelManager,
scheduler,
@@ -1691,6 +1698,17 @@ public class LeafQueue extends AbstractCSQueue {
}
}
+ @Override
+ protected vo
[hadoop] branch trunk updated: YARN-10802. Change Capacity Scheduler minimum-user-limit-percent to accept decimal values. Contributed by Benjamin Teke
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new e31d060 YARN-10802. Change Capacity Scheduler
minimum-user-limit-percent to accept decimal values. Contributed by Benjamin
Teke
e31d060 is described below
commit e31d06032baebaf651ce8c69eed8c934586f3386
Author: Szilard Nemeth
AuthorDate: Mon Jun 14 22:33:04 2021 +0200
YARN-10802. Change Capacity Scheduler minimum-user-limit-percent to accept
decimal values. Contributed by Benjamin Teke
---
.../capacity/CapacitySchedulerConfiguration.java | 8 +-
.../scheduler/capacity/LeafQueue.java | 10 +-
.../scheduler/capacity/PlanQueue.java | 40
.../scheduler/capacity/ReservationQueue.java | 2 +-
.../scheduler/capacity/UsersManager.java | 6 +-
.../webapp/CapacitySchedulerPage.java | 21 +++--
.../webapp/dao/CapacitySchedulerLeafQueueInfo.java | 4 +-
.../scheduler/capacity/TestLeafQueue.java | 105 +
.../scheduler/capacity/TestParentQueue.java| 14 ++-
.../webapp/TestRMWebServicesCapacitySched.java | 8 +-
10 files changed, 170 insertions(+), 48 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
index 074e371..58a8002 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
@@ -636,8 +636,8 @@ public class CapacitySchedulerConfiguration extends
ReservationSchedulerConfigur
absoluteResourceCapacity);
}
- public int getUserLimit(String queue) {
-int userLimit = getInt(getQueuePrefix(queue) + USER_LIMIT,
+ public float getUserLimit(String queue) {
+float userLimit = getFloat(getQueuePrefix(queue) + USER_LIMIT,
DEFAULT_USER_LIMIT);
return userLimit;
}
@@ -686,8 +686,8 @@ public class CapacitySchedulerConfiguration extends
ReservationSchedulerConfigur
return orderingPolicy;
}
- public void setUserLimit(String queue, int userLimit) {
-setInt(getQueuePrefix(queue) + USER_LIMIT, userLimit);
+public void setUserLimit(String queue, float userLimit) {
+setFloat(getQueuePrefix(queue) + USER_LIMIT, userLimit);
LOG.debug("here setUserLimit: queuePrefix={}, userLimit={}",
getQueuePrefix(queue), getUserLimit(queue));
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
index 71732f6..c76bf98 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
@@ -255,7 +255,7 @@ public class LeafQueue extends AbstractCSQueue {
conf.getDefaultApplicationPriorityConfPerQueue(getQueuePath()));
// Validate leaf queue's user's weights.
- int queueUL = Math.min(100, conf.getUserLimit(getQueuePath()));
+ float queueUL = Math.min(100.0f, conf.getUserLimit(getQueuePath()));
for (Entry e : getUserWeights().entrySet()) {
float val = e.getValue().floatValue();
if (val < 0.0f || val > (100.0f / queueUL)) {
@@ -367,17 +367,17 @@ public class LeafQueue extends AbstractCSQueue {
}
/**
- * Set user limit - used only for testing.
+ * Set user limit.
* @param userLimit new user limit
*/
@VisibleForTesting
- void setUserLimit(int userLimit) {
+ void setUserLimit(float userLimit) {
usersManager.setUserLimit(userLimit);
usersManager.userLimitNeedsRecompute();
}
/**
- * Set user limit factor - used only for testing.
+ * Set user limit factor.
* @param userLimitFactor new user limit factor
*/
@V
[hadoop] branch trunk updated: YARN-10816. Avoid doing delegation token ops when yarn.timeline-service.http-authentication.type=simple. Contributed by Tarun Parimi
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new f0bdc42 YARN-10816. Avoid doing delegation token ops when
yarn.timeline-service.http-authentication.type=simple. Contributed by Tarun
Parimi
f0bdc42 is described below
commit f0bdc422aa7093f5a5c1e3cc5c9fa57f4c1205d5
Author: Szilard Nemeth
AuthorDate: Sat Jun 12 15:18:41 2021 +0200
YARN-10816. Avoid doing delegation token ops when
yarn.timeline-service.http-authentication.type=simple. Contributed by Tarun
Parimi
---
.../yarn/client/api/impl/TimelineClientImpl.java | 25 ++
.../yarn/client/api/impl/TestTimelineClient.java | 39 ++
2 files changed, 64 insertions(+)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
index 07b41c3..eda38c5 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
@@ -29,6 +29,7 @@ import org.apache.commons.cli.GnuParser;
import org.apache.commons.cli.HelpFormatter;
import org.apache.commons.cli.Options;
import
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler;
+import
org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.hadoop.classification.InterfaceAudience.Private;
@@ -88,6 +89,7 @@ public class TimelineClientImpl extends TimelineClient {
private TimelineWriter timelineWriter;
private String timelineServiceAddress;
+ private String authType;
@Private
@VisibleForTesting
@@ -128,6 +130,12 @@ public class TimelineClientImpl extends TimelineClient {
conf.get(YarnConfiguration.TIMELINE_SERVICE_WEBAPP_ADDRESS,
YarnConfiguration.DEFAULT_TIMELINE_SERVICE_WEBAPP_ADDRESS);
}
+
+String defaultAuth = UserGroupInformation.isSecurityEnabled() ?
+KerberosAuthenticationHandler.TYPE :
+PseudoAuthenticationHandler.TYPE;
+authType = conf.get(YarnConfiguration.TIMELINE_HTTP_AUTH_TYPE,
+defaultAuth);
LOG.info("Timeline service address: " + getTimelineServiceAddress());
super.serviceInit(conf);
}
@@ -193,6 +201,12 @@ public class TimelineClientImpl extends TimelineClient {
@Override
public Token getDelegationToken(
final String renewer) throws IOException, YarnException {
+if(authType.equals(PseudoAuthenticationHandler.TYPE)) {
+ LOG.info("Skipping get timeline delegation token since authType="
+ + PseudoAuthenticationHandler.TYPE);
+ // Null tokens are ignored by YarnClient so this is safe
+ return null;
+}
PrivilegedExceptionAction>
getDTAction =
new
PrivilegedExceptionAction>() {
@@ -219,6 +233,12 @@ public class TimelineClientImpl extends TimelineClient {
public long renewDelegationToken(
final Token timelineDT)
throws IOException, YarnException {
+if(authType.equals(PseudoAuthenticationHandler.TYPE)) {
+ LOG.info("Skipping renew timeline delegation token since authType="
+ + PseudoAuthenticationHandler.TYPE);
+ // RM will skip renew if expirytime less than 0
+ return -1;
+}
final boolean isTokenServiceAddrEmpty =
timelineDT.getService().toString().isEmpty();
final String scheme = isTokenServiceAddrEmpty ? null
@@ -257,6 +277,11 @@ public class TimelineClientImpl extends TimelineClient {
public void cancelDelegationToken(
final Token timelineDT)
throws IOException, YarnException {
+if(authType.equals(PseudoAuthenticationHandler.TYPE)) {
+ LOG.info("Skipping cancel timeline delegation token since authType="
+ + PseudoAuthenticationHandler.TYPE);
+ return;
+}
final boolean isTokenServiceAddrEmpty =
timelineDT.getService().toString().isEmpty();
final String scheme = isTokenServiceAddrEmpty ? null
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClient.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClient.java
index 4d9c320..e7110dd 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClient.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/te
[hadoop] branch trunk updated: YARN-10789. RM HA startup can fail due to race conditions in ZKConfigurationStore. Contributed by Tarun Parimi
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 7003997 YARN-10789. RM HA startup can fail due to race conditions in
ZKConfigurationStore. Contributed by Tarun Parimi
7003997 is described below
commit 7003997e36014974eee15f2796c5ad7a1436d430
Author: Szilard Nemeth
AuthorDate: Sat Jun 12 14:49:52 2021 +0200
YARN-10789. RM HA startup can fail due to race conditions in
ZKConfigurationStore. Contributed by Tarun Parimi
---
.../capacity/conf/ZKConfigurationStore.java| 25 +-
1 file changed, 20 insertions(+), 5 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java
index 7276d3d..703193b 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java
@@ -19,6 +19,7 @@
package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf;
import
org.apache.hadoop.thirdparty.com.google.common.annotations.VisibleForTesting;
+import org.apache.zookeeper.KeeperException.NodeExistsException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.hadoop.conf.Configuration;
@@ -61,7 +62,8 @@ public class ZKConfigurationStore extends
YarnConfigurationStore {
private static final String CONF_STORE_PATH = "CONF_STORE";
private static final String FENCING_PATH = "FENCING";
private static final String CONF_VERSION_PATH = "CONF_VERSION";
-
+ private static final String NODEEXISTS_MSG = "Encountered NodeExists error."
+ + " Skipping znode creation since another RM has already created it";
private String zkVersionPath;
private String logsPath;
private String confStorePath;
@@ -92,7 +94,11 @@ public class ZKConfigurationStore extends
YarnConfigurationStore {
this.fencingNodePath = getNodePath(znodeParentPath, FENCING_PATH);
this.confVersionPath = getNodePath(znodeParentPath, CONF_VERSION_PATH);
-zkManager.createRootDirRecursively(znodeParentPath, zkAcl);
+try {
+ zkManager.createRootDirRecursively(znodeParentPath, zkAcl);
+} catch(NodeExistsException e) {
+ LOG.warn(NODEEXISTS_MSG, e);
+}
zkManager.delete(fencingNodePath);
if (createNewZkPath(logsPath)) {
@@ -244,7 +250,12 @@ public class ZKConfigurationStore extends
YarnConfigurationStore {
*/
private boolean createNewZkPath(String path) throws Exception {
if (!zkManager.exists(path)) {
- zkManager.create(path);
+ try {
+zkManager.create(path);
+ } catch(NodeExistsException e) {
+LOG.warn(NODEEXISTS_MSG, e);
+return false;
+ }
return true;
} else {
return false;
@@ -279,8 +290,12 @@ public class ZKConfigurationStore extends
YarnConfigurationStore {
@VisibleForTesting
protected void safeCreateZkData(String path, byte[] data) throws Exception {
-zkManager.safeCreate(path, data, zkAcl, CreateMode.PERSISTENT,
-zkAcl, fencingNodePath);
+try {
+ zkManager.safeCreate(path, data, zkAcl, CreateMode.PERSISTENT,
+ zkAcl, fencingNodePath);
+} catch(NodeExistsException e) {
+ LOG.warn(NODEEXISTS_MSG, e);
+}
}
private static String getNodePath(String root, String nodeName) {
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10796. Capacity Scheduler: dynamic queue cannot scale out properly if its capacity is 0%. Contributed by Peter Bacsko
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 200eec8 YARN-10796. Capacity Scheduler: dynamic queue cannot scale
out properly if its capacity is 0%. Contributed by Peter Bacsko
200eec8 is described below
commit 200eec8f2eaab29e2feddc7c3d38ab8f215bd36d
Author: Szilard Nemeth
AuthorDate: Thu Jun 3 13:27:36 2021 +0200
YARN-10796. Capacity Scheduler: dynamic queue cannot scale out properly if
its capacity is 0%. Contributed by Peter Bacsko
---
.../scheduler/capacity/UsersManager.java | 24 +++--
.../scheduler/capacity/TestUsersManager.java | 109 +
2 files changed, 126 insertions(+), 7 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/UsersManager.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/UsersManager.java
index 6f7d8f6..cf9dead 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/UsersManager.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/UsersManager.java
@@ -702,7 +702,8 @@ public class UsersManager implements AbstractUsersManager {
activeUsersWithOnlyPendingApps = new AtomicInteger(numPendingUsers);
}
- private Resource computeUserLimit(String userName, Resource clusterResource,
+ @VisibleForTesting
+ Resource computeUserLimit(String userName, Resource clusterResource,
String nodePartition, SchedulingMode schedulingMode, boolean activeUser)
{
Resource partitionResource = labelManager.getResourceByLabel(nodePartition,
clusterResource);
@@ -716,6 +717,7 @@ public class UsersManager implements AbstractUsersManager {
* (which extra resources we are allocating)
*/
Resource queueCapacity = lQueue.getEffectiveCapacity(nodePartition);
+Resource originalCapacity = queueCapacity;
/*
* Assume we have required resource equals to minimumAllocation, this can
@@ -791,16 +793,19 @@ public class UsersManager implements AbstractUsersManager
{
// IGNORE_PARTITION_EXCLUSIVITY allocation.
Resource maxUserLimit = Resources.none();
if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) {
- // If user-limit-factor set to -1, we should disabled user limit.
- if (getUserLimitFactor() != -1) {
-maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity,
-getUserLimitFactor());
- } else {
+ if (getUserLimitFactor() == -1 ||
+ originalCapacity.equals(Resources.none())) {
+// If user-limit-factor set to -1, we should disable user limit.
+//
+// Also prevent incorrect maxUserLimit due to low queueCapacity
+// Can happen if dynamic queue has capacity = 0%
maxUserLimit = lQueue.
getEffectiveMaxCapacityDown(
nodePartition, lQueue.getMinimumAllocation());
+ } else {
+maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity,
+getUserLimitFactor());
}
-
} else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY) {
maxUserLimit = partitionResource;
}
@@ -1131,4 +1136,9 @@ public class UsersManager implements AbstractUsersManager
{
public int getNumActiveUsersWithOnlyPendingApps() {
return activeUsersWithOnlyPendingApps.get();
}
+
+ @VisibleForTesting
+ void setUsageRatio(String label, float usage) {
+qUsageRatios.usageRatios.put(label, usage);
+ }
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUsersManager.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUsersManager.java
new file mode 100644
index 000..5b79ee2
--- /dev/null
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUsersManager.java
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache
[hadoop] branch trunk updated: YARN-10787. Queue submit ACL check is wrong when CS queue is ambiguous. Contributed by Gergely Pollak
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 2707f69 YARN-10787. Queue submit ACL check is wrong when CS queue is
ambiguous. Contributed by Gergely Pollak
2707f69 is described below
commit 2707f69251dbfec6448c6b23e0ad086bdb648878
Author: Szilard Nemeth
AuthorDate: Tue Jun 1 15:30:27 2021 +0200
YARN-10787. Queue submit ACL check is wrong when CS queue is ambiguous.
Contributed by Gergely Pollak
---
.../yarn/server/resourcemanager/RMAppManager.java | 26 +++--
.../TestCapacitySchedulerAmbiguousLeafs.java | 123 +
2 files changed, 137 insertions(+), 12 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
index 25ff384..d5297a2 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
@@ -84,9 +84,9 @@ import
org.apache.hadoop.thirdparty.com.google.common.util.concurrent.SettableFu
import org.apache.hadoop.yarn.util.StringHelper;
/**
- * This class manages the list of applications for the resource manager.
+ * This class manages the list of applications for the resource manager.
*/
-public class RMAppManager implements EventHandler,
+public class RMAppManager implements EventHandler,
Recoverable {
private static final Logger LOG =
@@ -143,7 +143,7 @@ public class RMAppManager implements
EventHandler,
static final Logger LOG = LoggerFactory.
getLogger(ApplicationSummary.class);
-// Escape sequences
+// Escape sequences
static final char EQUALS = '=';
static final char[] charsToEscape =
{StringUtils.COMMA, EQUALS, StringUtils.ESCAPE_CHAR};
@@ -182,7 +182,7 @@ public class RMAppManager implements
EventHandler,
/**
* create a summary of the application's runtime.
- *
+ *
* @param app {@link RMApp} whose summary is to be created, cannot
*be null.
*/
@@ -247,7 +247,7 @@ public class RMAppManager implements
EventHandler,
/**
* Log a summary of the application's runtime.
- *
+ *
* @param app {@link RMApp} whose summary is to be logged
*/
public static void logAppSummary(RMApp app) {
@@ -274,7 +274,7 @@ public class RMAppManager implements
EventHandler,
}
protected synchronized int getCompletedAppsListSize() {
-return this.completedApps.size();
+return this.completedApps.size();
}
protected synchronized void finishApplication(ApplicationId applicationId) {
@@ -285,7 +285,7 @@ public class RMAppManager implements
EventHandler,
if (UserGroupInformation.isSecurityEnabled()) {
rmContext.getDelegationTokenRenewer().applicationFinished(applicationId);
}
-
+
completedApps.add(applicationId);
completedAppsInStateStore++;
writeAuditLog(applicationId);
@@ -297,26 +297,26 @@ public class RMAppManager implements
EventHandler,
String operation = "UNKONWN";
boolean success = false;
switch (app.getState()) {
- case FAILED:
+ case FAILED:
operation = AuditConstants.FINISH_FAILED_APP;
break;
case FINISHED:
operation = AuditConstants.FINISH_SUCCESS_APP;
success = true;
break;
- case KILLED:
+ case KILLED:
operation = AuditConstants.FINISH_KILLED_APP;
success = true;
break;
default:
break;
}
-
+
if (success) {
RMAuditLogger.logSuccess(app.getUser(), operation,
"RMAppManager", app.getApplicationId());
} else {
- StringBuilder diag = app.getDiagnostics();
+ StringBuilder diag = app.getDiagnostics();
String msg = diag == null ? null : diag.toString();
RMAuditLogger.logFailure(app.getUser(), operation, msg, "RMAppManager",
"App failed with state: " + app.getState(), appId);
@@ -445,7 +445,9 @@ public class RMAppManager implements
EventHandler,
if (!isRecovery && YarnConfiguration.isAclEnabled(conf)) {
if (scheduler instanceof CapacityScheduler) {
-String queueName = submissionContext.getQueue();
+String queueName = placementContext == null ?
+
[hadoop] branch trunk updated: YARN-10782. Extend /scheduler endpoint with template properties. Contributed by Andras Gyori
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new b86a6eb YARN-10782. Extend /scheduler endpoint with template
properties. Contributed by Andras Gyori
b86a6eb is described below
commit b86a6eb8715e0d722e577fc97ced82c3795ad2d2
Author: Szilard Nemeth
AuthorDate: Tue May 25 18:27:53 2021 +0200
YARN-10782. Extend /scheduler endpoint with template properties.
Contributed by Andras Gyori
---
.../dao/AutoQueueTemplatePropertiesInfo.java | 45 ++
.../webapp/dao/CapacitySchedulerInfo.java | 3 ++
.../webapp/dao/CapacitySchedulerQueueInfo.java | 6 ++-
.../dao/helper/CapacitySchedulerInfoHelper.java| 17
.../webapp/TestRMWebServicesCapacitySched.java | 6 +--
...estRMWebServicesCapacitySchedDynamicConfig.java | 43 -
.../TestRMWebServicesForCSWithPartitions.java | 2 +-
7 files changed, 115 insertions(+), 7 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AutoQueueTemplatePropertiesInfo.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AutoQueueTemplatePropertiesInfo.java
new file mode 100644
index 000..43c9ff2
--- /dev/null
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AutoQueueTemplatePropertiesInfo.java
@@ -0,0 +1,45 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.webapp.dao;
+
+import
org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.LeafQueueTemplateInfo.ConfItem;
+
+import javax.xml.bind.annotation.XmlAccessType;
+import javax.xml.bind.annotation.XmlAccessorType;
+import javax.xml.bind.annotation.XmlRootElement;
+import java.util.ArrayList;
+
+@XmlRootElement
+@XmlAccessorType(XmlAccessType.FIELD)
+public class AutoQueueTemplatePropertiesInfo {
+
+ private ArrayList property =
+ new ArrayList<>();
+
+ public AutoQueueTemplatePropertiesInfo() {
+ }
+
+ public ArrayList getProperty() {
+return property;
+ }
+
+ public void add(ConfItem confItem) {
+property.add(confItem);
+ }
+}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
index 78e2133..cc02c23 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
@@ -62,6 +62,7 @@ public class CapacitySchedulerInfo extends SchedulerInfo {
protected String creationMethod;
protected String autoCreationEligibility;
protected String defaultNodeLabelExpression;
+ protected AutoQueueTemplatePropertiesInfo autoQueueTemplateProperties;
@XmlTransient
static final float EPSILON = 1e-8f;
@@ -109,6 +110,8 @@ public class CapacitySchedulerInfo extends SchedulerInfo {
if (parent instanceof ParentQueue) {
orderingPolicyInfo = ((ParentQueue) parent).getQueueOrderingPolicy()
.getConfigName();
+ autoQueueTemplateProperties = CapacitySchedulerInfoHelper
+.getAutoCreatedTemplate((ParentQueue) parent);
}
mode = CapacitySchedulerInfoHelper.getMode(parent);
queueType = CapacitySchedulerInfoHelper.getQueueType(parent);
diff --git
a/
[hadoop] branch trunk updated: YARN-10783. Allow definition of auto queue template properties in root. Contributed by Andras Gyori
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 2541efa YARN-10783. Allow definition of auto queue template
properties in root. Contributed by Andras Gyori
2541efa is described below
commit 2541efa496ba0e7e096ee5ec3c08d64b62036402
Author: Szilard Nemeth
AuthorDate: Tue May 25 13:55:59 2021 +0200
YARN-10783. Allow definition of auto queue template properties in root.
Contributed by Andras Gyori
---
.../scheduler/capacity/AutoCreatedQueueTemplate.java | 14 +++---
.../scheduler/capacity/TestAutoCreatedQueueTemplate.java | 11 +++
2 files changed, 22 insertions(+), 3 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AutoCreatedQueueTemplate.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AutoCreatedQueueTemplate.java
index 6c516c0..203ec4d 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AutoCreatedQueueTemplate.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AutoCreatedQueueTemplate.java
@@ -29,6 +29,7 @@ import java.util.Map;
import java.util.Set;
import static
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.AUTO_QUEUE_CREATION_V2_PREFIX;
+import static
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.ROOT;
/**
* A handler for storing and setting auto created queue template settings.
@@ -68,6 +69,10 @@ public class AutoCreatedQueueTemplate {
*/
public void setTemplateEntriesForChild(Configuration conf,
String childQueuePath) {
+if (childQueuePath.equals(ROOT)) {
+ return;
+}
+
// Get all properties that are explicitly set
Set alreadySetProps = conf.getPropsWithPrefix(
CapacitySchedulerConfiguration.getQueuePrefix(childQueuePath)).keySet();
@@ -94,8 +99,8 @@ public class AutoCreatedQueueTemplate {
List queuePathParts = new ArrayList<>(Arrays.asList(
queuePath.split("\\.")));
-if (queuePathParts.size() <= 1) {
- // This is either root or an empty queue name
+if (queuePathParts.size() <= 1 && !queuePath.equals(ROOT)) {
+ // This is an invalid queue path
return;
}
int queuePathMaxIndex = queuePathParts.size() - 1;
@@ -106,7 +111,10 @@ public class AutoCreatedQueueTemplate {
// MAX_WILDCARD_LEVEL will be configurable in the future
int supportedWildcardLevel = Math.min(queuePathMaxIndex - 1,
MAX_WILDCARD_LEVEL);
-
+// Allow root to have template properties
+if (queuePath.equals(ROOT)) {
+ supportedWildcardLevel = 0;
+}
// Collect all template entries
while (wildcardLevel <= supportedWildcardLevel) {
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestAutoCreatedQueueTemplate.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestAutoCreatedQueueTemplate.java
index 1c021f6..2763af0 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestAutoCreatedQueueTemplate.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestAutoCreatedQueueTemplate.java
@@ -109,6 +109,17 @@ public class TestAutoCreatedQueueTemplate {
newConf.getNonLabeledQueueWeight(TEST_QUEUE_ABC), 10e-6);
}
+ @Test
+ public void testRootTemplate() {
+conf.set(getTemplateKey("root", "capacity"), "2w");
+
+AutoCreatedQueueTemplate template =
+new AutoCreatedQueueTemplate(conf, ROOT);
+template.setTemplateEntriesForChild(conf, TEST_QUEUE_A);
+Assert.assertEquals("root property is not set", 2f,
+conf.getNonLabeledQueueWeight(TEST_QUEUE_A), 10e-6);
+ }
+
private String getTemplateKey(String queuePath, String entryKey) {
return CapacitySchedulerConfiguration.getQueuePrefix(queuePath)
[hadoop] branch trunk updated: YARN-10753. Document the removal of FS default queue creation. Contributed by Benjamin Teke
This is an automated email from the ASF dual-hosted git repository. snemeth pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 6bb0892 YARN-10753. Document the removal of FS default queue creation. Contributed by Benjamin Teke 6bb0892 is described below commit 6bb0892f6edc112623ed8fb56d0a5914af9dd238 Author: Szilard Nemeth AuthorDate: Sat May 22 15:00:08 2021 +0200 YARN-10753. Document the removal of FS default queue creation. Contributed by Benjamin Teke --- .../hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md index 7a11279..f0a43a5 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md @@ -63,7 +63,7 @@ Customizing the Fair Scheduler typically involves altering two files. First, sch | Property | Description | |: |: | | `yarn.scheduler.fair.allocation.file` | Path to allocation file. An allocation file is an XML manifest describing queues and their properties, in addition to certain policy defaults. This file must be in the XML format described in the next section. If a relative path is given, the file is searched for on the classpath (which typically includes the Hadoop conf directory). Defaults to fair-scheduler.xml. | -| `yarn.scheduler.fair.user-as-default-queue` | Whether to use the username associated with the allocation as the default queue name, in the event that a queue name is not specified. If this is set to "false" or unset, all jobs have a shared default queue, named "default". Defaults to true. If a queue placement policy is given in the allocations file, this property is ignored. | +| `yarn.scheduler.fair.user-as-default-queue` | Whether to use the username associated with the allocation as the default queue name, in the event that a queue name is not specified. If this is set to "false" or unset, all jobs have a shared default queue, named "default". Defaults to true. If a queue placement policy is given in the allocations file, this property is ignored. **Note:** If this is set to false, the "default" queue needs to be declared in the allocations file. | | `yarn.scheduler.fair.preemption` | Whether to use preemption. Defaults to false. | | `yarn.scheduler.fair.preemption.cluster-utilization-threshold` | The utilization threshold after which preemption kicks in. The utilization is computed as the maximum ratio of usage to capacity among all resources. Defaults to 0.8f. | | `yarn.scheduler.fair.sizebasedweight` | Whether to assign shares to individual apps based on their size, rather than providing an equal share to all apps regardless of size. When set to true, apps are weighted by the natural logarithm of one plus the app's total requested memory, divided by the natural logarithm of 2. Defaults to false. | @@ -72,7 +72,7 @@ Customizing the Fair Scheduler typically involves altering two files. First, sch | `yarn.scheduler.fair.max.assign` | If assignmultiple is true and dynamic.max.assign is false, the maximum amount of containers that can be assigned in one heartbeat. Defaults to -1, which sets no limit. | | `yarn.scheduler.fair.locality.threshold.node` | For applications that request containers on particular nodes, the number of scheduling opportunities since the last container assignment to wait before accepting a placement on another node. Expressed as a float between 0 and 1, which, as a fraction of the cluster size, is the number of scheduling opportunities to pass up. The default value of -1.0 means don't pass up any scheduling opportunities. | | `yarn.scheduler.fair.locality.threshold.rack` | For applications that request containers on particular racks, the number of scheduling opportunities since the last container assignment to wait before accepting a placement on another rack. Expressed as a float between 0 and 1, which, as a fraction of the cluster size, is the number of scheduling opportunities to pass up. The default value of -1.0 means don't pass up any scheduling opportunities. | -| `yarn.scheduler.fair.allow-undeclared-pools` | If this is true, new queues can be created at application submission time, whether because they are specified as the application's queue by the submitter or because they are placed there by the user-as-default-queue property. If this is false, any time an app would be placed in a queue that is not specified in the allocations file, it is placed in the "default"
[hadoop] branch trunk updated: YARN-7769. FS QueueManager should not create default queue at init. Contributed by Benjamin Teke
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 1e44bdb YARN-7769. FS QueueManager should not create default queue at
init. Contributed by Benjamin Teke
1e44bdb is described below
commit 1e44bdb84ca01288e48aeecc814d8378e99bb5ae
Author: Szilard Nemeth
AuthorDate: Sat May 22 14:55:01 2021 +0200
YARN-7769. FS QueueManager should not create default queue at init.
Contributed by Benjamin Teke
---
.../scheduler/fair/QueueManager.java | 4 --
.../applicationsmanager/TestAMRestart.java | 10 +++-
.../scheduler/fair/TestAppRunnability.java | 6 +--
.../scheduler/fair/TestFSLeafQueue.java| 2 +-
.../scheduler/fair/TestFairScheduler.java | 58 ++
.../scheduler/fair/TestFairSchedulerFairShare.java | 3 ++
.../scheduler/fair/TestQueueManager.java | 7 +--
.../fair/converter/TestFSQueueConverter.java | 2 +-
.../TestRMWebServicesFairScheduler.java| 2 +-
...ebServicesFairSchedulerCustomResourceTypes.java | 5 +-
10 files changed, 60 insertions(+), 39 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
index d620077..62f63bb 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
@@ -107,10 +107,6 @@ public class QueueManager {
rootQueue.setDynamic(false);
queues.put(rootQueue.getName(), rootQueue);
-// Create the default queue
-FSLeafQueue defaultQueue =
-getLeafQueue(YarnConfiguration.DEFAULT_QUEUE_NAME, true);
-defaultQueue.setDynamic(false);
// Recursively reinitialize to propagate queue properties
rootQueue.reinit(true);
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java
index 46311e2..b2af776 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java
@@ -20,6 +20,7 @@ package
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager;
import java.io.IOException;
import java.util.ArrayList;
+import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
@@ -1313,8 +1314,13 @@ public class TestAMRestart extends
ParameterizedSchedulerTestBase {
Assert.assertEquals(0,
queue.getQueueResourceUsage().getUsed().getVirtualCores());
} else if (getSchedulerType() == SchedulerType.FAIR) {
- FSLeafQueue queue = ((FairScheduler) scheduler).getQueueManager()
- .getLeafQueue("root.default", false);
+ // The default queue is not auto created after YARN-7769 so
+ // user-as-default-queue option is used
+ Collection queues = ((FairScheduler) scheduler)
+ .getQueueManager().getLeafQueues();
+ Assert.assertEquals(1, queues.size());
+
+ FSLeafQueue queue = queues.iterator().next();
Assert.assertEquals(0, queue.getResourceUsage().getMemorySize());
Assert.assertEquals(0, queue.getResourceUsage().getVirtualCores());
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAppRunnability.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAppRunnability.java
index 9777e14..be685ee 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAppRunnability.j
[hadoop] branch branch-3.3 updated: YARN-10123. Error message around yarn app -stop/start can be improved. Contributed by Siddharth Ahuja
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new 6777acb YARN-10123. Error message around yarn app -stop/start can be
improved. Contributed by Siddharth Ahuja
6777acb is described below
commit 6777acb08ca7a81e77348016e79f8da2d15784a3
Author: Szilard Nemeth
AuthorDate: Sat May 22 14:36:51 2021 +0200
YARN-10123. Error message around yarn app -stop/start can be improved.
Contributed by Siddharth Ahuja
---
.../apache/hadoop/yarn/client/api/AppAdminClient.java | 4 +++-
.../hadoop-yarn-site/src/site/markdown/YarnCommands.md | 18 +-
2 files changed, 12 insertions(+), 10 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/AppAdminClient.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/AppAdminClient.java
index 2fd59a8..b9f7248 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/AppAdminClient.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/AppAdminClient.java
@@ -74,7 +74,9 @@ public abstract class AppAdminClient extends CompositeService
{
}
if (!clientClassMap.containsKey(appType)) {
throw new IllegalArgumentException("App admin client class name not " +
- "specified for type " + appType);
+ "specified for type " + appType + ". Please ensure " + appType +
+ " framework has provided a client implementation to handle" +
+ " this functionality.");
}
String clientClassName = clientClassMap.get(appType);
Class clientClass;
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md
index 0f2b379..d05c6cd 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md
@@ -51,21 +51,21 @@ Usage: `yarn app [options] `
| -changeQueue \ | Moves application to a new queue.
ApplicationId can be passed using 'appId' option. 'movetoqueue' command is
deprecated, this new command 'changeQueue' performs same functionality. |
| -component \ \ | Works with -flex option to change
the number of components/containers running for an application / long-running
service. Supports absolute or relative changes, such as +1, 2, or -3. |
| -components \ | Works with -upgrade option to trigger the
upgrade of specified components of the application. Multiple components should
be separated by commas. |
-| -decommission \ | Decommissions component instances for
an application / long-running service. Requires -instances option. Supports
-appTypes option to specify which client implementation to use. |
-| -destroy \ | Destroys a saved application specification
and removes all application data permanently. Supports -appTypes option to
specify which client implementation to use. |
-| -enableFastLaunch | Uploads AM dependencies to HDFS to make future launches
faster. Supports -appTypes option to specify which client implementation to
use. |
-| -flex \ | Changes number of running containers for
a component of an application / long-running service. Requires -component
option. If name is provided, appType must be provided unless it is the default
yarn-service. If ID is provided, the appType will be looked up. Supports
-appTypes option to specify which client implementation to use. |
+| -decommission \ | Decommissions component instances for
an application / long-running service. Requires -instances option. Supports
-appTypes option to specify which client implementation to use. Please ensure
the framework corresponding to appType has provided the appropriate client
implementation to handle this specific functionality. |
+| -destroy \ | Destroys a saved application specification
and removes all application data permanently. Supports -appTypes option to
specify which client implementation to use. Please ensure the framework
corresponding to appType has provided the appropriate client implementation to
handle this specific functionality. |
+| -enableFastLaunch | Uploads AM dependencies to HDFS to make future launches
faster. Supports -appTypes option to specify which client implementation to
use. Please ensure the framework corresponding to appType has provided the
appropriate client implementation to handle this specific functionality. |
+| -flex \ | Changes number of running containers for
a component of an application / long-running
[hadoop] branch branch-3.2 updated: YARN-10123. Error message around yarn app -stop/start can be improved. Contributed by Siddharth Ahuja
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 3951fb6 YARN-10123. Error message around yarn app -stop/start can be
improved. Contributed by Siddharth Ahuja
3951fb6 is described below
commit 3951fb6dea172cf669cf3c5a19ffa7486aeccc6f
Author: Szilard Nemeth
AuthorDate: Sat May 22 14:35:47 2021 +0200
YARN-10123. Error message around yarn app -stop/start can be improved.
Contributed by Siddharth Ahuja
---
.../apache/hadoop/yarn/client/api/AppAdminClient.java | 4 +++-
.../hadoop-yarn-site/src/site/markdown/YarnCommands.md | 18 +-
2 files changed, 12 insertions(+), 10 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/AppAdminClient.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/AppAdminClient.java
index b0d737a..a70c5f3 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/AppAdminClient.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/AppAdminClient.java
@@ -74,7 +74,9 @@ public abstract class AppAdminClient extends CompositeService
{
}
if (!clientClassMap.containsKey(appType)) {
throw new IllegalArgumentException("App admin client class name not " +
- "specified for type " + appType);
+ "specified for type " + appType + ". Please ensure " + appType +
+ " framework has provided a client implementation to handle" +
+ " this functionality.");
}
String clientClassName = clientClassMap.get(appType);
Class clientClass;
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md
index 44819d9..3617bb6 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md
@@ -51,21 +51,21 @@ Usage: `yarn app [options] `
| -changeQueue \ | Moves application to a new queue.
ApplicationId can be passed using 'appId' option. 'movetoqueue' command is
deprecated, this new command 'changeQueue' performs same functionality. |
| -component \ \ | Works with -flex option to change
the number of components/containers running for an application / long-running
service. Supports absolute or relative changes, such as +1, 2, or -3. |
| -components \ | Works with -upgrade option to trigger the
upgrade of specified components of the application. Multiple components should
be separated by commas. |
-| -decommission \ | Decommissions component instances for
an application / long-running service. Requires -instances option. Supports
-appTypes option to specify which client implementation to use. |
-| -destroy \ | Destroys a saved application specification
and removes all application data permanently. Supports -appTypes option to
specify which client implementation to use. |
-| -enableFastLaunch | Uploads AM dependencies to HDFS to make future launches
faster. Supports -appTypes option to specify which client implementation to
use. |
-| -flex \ | Changes number of running containers for
a component of an application / long-running service. Requires -component
option. If name is provided, appType must be provided unless it is the default
yarn-service. If ID is provided, the appType will be looked up. Supports
-appTypes option to specify which client implementation to use. |
+| -decommission \ | Decommissions component instances for
an application / long-running service. Requires -instances option. Supports
-appTypes option to specify which client implementation to use. Please ensure
the framework corresponding to appType has provided the appropriate client
implementation to handle this specific functionality. |
+| -destroy \ | Destroys a saved application specification
and removes all application data permanently. Supports -appTypes option to
specify which client implementation to use. Please ensure the framework
corresponding to appType has provided the appropriate client implementation to
handle this specific functionality. |
+| -enableFastLaunch | Uploads AM dependencies to HDFS to make future launches
faster. Supports -appTypes option to specify which client implementation to
use. Please ensure the framework corresponding to appType has provided the
appropriate client implementation to handle this specific functionality. |
+| -flex \ | Changes number of running containers for
a component of an application / long-running
[hadoop] branch trunk updated: YARN-10766. [UI2] Bump moment-timezone to 0.5.33. Contributed by Andras Gyori
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new d146ab3 YARN-10766. [UI2] Bump moment-timezone to 0.5.33. Contributed
by Andras Gyori
d146ab3 is described below
commit d146ab3af71cd3265679ccc51f50e6c88d61b740
Author: Szilard Nemeth
AuthorDate: Fri May 21 18:07:32 2021 +0200
YARN-10766. [UI2] Bump moment-timezone to 0.5.33. Contributed by Andras
Gyori
---
.../hadoop-yarn/hadoop-yarn-ui/src/main/webapp/bower.json | 2 +-
.../hadoop-yarn/hadoop-yarn-ui/src/main/webapp/config/environment.js | 4
.../hadoop-yarn/hadoop-yarn-ui/src/main/webapp/ember-cli-build.js | 2 +-
3 files changed, 6 insertions(+), 2 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/bower.json
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/bower.json
index 9b5cc9f..0ef1367 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/bower.json
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/bower.json
@@ -13,7 +13,7 @@
"qunit": "1.19.0",
"jquery-ui": "1.11.4",
"moment": "2.12.0",
-"moment-timezone": "0.5.1",
+"moment-timezone": "0.5.33",
"more-js": "0.8.2",
"bootstrap": "3.3.6",
"d3": "~3.5.6",
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/config/environment.js
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/config/environment.js
index 3ff3c02..c6ed5ed 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/config/environment.js
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/config/environment.js
@@ -33,6 +33,10 @@ module.exports = function(environment) {
}
},
+moment: {
+ includeTimezone: 'all'
+},
+
APP: DEFAULT_CONFIG,
contentSecurityPolicy: {
'connect-src': "* 'self'",
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/ember-cli-build.js
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/ember-cli-build.js
index c4bb95f..09da39b 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/ember-cli-build.js
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/ember-cli-build.js
@@ -42,7 +42,7 @@ module.exports = function(defaults) {
app.import("bower_components/datatables/media/css/jquery.dataTables.min.css");
app.import("bower_components/datatables/media/js/jquery.dataTables.min.js");
app.import("bower_components/momentjs/min/moment.min.js");
-
app.import("bower_components/moment-timezone/builds/moment-timezone-with-data-2010-2020.min.js");
+
app.import("bower_components/moment-timezone/builds/moment-timezone-with-data-10-year-range.min.js");
app.import("bower_components/select2/dist/css/select2.min.css");
app.import("bower_components/select2/dist/js/select2.min.js");
app.import('bower_components/jquery-ui/jquery-ui.js');
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10779. Add option to disable lowercase conversion in GetApplicationsRequestPBImpl and ApplicationSubmissionContextPBImpl. Contributed by Peter Bacsko
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 43e77ca YARN-10779. Add option to disable lowercase conversion in
GetApplicationsRequestPBImpl and ApplicationSubmissionContextPBImpl.
Contributed by Peter Bacsko
43e77ca is described below
commit 43e77caa713e3a679d2cca399fa4cb75a3c58dd9
Author: Szilard Nemeth
AuthorDate: Fri May 21 18:04:24 2021 +0200
YARN-10779. Add option to disable lowercase conversion in
GetApplicationsRequestPBImpl and ApplicationSubmissionContextPBImpl.
Contributed by Peter Bacsko
---
.../apache/hadoop/yarn/conf/YarnConfiguration.java | 5 +-
.../impl/pb/GetApplicationsRequestPBImpl.java | 28 +++-
.../pb/ApplicationSubmissionContextPBImpl.java | 30 -
.../src/main/resources/yarn-default.xml| 6 ++
.../impl/pb/TestGetApplicationsRequestPBImpl.java | 74 ++
.../pb/TestApplicationSubmissionContextPBImpl.java | 74 ++
6 files changed, 211 insertions(+), 6 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index 833aecc..2ebf79c 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -1954,7 +1954,10 @@ public class YarnConfiguration extends Configuration {
false;
public static final String APPLICATION_TAG_BASED_PLACEMENT_USER_WHITELIST =
APPLICATION_TAG_BASED_PLACEMENT_PREFIX + ".username.whitelist";
-
+ public static final String APPLICATION_TAG_FORCE_LOWERCASE_CONVERSION =
+ APPLICATION_TAG_BASED_PLACEMENT_PREFIX + ".force-lowercase";
+ public static final boolean
DEFAULT_APPLICATION_TAG_FORCE_LOWERCASE_CONVERSION =
+ true;
/** Enable switch for container log monitoring. */
public static final String NM_CONTAINER_LOG_MONITOR_ENABLED =
NM_PREFIX + "container-log-monitor.enable";
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsRequestPBImpl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsRequestPBImpl.java
index 32b576a..97f4ad0 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsRequestPBImpl.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsRequestPBImpl.java
@@ -18,6 +18,9 @@
package org.apache.hadoop.yarn.api.protocolrecords.impl.pb;
+import static
org.apache.hadoop.yarn.conf.YarnConfiguration.APPLICATION_TAG_FORCE_LOWERCASE_CONVERSION;
+import static
org.apache.hadoop.yarn.conf.YarnConfiguration.DEFAULT_APPLICATION_TAG_FORCE_LOWERCASE_CONVERSION;
+
import java.util.EnumSet;
import java.util.HashSet;
import java.util.List;
@@ -25,6 +28,7 @@ import java.util.Set;
import org.apache.commons.lang3.Range;
import org.apache.hadoop.classification.InterfaceAudience.Private;
import org.apache.hadoop.classification.InterfaceStability.Unstable;
+import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.util.StringUtils;
import org.apache.hadoop.yarn.api.protocolrecords.ApplicationsRequestScope;
import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationsRequest;
@@ -33,12 +37,14 @@ import
org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils;
import org.apache.hadoop.yarn.proto.YarnProtos.YarnApplicationStateProto;
import
org.apache.hadoop.yarn.proto.YarnServiceProtos.GetApplicationsRequestProto;
import
org.apache.hadoop.yarn.proto.YarnServiceProtos.GetApplicationsRequestProtoOrBuilder;
-
+import
org.apache.hadoop.thirdparty.com.google.common.annotations.VisibleForTesting;
import org.apache.hadoop.thirdparty.protobuf.TextFormat;
@Private
@Unstable
public class GetApplicationsRequestPBImpl extends GetApplicationsRequest {
+ private static volatile Boolean forceLowerCaseTags;
+
GetApplicationsRequestProto proto =
GetApplicationsRequestProto.getDefaultInstance();
GetApplicationsRequestProto.Builder builder = null;
boolean viaProto = false;
@@ -56,11 +62,23 @@ public class GetApplicationsRequestPBImpl extends
GetApplicationsRequest {
public GetApplicationsRequestPBImpl() {
builder = GetApplicationsRequestProto.newBuilder();
+initLowerCaseConfig();
}
public GetApplicationsRequestPBImpl(Get
[hadoop-site] branch asf-site updated: Add Szilard Nemeth to Hadoop PMC list
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hadoop-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new ff6a19e Add Szilard Nemeth to Hadoop PMC list
ff6a19e is described below
commit ff6a19ed89773757a29599f8151ffaa6041efb49
Author: Szilard Nemeth
AuthorDate: Tue Apr 27 00:15:13 2021 +0200
Add Szilard Nemeth to Hadoop PMC list
---
content/who.html | 8 +++-
src/who.md | 1 +
2 files changed, 8 insertions(+), 1 deletion(-)
diff --git a/content/who.html b/content/who.html
index d88efb2..ca2b567 100644
--- a/content/who.html
+++ b/content/who.html
@@ -663,6 +663,13 @@ order):
-8
+snemeth
+https://github.com/szilard-nemeth";>Szilard Nemeth
+Cloudera
+
++1
+
+
sradia
http://people.apache.org/~sradia";>Sanjay Radia
Hortonworks
@@ -2184,7 +2191,6 @@ if (!doNotTrack) {
ga('send', 'pageview');
}
-
diff --git a/src/who.md b/src/who.md
index 70ea271..2da4250 100644
--- a/src/who.md
+++ b/src/who.md
@@ -106,6 +106,7 @@ order):
sandy|[Sandy Ryza](http://people.apache.org/~sandy)
|Cloudera ||-8
shashikant |Shashikant Banerjee
|Cloudera ||+5.5
sjlee|[Sangjin Lee](http://people.apache.org/~sjlee)
|PayPal ||-8
+ snemeth |[Szilard Nemeth](https://github.com/szilard-nemeth)
|Cloudera ||+1
sradia |[Sanjay Radia](http://people.apache.org/~sradia)
|Hortonworks||-8
sseth|Siddharth Seth
|Hortonworks||-8
stack|Michael Stack
|Apple |HBase |-8
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10123. Error message around yarn app -stop/start can be improved. Contributed by Siddharth Ahuja
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new ebf6b14 YARN-10123. Error message around yarn app -stop/start can be
improved. Contributed by Siddharth Ahuja
ebf6b14 is described below
commit ebf6b14c6794acd89d947c7b1ddb4d8d707b6f45
Author: Szilard Nemeth
AuthorDate: Fri Apr 23 22:48:33 2021 +0200
YARN-10123. Error message around yarn app -stop/start can be improved.
Contributed by Siddharth Ahuja
This is required to highlight that an implementation at framework level is
needed for the stop/start functionality to work.
---
.../apache/hadoop/yarn/client/api/AppAdminClient.java | 4 +++-
.../hadoop-yarn-site/src/site/markdown/YarnCommands.md | 18 +-
2 files changed, 12 insertions(+), 10 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/AppAdminClient.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/AppAdminClient.java
index 2fd59a8..b9f7248 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/AppAdminClient.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/AppAdminClient.java
@@ -74,7 +74,9 @@ public abstract class AppAdminClient extends CompositeService
{
}
if (!clientClassMap.containsKey(appType)) {
throw new IllegalArgumentException("App admin client class name not " +
- "specified for type " + appType);
+ "specified for type " + appType + ". Please ensure " + appType +
+ " framework has provided a client implementation to handle" +
+ " this functionality.");
}
String clientClassName = clientClassMap.get(appType);
Class clientClass;
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md
index 3082ff8..123ff6b 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md
@@ -51,21 +51,21 @@ Usage: `yarn app [options] `
| -changeQueue \ | Moves application to a new queue.
ApplicationId can be passed using 'appId' option. 'movetoqueue' command is
deprecated, this new command 'changeQueue' performs same functionality. |
| -component \ \ | Works with -flex option to change
the number of components/containers running for an application / long-running
service. Supports absolute or relative changes, such as +1, 2, or -3. |
| -components \ | Works with -upgrade option to trigger the
upgrade of specified components of the application. Multiple components should
be separated by commas. |
-| -decommission \ | Decommissions component instances for
an application / long-running service. Requires -instances option. Supports
-appTypes option to specify which client implementation to use. |
-| -destroy \ | Destroys a saved application specification
and removes all application data permanently. Supports -appTypes option to
specify which client implementation to use. |
-| -enableFastLaunch | Uploads AM dependencies to HDFS to make future launches
faster. Supports -appTypes option to specify which client implementation to
use. |
-| -flex \ | Changes number of running containers for
a component of an application / long-running service. Requires -component
option. If name is provided, appType must be provided unless it is the default
yarn-service. If ID is provided, the appType will be looked up. Supports
-appTypes option to specify which client implementation to use. |
+| -decommission \ | Decommissions component instances for
an application / long-running service. Requires -instances option. Supports
-appTypes option to specify which client implementation to use. Please ensure
the framework corresponding to appType has provided the appropriate client
implementation to handle this specific functionality. |
+| -destroy \ | Destroys a saved application specification
and removes all application data permanently. Supports -appTypes option to
specify which client implementation to use. Please ensure the framework
corresponding to appType has provided the appropriate client implementation to
handle this specific functionality. |
+| -enableFastLaunch | Uploads AM dependencies to HDFS to make future launches
faster. Supports -appTypes option to specify which client implementation to
use. Please ensure the framework corresponding to appType has provided the
appropriate client implementation to handle this sp
[hadoop] branch trunk updated: YARN-10750. TestMetricsInvariantChecker.testManyRuns is broken since HADOOP-17524. Contributed by Gergely Pollak
This is an automated email from the ASF dual-hosted git repository. snemeth pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 3b6b21e YARN-10750. TestMetricsInvariantChecker.testManyRuns is broken since HADOOP-17524. Contributed by Gergely Pollak 3b6b21e is described below commit 3b6b21e459c6d91c93f6aaa32bde53047a0f536f Author: Szilard Nemeth AuthorDate: Fri Apr 23 16:17:29 2021 +0200 YARN-10750. TestMetricsInvariantChecker.testManyRuns is broken since HADOOP-17524. Contributed by Gergely Pollak --- .../src/test/resources/invariants.txt | 4 .../src/test/resources/invariants_jdk9.txt| 4 2 files changed, 8 deletions(-) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/invariants.txt b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/invariants.txt index 363ed0d..a77fdf0 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/invariants.txt +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/invariants.txt @@ -48,7 +48,3 @@ ThreadsBlocked >= 0 ThreadsWaiting >= 0 ThreadsTimedWaiting >= 0 ThreadsTerminated >= 0 -LogFatal >= 0 -LogError >= 0 -LogWarn >= 0 -LogInfo >= 0 diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/invariants_jdk9.txt b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/invariants_jdk9.txt index e081334..b40a7aa 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/invariants_jdk9.txt +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/invariants_jdk9.txt @@ -48,7 +48,3 @@ ThreadsBlocked >= 0 ThreadsWaiting >= 0 ThreadsTimedWaiting >= 0 ThreadsTerminated >= 0 -LogFatal >= 0 -LogError >= 0 -LogWarn >= 0 -LogInfo >= 0 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10746. RmWebApp add default-node-label-expression to the queue info. Contributed by Gergely Pollak
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 919daec YARN-10746. RmWebApp add default-node-label-expression to the
queue info. Contributed by Gergely Pollak
919daec is described below
commit 919daec36b23457c107e819855791e7e6d144594
Author: Szilard Nemeth
AuthorDate: Fri Apr 23 16:12:12 2021 +0200
YARN-10746. RmWebApp add default-node-label-expression to the queue info.
Contributed by Gergely Pollak
---
.../webapp/dao/CapacitySchedulerInfo.java | 3 ++
.../webapp/dao/CapacitySchedulerLeafQueueInfo.java | 12 ++---
.../webapp/dao/CapacitySchedulerQueueInfo.java | 8 +++-
.../webapp/TestRMWebServicesCapacitySched.java | 52 ++
4 files changed, 65 insertions(+), 10 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
index fc18d4d..78e2133 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
@@ -61,6 +61,7 @@ public class CapacitySchedulerInfo extends SchedulerInfo {
protected String queueType;
protected String creationMethod;
protected String autoCreationEligibility;
+ protected String defaultNodeLabelExpression;
@XmlTransient
static final float EPSILON = 1e-8f;
@@ -114,6 +115,8 @@ public class CapacitySchedulerInfo extends SchedulerInfo {
creationMethod = CapacitySchedulerInfoHelper.getCreationMethod(parent);
autoCreationEligibility = CapacitySchedulerInfoHelper
.getAutoCreationEligibility(parent);
+
+defaultNodeLabelExpression = parent.getDefaultNodeLabelExpression();
}
public float getCapacity() {
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java
index 6aef01b..16abb99 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java
@@ -52,7 +52,6 @@ public class CapacitySchedulerLeafQueueInfo extends
CapacitySchedulerQueueInfo {
protected ResourceInfo userAMResourceLimit;
protected boolean preemptionDisabled;
protected boolean intraQueuePreemptionDisabled;
- protected String defaultNodeLabelExpression;
protected int defaultPriority;
protected boolean isAutoCreatedLeafQueue;
protected long maxApplicationLifetime;
@@ -81,7 +80,6 @@ public class CapacitySchedulerLeafQueueInfo extends
CapacitySchedulerQueueInfo {
intraQueuePreemptionDisabled = q.getIntraQueuePreemptionDisabled();
orderingPolicyDisplayName = q.getOrderingPolicy().getInfo();
orderingPolicyInfo = q.getOrderingPolicy().getConfigName();
-defaultNodeLabelExpression = q.getDefaultNodeLabelExpression();
defaultPriority = q.getDefaultApplicationPriority().getPriority();
ArrayList usersList = users.getUsersList();
if (usersList.isEmpty()) {
@@ -148,17 +146,17 @@ public class CapacitySchedulerLeafQueueInfo extends
CapacitySchedulerQueueInfo {
public float getConfiguredMaxAMResourceLimit() {
return configuredMaxAMResourceLimit;
}
-
+
public ResourceInfo getAMResourceLimit() {
return AMResourceLimit;
}
-
+
public ResourceInfo getUsedAMResource() {
return usedAMResource;
}
public ResourceInfo getUserAMResourceLimit() {
-return userAMResourceLimit;
+return userAMResourceLimit;
}
public boolean getPreemptionDisabled() {
@@ -172,10 +170,6 @@ public class CapacitySchedulerLeafQueueInfo extends
CapacitySchedulerQueueInfo {
public String getOrderingPolicyDisplayName() {
return orderingPolicyDisplayName;
}
-
- public String getDefaultNodeLabelExpression() {
-return
[hadoop] branch trunk updated: YARN-10654. Dots '.' in CSMappingRule path variables should be replaced. Contributed by Peter Bacsko
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new f76a2a7 YARN-10654. Dots '.' in CSMappingRule path variables should
be replaced. Contributed by Peter Bacsko
f76a2a7 is described below
commit f76a2a7606760eaf9967dae7ab769d22b11b4447
Author: Szilard Nemeth
AuthorDate: Fri Apr 23 16:07:58 2021 +0200
YARN-10654. Dots '.' in CSMappingRule path variables should be replaced.
Contributed by Peter Bacsko
---
.../placement/CSMappingPlacementRule.java | 22 +--
.../csmappingrule/TestCSMappingPlacementRule.java | 67 +++---
2 files changed, 77 insertions(+), 12 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
index 894bc82..9648769 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
@@ -42,8 +42,6 @@ import java.util.Iterator;
import java.util.List;
import java.util.Set;
-import static
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.DOT;
-
/**
* This class is responsible for making application submissions to queue
* assignments, based on the configured ruleset. This class supports all
@@ -55,6 +53,8 @@ import static
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.C
public class CSMappingPlacementRule extends PlacementRule {
private static final Logger LOG = LoggerFactory
.getLogger(CSMappingPlacementRule.class);
+ private static final String DOT = ".";
+ private static final String DOT_REPLACEMENT = "_dot_";
private CapacitySchedulerQueueManager queueManager;
private List mappingRules;
@@ -194,12 +194,13 @@ public class CSMappingPlacementRule extends PlacementRule
{
return;
}
Iterator it = groupsSet.iterator();
-String primaryGroup = it.next();
+String primaryGroup = cleanName(it.next());
ArrayList secondaryGroupList = new ArrayList<>();
while (it.hasNext()) {
- secondaryGroupList.add(it.next());
+ String groupName = cleanName(it.next());
+ secondaryGroupList.add(groupName);
}
if (secondaryGroupList.size() == 0) {
@@ -226,7 +227,7 @@ public class CSMappingPlacementRule extends PlacementRule {
ApplicationSubmissionContext asc, String user) {
VariableContext vctx = new VariableContext();
-vctx.put("%user", user);
+vctx.put("%user", cleanName(user));
//If the specified matches the default it means NO queue have been
specified
//as per ClientRMService#submitApplication which sets the queue to default
//when no queue is provided.
@@ -522,4 +523,15 @@ public class CSMappingPlacementRule extends PlacementRule {
" mapping rule. Please see the logs for details");
}
}
+
+ private String cleanName(String name) {
+if (name.contains(DOT)) {
+ String converted = name.replaceAll("\\.", DOT_REPLACEMENT);
+ LOG.warn("Name {} is converted to {} when it is used as a queue name.",
+ name, converted);
+ return converted;
+} else {
+ return name;
+}
+ }
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/csmappingrule/TestCSMappingPlacementRule.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/csmappingrule/TestCSMappingPlacementRule.java
index f6bb2a1..0cf1059 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/csmappingrule/TestCSMappingPlacementRule.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/csmappingrule/TestCSMappingPlacementRule.java
@@ -63,13 +63,15 @@ public class TestCSMappingPlacementRule {
@Rule
public TemporaryFolder folder = new TemporaryFolder();
- private Map> user
[hadoop] branch trunk updated (7d6f0ca -> ba3bc53)
This is an automated email from the ASF dual-hosted git repository. snemeth pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git. from 7d6f0ca Revert "HDFS-15884. RBF: Remove unused method getCreateLocation in RouterRpcServer (#2754). Contributed by tomscut." add ba3bc53 YARN-10736. Fix GetApplicationsRequest JavaDoc. Contributed by Miklos Gergely. (#2906) No new revisions were added by this update. Summary of changes: .../hadoop/yarn/api/protocolrecords/GetApplicationsRequest.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10714. Remove dangling dynamic queues on reinitialization. Contributed by Andras Gyori
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 9cd69c2 YARN-10714. Remove dangling dynamic queues on
reinitialization. Contributed by Andras Gyori
9cd69c2 is described below
commit 9cd69c20c4aaad51ce77933ed430b66266156ca7
Author: Szilard Nemeth
AuthorDate: Wed Apr 7 11:52:21 2021 +0200
YARN-10714. Remove dangling dynamic queues on reinitialization. Contributed
by Andras Gyori
---
.../capacity/CapacitySchedulerQueueManager.java| 43 ---
.../TestCapacitySchedulerNewQueueAutoCreation.java | 61 ++
2 files changed, 97 insertions(+), 7 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueManager.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueManager.java
index 00d1cda..5cd1490 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueManager.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueManager.java
@@ -27,8 +27,6 @@ import java.util.List;
import java.util.Map;
import java.util.Set;
-import org.apache.hadoop.yarn.util.resource.ResourceUtils;
-import org.apache.hadoop.yarn.util.resource.Resources;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.hadoop.classification.InterfaceAudience.Private;
@@ -340,11 +338,14 @@ public class CapacitySchedulerQueueManager implements
SchedulerQueueManager<
}
for (CSQueue queue : existingQueues.getQueues()) {
- if (!((AbstractCSQueue) queue).isDynamicQueue() && newQueues.get(
- queue.getQueuePath()) == null && !(
- queue instanceof AutoCreatedLeafQueue && conf
- .isAutoCreateChildQueueEnabled(
- queue.getParent().getQueuePath( {
+ boolean isDanglingDynamicQueue = isDanglingDynamicQueue(
+ newQueues, existingQueues, queue);
+ boolean isRemovable = isDanglingDynamicQueue || !isDynamicQueue(queue)
+ && newQueues.get(queue.getQueuePath()) == null
+ && !(queue instanceof AutoCreatedLeafQueue &&
+
conf.isAutoCreateChildQueueEnabled(queue.getParent().getQueuePath()));
+
+ if (isRemovable) {
existingQueues.remove(queue);
}
}
@@ -435,4 +436,32 @@ public class CapacitySchedulerQueueManager implements
SchedulerQueueManager<
getQueueStateManager() {
return this.queueStateManager;
}
+
+ private boolean isDynamicQueue(CSQueue queue) {
+return (queue instanceof AbstractCSQueue) &&
+((AbstractCSQueue) queue).isDynamicQueue();
+ }
+
+ private boolean isDanglingDynamicQueue(
+ CSQueueStore newQueues, CSQueueStore existingQueues,
+ CSQueue queue) {
+if (!isDynamicQueue(queue)) {
+ return false;
+}
+if (queue.getParent() == null) {
+ return true;
+}
+if (newQueues.get(queue.getParent().getQueuePath()) != null) {
+ return false;
+}
+CSQueue parent = existingQueues.get(queue.getParent().getQueuePath());
+if (parent == null) {
+ return true;
+}
+// A dynamic queue is dangling, if its parent is not parsed in newQueues
+// or if its parent is not a dynamic queue. Dynamic queues are not parsed
in
+// newQueues but they are deleted automatically, so it is safe to assume
+// that existingQueues contain valid dynamic queues.
+return !isDynamicQueue(parent);
+ }
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerNewQueueAutoCreation.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerNewQueueAutoCreation.java
index c403d23..4dae4fd 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerNewQueueAutoCreation.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity
[hadoop] branch trunk updated: YARN-10597. CSMappingPlacementRule should not create new instance of Groups. Contributed by Gergely Pollak
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 6fd0c66 YARN-10597. CSMappingPlacementRule should not create new
instance of Groups. Contributed by Gergely Pollak
6fd0c66 is described below
commit 6fd0c661b64c78e865d8328f31326eab8eb58de4
Author: Szilard Nemeth
AuthorDate: Wed Mar 31 16:14:03 2021 +0200
YARN-10597. CSMappingPlacementRule should not create new instance of
Groups. Contributed by Gergely Pollak
---
.../server/resourcemanager/placement/CSMappingPlacementRule.java| 6 +-
1 file changed, 1 insertion(+), 5 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
index 07aeb12..894bc82 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
@@ -132,11 +132,7 @@ public class CSMappingPlacementRule extends PlacementRule {
overrideWithQueueMappings = conf.getOverrideWithQueueMappings();
if (groups == null) {
- //We cannot use Groups#getUserToGroupsMappingService here, because when
- //tests change the HADOOP_SECURITY_GROUP_MAPPING, Groups won't refresh
its
- //cached instance of groups, so we might get a Group instance which
- //ignores the HADOOP_SECURITY_GROUP_MAPPING settings.
- groups = new Groups(conf);
+ groups = Groups.getUserToGroupsMappingService(conf);
}
MappingRuleValidationContext validationContext = buildValidationContext();
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10659. Improve CS MappingRule %secondary_group evaluation. Contributed by Gergely Pollak
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new a574571 YARN-10659. Improve CS MappingRule %secondary_group
evaluation. Contributed by Gergely Pollak
a574571 is described below
commit a5745711ddd8eef18ebc702da4c1f08caf3a6c59
Author: Szilard Nemeth
AuthorDate: Thu Mar 18 12:43:01 2021 +0100
YARN-10659. Improve CS MappingRule %secondary_group evaluation. Contributed
by Gergely Pollak
---
.../placement/CSMappingPlacementRule.java | 31 --
.../resourcemanager/placement/VariableContext.java | 49 +++-
.../MappingRuleConditionalVariable.java| 22
.../MappingRuleConditionalVariables.java | 123 +
.../placement/csmappingrule/MappingRuleResult.java | 2 +
.../csmappingrule/TestCSMappingPlacementRule.java | 111 ---
.../TestCapacitySchedulerQueueMappingFactory.java | 10 +-
7 files changed, 315 insertions(+), 33 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
index 04dbb64..07aeb12 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
@@ -37,6 +37,7 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.IOException;
+import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.Set;
@@ -184,6 +185,10 @@ public class CSMappingPlacementRule extends PlacementRule {
LOG.warn(
"Group provider hasn't been set, cannot query groups for user {}",
user);
+ //enforcing empty primary group instead of null, which would be
considered
+ //as unknown variable and would evaluate to '%primary_group'
+ vctx.put("%primary_group", "");
+ vctx.put("%secondary_group", "");
return;
}
Set groupsSet = groups.getGroupsSet(user);
@@ -192,24 +197,32 @@ public class CSMappingPlacementRule extends PlacementRule
{
vctx.putExtraDataset("groups", groupsSet);
return;
}
-String secondaryGroup = null;
Iterator it = groupsSet.iterator();
String primaryGroup = it.next();
+ArrayList secondaryGroupList = new ArrayList<>();
+
while (it.hasNext()) {
- String group = it.next();
- if (this.queueManager.getQueue(group) != null) {
-secondaryGroup = group;
-break;
- }
+ secondaryGroupList.add(it.next());
}
-if (secondaryGroup == null && LOG.isDebugEnabled()) {
- LOG.debug("User {} is not associated with any Secondary group", user);
+if (secondaryGroupList.size() == 0) {
+ //if we have no chance to have a secondary group to speed up evaluation
+ //we simply register it as a regular variable with "" as a value
+ vctx.put("%secondary_group", "");
+ if (LOG.isDebugEnabled()) {
+LOG.debug("User {} does not have any potential Secondary group", user);
+ }
+} else {
+ vctx.putConditional(
+ MappingRuleConditionalVariables.SecondaryGroupVariable.VARIABLE_NAME,
+ new MappingRuleConditionalVariables.SecondaryGroupVariable(
+ this.queueManager,
+ secondaryGroupList
+ ));
}
vctx.put("%primary_group", primaryGroup);
-vctx.put("%secondary_group", secondaryGroup);
vctx.putExtraDataset("groups", groupsSet);
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/VariableContext.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/VariableContext.java
index 0a30e92..d60e7b5 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/VariableContext.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/o
[hadoop] branch trunk updated: YARN-10412. Move CS placement rule related changes to a separate package. Contributed by Gergely Pollak
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 5db4c0b YARN-10412. Move CS placement rule related changes to a
separate package. Contributed by Gergely Pollak
5db4c0b is described below
commit 5db4c0bf702b91ee73910eb9d4f51ac7fa5054fa
Author: Szilard Nemeth
AuthorDate: Fri Mar 12 14:10:16 2021 +0100
YARN-10412. Move CS placement rule related changes to a separate package.
Contributed by Gergely Pollak
---
.../placement/CSMappingPlacementRule.java | 5 +-
.../resourcemanager/placement/VariableContext.java | 2 +-
.../{ => csmappingrule}/MappingQueuePath.java | 2 +-
.../placement/{ => csmappingrule}/MappingRule.java | 3 +-
.../{ => csmappingrule}/MappingRuleAction.java | 3 +-
.../{ => csmappingrule}/MappingRuleActionBase.java | 4 +-
.../{ => csmappingrule}/MappingRuleActions.java| 3 +-
.../{ => csmappingrule}/MappingRuleMatcher.java| 4 +-
.../{ => csmappingrule}/MappingRuleMatchers.java | 4 +-
.../{ => csmappingrule}/MappingRuleResult.java | 2 +-
.../{ => csmappingrule}/MappingRuleResultType.java | 2 +-
.../MappingRuleValidationContext.java | 4 +-
.../MappingRuleValidationContextImpl.java | 4 +-
.../MappingRuleValidationHelper.java | 2 +-
.../capacity/CapacitySchedulerConfiguration.java | 148 ++---
.../capacity/placement/MappingRuleCreator.java | 10 +-
.../converter/LegacyMappingRuleToJson.java | 3 +-
.../scheduler/fair/QueueManager.java | 23 ++--
.../placement/MockQueueHierarchyBuilder.java | 2 +-
.../placement/TestAppNameMappingPlacementRule.java | 1 +
.../TestUserGroupMappingPlacementRule.java | 1 +
.../TestCSMappingPlacementRule.java| 7 +-
.../{ => csmappingrule}/TestMappingRule.java | 3 +-
.../TestMappingRuleActions.java| 3 +-
.../TestMappingRuleMatchers.java | 3 +-
.../TestMappingRuleValidationContextImpl.java | 3 +-
.../{ => csmappingrule}/TestVariableContext.java | 3 +-
.../scheduler/capacity/TestQueueMappings.java | 10 +-
.../capacity/placement/TestMappingRuleCreator.java | 6 +-
.../converter/TestLegacyMappingRuleToJson.java | 2 +-
30 files changed, 144 insertions(+), 128 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
index 821d055..04dbb64 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
@@ -25,6 +25,7 @@ import org.apache.hadoop.security.Groups;
import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.resourcemanager.placement.csmappingrule.*;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceScheduler;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueue;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler;
@@ -74,12 +75,12 @@ public class CSMappingPlacementRule extends PlacementRule {
private boolean failOnConfigError = true;
@VisibleForTesting
- void setGroups(Groups groups) {
+ public void setGroups(Groups groups) {
this.groups = groups;
}
@VisibleForTesting
- void setFailOnConfigError(boolean failOnConfigError) {
+ public void setFailOnConfigError(boolean failOnConfigError) {
this.failOnConfigError = failOnConfigError;
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/VariableContext.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/VariableContext.java
index 9fc42af..0a30e92 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/VariableContext.java
+++
[hadoop] branch trunk updated: YARN-10625. FairScheduler: add global flag to disable AM-preemption. Contributed by Peter Bacsko
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 9882a22 YARN-10625. FairScheduler: add global flag to disable
AM-preemption. Contributed by Peter Bacsko
9882a22 is described below
commit 9882a223652bf5d959e749b5d19d2ac3b53b00f1
Author: Szilard Nemeth
AuthorDate: Tue Feb 16 18:06:57 2021 +0100
YARN-10625. FairScheduler: add global flag to disable AM-preemption.
Contributed by Peter Bacsko
---
.../scheduler/fair/FairScheduler.java | 7 +++
.../scheduler/fair/FairSchedulerConfiguration.java | 14 -
.../fair/TestFairSchedulerPreemption.java | 24 +-
3 files changed, 43 insertions(+), 2 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
index 813f87b..bc21a17 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
@@ -1442,6 +1442,13 @@ public class FairScheduler extends
+ " ms instead");
}
+ boolean globalAmPreemption = conf.getBoolean(
+ FairSchedulerConfiguration.AM_PREEMPTION,
+ FairSchedulerConfiguration.DEFAULT_AM_PREEMPTION);
+ if (!globalAmPreemption) {
+LOG.info("AM preemption is DISABLED globally");
+ }
+
rootMetrics = FSQueueMetrics.forQueue("root", null, true, conf);
fsOpDurations = FSOpDurations.getInstance(true);
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
index 4de6a0b..80fb144 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
@@ -176,6 +176,8 @@ public class FairSchedulerConfiguration extends
Configuration {
public static final String PREEMPTION = CONF_PREFIX + "preemption";
public static final boolean DEFAULT_PREEMPTION = false;
+ protected static final String AM_PREEMPTION =
+ CONF_PREFIX + "am.preemption";
protected static final String AM_PREEMPTION_PREFIX =
CONF_PREFIX + "am.preemption.";
protected static final boolean DEFAULT_AM_PREEMPTION = true;
@@ -407,7 +409,17 @@ public class FairSchedulerConfiguration extends
Configuration {
}
public boolean getAMPreemptionEnabled(String queueName) {
-return getBoolean(AM_PREEMPTION_PREFIX + queueName, DEFAULT_AM_PREEMPTION);
+String propertyName = AM_PREEMPTION_PREFIX + queueName;
+
+if (get(propertyName) != null) {
+ boolean amPreemptionEnabled =
+ getBoolean(propertyName, DEFAULT_AM_PREEMPTION);
+ LOG.debug("AM preemption enabled for queue {}: {}",
+ queueName, amPreemptionEnabled);
+ return amPreemptionEnabled;
+}
+
+return getBoolean(AM_PREEMPTION, DEFAULT_AM_PREEMPTION);
}
public float getPreemptionUtilizationThreshold() {
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerPreemption.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerPreemption.java
index 0240132..8d7665a 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerPreemption.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/
[hadoop] branch trunk updated: YARN-10622. Fix preemption policy to exclude childless ParentQueues. Contributed by Andras Gyori
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new bad6038 YARN-10622. Fix preemption policy to exclude childless
ParentQueues. Contributed by Andras Gyori
bad6038 is described below
commit bad6038a4879be7b93eb52cfb54ddfd4ce7111cd
Author: Szilard Nemeth
AuthorDate: Mon Feb 15 14:48:42 2021 +0100
YARN-10622. Fix preemption policy to exclude childless ParentQueues.
Contributed by Andras Gyori
---
.../ProportionalCapacityPreemptionPolicy.java | 9 +++--
.../TestProportionalCapacityPreemptionPolicy.java | 42 ++
2 files changed, 48 insertions(+), 3 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java
index ccbaf69..b343678 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java
@@ -17,6 +17,7 @@
*/
package org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity;
+import org.apache.commons.collections.CollectionUtils;
import
org.apache.hadoop.thirdparty.com.google.common.annotations.VisibleForTesting;
import org.apache.hadoop.thirdparty.com.google.common.collect.ImmutableSet;
import org.apache.commons.lang3.StringUtils;
@@ -409,9 +410,11 @@ public class ProportionalCapacityPreemptionPolicy
}
private Set getLeafQueueNames(TempQueuePerPartition q) {
-// If its a ManagedParentQueue, it might not have any children
-if ((q.children == null || q.children.isEmpty())
-&& !(q.parentQueue instanceof ManagedParentQueue)) {
+// Also exclude ParentQueues, which might be without children
+if (CollectionUtils.isEmpty(q.children)
+&& !(q.parentQueue instanceof ManagedParentQueue)
+&& (q.parentQueue == null
+|| !q.parentQueue.isEligibleForAutoQueueCreation())) {
return ImmutableSet.of(q.queueName);
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java
index 34b86bb..fe89a69 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java
@@ -81,6 +81,7 @@ import java.util.concurrent.locks.ReentrantReadWriteLock;
import static
org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.SchedulerEventType.MARK_CONTAINER_FOR_KILLABLE;
import static
org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.SchedulerEventType.MARK_CONTAINER_FOR_PREEMPTION;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
@@ -1071,6 +1072,47 @@ public class TestProportionalCapacityPreemptionPolicy {
assertEquals(newObserveOnly, policy.isObserveOnly());
}
+ @Test
+ public void testLeafQueueNameExtraction() throws Exception {
+ProportionalCapacityPreemptionPolicy policy =
+buildPolicy(Q_DATA_FOR_IGNORE);
+ParentQueue root = (ParentQueue) mCS.getRootQueue();
+root.addDynamicParentQueue("childlessFlexible");
+List queues = root.getChildQueues();
+ArrayList extendedQueues = new ArrayList<>();
+LinkedList pqs = new LinkedList<>();
+ParentQueue dynamicParent = mockParentQueue(
+null, 0, pqs);
+when(dynamicParent.getQueuePath()).thenReturn("root.dynamicParent");
+when(dynamicParent.getQueueCapacities()).thenReturn(
+new QueueCapacities
[hadoop] branch trunk updated: Support max queues limit configuration in new auto created queue, consistent with old auto created. Contributed by Qi Zhu
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 852aac3 Support max queues limit configuration in new auto created
queue, consistent with old auto created. Contributed by Qi Zhu
852aac3 is described below
commit 852aac34f2c4bea48400158098db833cca2b2611
Author: Szilard Nemeth
AuthorDate: Mon Feb 15 14:20:26 2021 +0100
Support max queues limit configuration in new auto created queue,
consistent with old auto created. Contributed by Qi Zhu
---
.../capacity/CapacitySchedulerConfiguration.java | 34 --
.../scheduler/capacity/ParentQueue.java| 10 +++
.../TestCapacitySchedulerNewQueueAutoCreation.java | 21 +
3 files changed, 63 insertions(+), 2 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
index abbc2d7..08b38a1 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
@@ -2010,11 +2010,19 @@ public class CapacitySchedulerConfiguration extends
ReservationSchedulerConfigur
@Private
private static final String AUTO_QUEUE_CREATION_V2_PREFIX =
- "auto-queue-creation-v2";
+ "auto-queue-creation-v2.";
@Private
public static final String AUTO_QUEUE_CREATION_V2_ENABLED =
- AUTO_QUEUE_CREATION_V2_PREFIX + ".enabled";
+ AUTO_QUEUE_CREATION_V2_PREFIX + "enabled";
+
+ @Private
+ public static final String AUTO_QUEUE_CREATION_V2_MAX_QUEUES =
+ AUTO_QUEUE_CREATION_V2_PREFIX + "max-queues";
+
+ @Private
+ public static final int
+ DEFAULT_AUTO_QUEUE_CREATION_V2_MAX_QUEUES = 1000;
@Private
public static final boolean DEFAULT_AUTO_QUEUE_CREATION_ENABLED = false;
@@ -2130,6 +2138,28 @@ public class CapacitySchedulerConfiguration extends
ReservationSchedulerConfigur
DEFAULT_AUTO_CREATE_QUEUE_MAX_QUEUES);
}
+ /**
+ * Get the max number of queues that are allowed to be created under
+ * a parent queue which allowed auto creation v2.
+ *
+ * @param queuePath the parent queue's path
+ * @return the max number of queues allowed to be auto created,
+ * in new auto created.
+ */
+ @Private
+ public int getAutoCreatedQueuesV2MaxChildQueuesLimit(String queuePath) {
+return getInt(getQueuePrefix(queuePath) +
+AUTO_QUEUE_CREATION_V2_MAX_QUEUES,
+DEFAULT_AUTO_QUEUE_CREATION_V2_MAX_QUEUES);
+ }
+
+ @VisibleForTesting
+ public void setAutoCreatedQueuesV2MaxChildQueuesLimit(String queuePath,
+ int maxQueues) {
+setInt(getQueuePrefix(queuePath) +
+AUTO_QUEUE_CREATION_V2_MAX_QUEUES, maxQueues);
+ }
+
@Private
public static final String AUTO_CREATED_QUEUE_MANAGEMENT_POLICY =
AUTO_CREATE_CHILD_QUEUE_PREFIX + "management-policy";
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
index b412e8a..f79ffcb 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
@@ -543,6 +543,16 @@ public class ParentQueue extends AbstractCSQueue {
return queue;
}
+ // Check if the max queue limit is exceeded.
+ int maxQueues = csContext.getConfiguration().
+ getAutoCreatedQueuesV2MaxChildQueuesLimit(getQueuePath());
+ if (childQueues.size() >= maxQueues) {
+throw new SchedulerDynamicEditException(
+"Cannot auto create queue " + childQueuePath + ". Max Child "
++ "Queue limit exceeded which is configur
[hadoop] branch trunk updated: YARN-10618. RM UI2 Application page shows the AM preempted containers instead of the nonAM ones. Contributed by Benjamin Teke
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 98ca6af YARN-10618. RM UI2 Application page shows the AM preempted
containers instead of the nonAM ones. Contributed by Benjamin Teke
98ca6af is described below
commit 98ca6afd177fff508eb22a4b6fc3ba8cfbf9ba17
Author: Szilard Nemeth
AuthorDate: Thu Feb 11 09:13:50 2021 +0100
YARN-10618. RM UI2 Application page shows the AM preempted containers
instead of the nonAM ones. Contributed by Benjamin Teke
---
.../hadoop-yarn-ui/src/main/webapp/app/templates/yarn-app/info.hbs | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/yarn-app/info.hbs
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/yarn-app/info.hbs
index 7d024c5..4c5da63 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/yarn-app/info.hbs
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/yarn-app/info.hbs
@@ -128,7 +128,7 @@
{{model.app.allocatedResource}}
{{model.app.runningContainersNumber}}
{{model.app.preemptedResource}}
-{{model.app.numAMContainerPreempted}}
+{{model.app.numNonAMContainerPreempted}}
{{model.app.numAMContainerPreempted}}
{{model.app.aggregatedResourceUsage}}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10593. Fix incorrect string comparison in GpuDiscoverer. Contributed by Peter Bacsko
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new cacc870 YARN-10593. Fix incorrect string comparison in GpuDiscoverer.
Contributed by Peter Bacsko
cacc870 is described below
commit cacc87038966a62e1d5a603a4aa40c9d28bc7d87
Author: Szilard Nemeth
AuthorDate: Wed Feb 10 13:09:11 2021 +0100
YARN-10593. Fix incorrect string comparison in GpuDiscoverer. Contributed
by Peter Bacsko
---
.../resourceplugin/gpu/GpuDiscoverer.java | 7 +++---
.../resourceplugin/gpu/TestGpuDiscoverer.java | 29 --
2 files changed, 31 insertions(+), 5 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/gpu/GpuDiscoverer.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/gpu/GpuDiscoverer.java
index 0d05c9e..f9be9fe 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/gpu/GpuDiscoverer.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/gpu/GpuDiscoverer.java
@@ -284,11 +284,12 @@ public class GpuDiscoverer extends Configured {
binaryPath = configuredBinaryFile;
// If path exists but file name is incorrect don't execute the file
String fileName = binaryPath.getName();
- if (DEFAULT_BINARY_NAME.equals(fileName)) {
+ if (!DEFAULT_BINARY_NAME.equals(fileName)) {
String msg = String.format("Please check the configuration value of"
- +" %s. It should point to an %s binary.",
+ +" %s. It should point to an %s binary, which is now %s",
YarnConfiguration.NM_GPU_PATH_TO_EXEC,
- DEFAULT_BINARY_NAME);
+ DEFAULT_BINARY_NAME,
+ fileName);
throwIfNecessary(new YarnException(msg), config);
LOG.warn(msg);
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/gpu/TestGpuDiscoverer.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/gpu/TestGpuDiscoverer.java
index 6da23858..06791c8 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/gpu/TestGpuDiscoverer.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/gpu/TestGpuDiscoverer.java
@@ -79,12 +79,23 @@ public class TestGpuDiscoverer {
}
private File setupFakeBinary(Configuration conf) {
+return setupFakeBinary(conf,
+GpuDiscoverer.DEFAULT_BINARY_NAME, false);
+ }
+
+ private File setupFakeBinary(Configuration conf, String filename,
+ boolean useFullPath) {
File fakeBinary;
try {
fakeBinary = new File(getTestParentFolder(),
- GpuDiscoverer.DEFAULT_BINARY_NAME);
+ filename);
touchFile(fakeBinary);
- conf.set(YarnConfiguration.NM_GPU_PATH_TO_EXEC, getTestParentFolder());
+ if (useFullPath) {
+conf.set(YarnConfiguration.NM_GPU_PATH_TO_EXEC,
+fakeBinary.getAbsolutePath());
+ } else {
+conf.set(YarnConfiguration.NM_GPU_PATH_TO_EXEC, getTestParentFolder());
+ }
} catch (Exception e) {
throw new RuntimeException("Failed to init fake binary", e);
}
@@ -513,4 +524,18 @@ public class TestGpuDiscoverer {
verify(gpuSpy, never()).getGpuDeviceInformation();
}
+
+ @Test
+ public void testBinaryIsNotNvidiaSmi() throws YarnException {
+exception.expect(YarnException.class);
+exception.expectMessage(String.format(
+"It should point to an %s binary, which is now %s",
+"nvidia-smi", "badfile"));
+
+Configuration conf = new Configuration(false);
+setupFakeBinary(conf, "badfile", true);
+
+GpuDiscoverer plugin = new GpuDiscoverer();
+plugin.initialize(conf, binaryHelper);
+ }
}
\ No newline at end of file
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10620. fs2cs: parentQueue for certain placement rules are not set during conversion. Contributed by Peter Bacsko
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new a8bd516 YARN-10620. fs2cs: parentQueue for certain placement rules
are not set during conversion. Contributed by Peter Bacsko
a8bd516 is described below
commit a8bd516e39fa82eea2a42b8085f3171fb0e1a883
Author: Szilard Nemeth
AuthorDate: Wed Feb 10 11:15:53 2021 +0100
YARN-10620. fs2cs: parentQueue for certain placement rules are not set
during conversion. Contributed by Peter Bacsko
---
.../fair/converter/QueuePlacementConverter.java| 20 ++
.../converter/TestQueuePlacementConverter.java | 45 ++
2 files changed, 65 insertions(+)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/QueuePlacementConverter.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/QueuePlacementConverter.java
index 0dd6ba4..65d9505 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/QueuePlacementConverter.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/QueuePlacementConverter.java
@@ -17,7 +17,9 @@ package
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter;
import java.util.ArrayList;
import java.util.List;
+import java.util.Set;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Sets;
import
org.apache.hadoop.yarn.server.resourcemanager.placement.DefaultPlacementRule;
import org.apache.hadoop.yarn.server.resourcemanager.placement.FSPlacementRule;
import
org.apache.hadoop.yarn.server.resourcemanager.placement.PlacementManager;
@@ -38,6 +40,12 @@ class QueuePlacementConverter {
private static final FallbackResult SKIP_RESULT = FallbackResult.SKIP;
private static final String DEFAULT_QUEUE = "root.default";
private static final String MATCH_ALL_USER = "*";
+ private static final Set NEED_ROOT_PARENT = Sets.newHashSet(
+ Policy.USER,
+ Policy.PRIMARY_GROUP,
+ Policy.PRIMARY_GROUP_USER,
+ Policy.SECONDARY_GROUP,
+ Policy.SECONDARY_GROUP_USER);
MappingRulesDescription convertPlacementPolicy(
PlacementManager placementManager,
@@ -162,6 +170,16 @@ class QueuePlacementConverter {
}
}
+// Need to set the parent queue in weight mode.
+//
+// We *don't* set in pct mode, because auto-creation under "root"
+// is not possible and probably it can cause the validation step to fail
+// if create=true.
+if (!usePercentages &&
+NEED_ROOT_PARENT.contains(policy)) {
+ rule.setParentQueue("root");
+}
+
return rule;
}
@@ -175,6 +193,8 @@ class QueuePlacementConverter {
Rule rule = createRule(policy, create, ruleHandler, usePercentages);
+// "parent" is already set to "root" at this point,
+// so we override it if necessary
if (parentQueue != null) {
rule.setParentQueue(parentQueue);
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestQueuePlacementConverter.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestQueuePlacementConverter.java
index 2f041c2..a1a19f0 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestQueuePlacementConverter.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestQueuePlacementConverter.java
@@ -369,6 +369,51 @@ public class TestQueuePlacementConverter {
}
@Test
+ public void testParentSetToRootInWeightModeUserPolicy() {
+UserPlacementRule fsRule = mock(UserPlacementRule.class);
+testParentSetToRootInWeightMode(fsRule);
+ }
+
+ @Test
+ public void testParentSetToRootInWeightModePrimaryGroupPolicy() {
+PrimaryGroupPlacementRule fsRule = mock(PrimaryGroupPlacementRule.class);
+testParentSetToRootInWeightMode(fsRule);
+ }
+
+ @Test
+ public void testParentSetToRootInWeightModePrimaryGroupUs
[hadoop] branch trunk updated: YARN-10619. CS Mapping Rule %specified rule catches default submissions. Contributed by Gergely Pollak
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new fc5f241 YARN-10619. CS Mapping Rule %specified rule catches default
submissions. Contributed by Gergely Pollak
fc5f241 is described below
commit fc5f2415350ffe77bcb552dc18b794b9ef0208df
Author: Szilard Nemeth
AuthorDate: Tue Feb 9 18:16:42 2021 +0100
YARN-10619. CS Mapping Rule %specified rule catches default submissions.
Contributed by Gergely Pollak
---
.../placement/CSMappingPlacementRule.java | 9 +++-
.../placement/TestCSMappingPlacementRule.java | 49 ++
2 files changed, 56 insertions(+), 2 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
index 908498d..28d2cb6 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
@@ -217,7 +217,13 @@ public class CSMappingPlacementRule extends PlacementRule {
VariableContext vctx = new VariableContext();
vctx.put("%user", user);
-vctx.put("%specified", asc.getQueue());
+//If the specified matches the default it means NO queue have been
specified
+//as per ClientRMService#submitApplication which sets the queue to default
+//when no queue is provided.
+//To place queues specifically to default, users must use root.default
+if (!asc.getQueue().equals(YarnConfiguration.DEFAULT_QUEUE_NAME)) {
+ vctx.put("%specified", asc.getQueue());
+}
vctx.put("%application", asc.getApplicationName());
vctx.put("%default", "root.default");
try {
@@ -379,7 +385,6 @@ public class CSMappingPlacementRule extends PlacementRule {
asc.getApplicationName(), appQueue, overrideWithQueueMappings);
if (appQueue != null &&
!appQueue.equals(YarnConfiguration.DEFAULT_QUEUE_NAME) &&
-!appQueue.equals(YarnConfiguration.DEFAULT_QUEUE_FULL_NAME) &&
!overrideWithQueueMappings &&
!recovery) {
LOG.info("Have no jurisdiction over application submission '{}', " +
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestCSMappingPlacementRule.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestCSMappingPlacementRule.java
index 703d517..69f56ec 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestCSMappingPlacementRule.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestCSMappingPlacementRule.java
@@ -425,6 +425,55 @@ public class TestCSMappingPlacementRule {
}
+ @Test
+ public void testSpecified() throws IOException {
+ArrayList rules = new ArrayList<>();
+rules.add(
+new MappingRule(
+MappingRuleMatchers.createAllMatcher(),
+(new MappingRuleActions.PlaceToQueueAction("%specified", true))
+.setFallbackSkip()));
+
+rules.add(
+new MappingRule(
+MappingRuleMatchers.createAllMatcher(),
+(new MappingRuleActions.PlaceToQueueAction(
+"root.ambiguous.group.tester", true))
+.setFallbackSkip()));
+
+rules.add(
+new MappingRule(
+MappingRuleMatchers.createAllMatcher(),
+(new MappingRuleActions.RejectAction())
+.setFallbackReject()));
+
+CSMappingPlacementRule engine = setupEngine(true, rules);
+ApplicationSubmissionContext appNoQueue = createApp("app");
+ApplicationSubmissionContext appDefault = createApp("app", "default");
+ApplicationSubmissionContext appRootDefault =
+createApp("app", "root.default");
+ApplicationSubmissionContext appBob =
+createApp("app",
[hadoop] branch trunk updated: YARN-10615. Fix Auto Queue Creation hierarchy construction to use queue path instead of short queue name. Contributed by Andras Gyori
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 5aa9866 YARN-10615. Fix Auto Queue Creation hierarchy construction to
use queue path instead of short queue name. Contributed by Andras Gyori
5aa9866 is described below
commit 5aa9866ec2bd33032dd1561bb19ed24d1725b721
Author: Szilard Nemeth
AuthorDate: Fri Feb 5 17:43:01 2021 +0100
YARN-10615. Fix Auto Queue Creation hierarchy construction to use queue
path instead of short queue name. Contributed by Andras Gyori
---
.../capacity/CapacitySchedulerAutoQueueHandler.java | 6 --
.../TestCapacitySchedulerNewQueueAutoCreation.java| 19 +++
2 files changed, 23 insertions(+), 2 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerAutoQueueHandler.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerAutoQueueHandler.java
index 55ab1ff..e847737 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerAutoQueueHandler.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerAutoQueueHandler.java
@@ -55,13 +55,15 @@ public class CapacitySchedulerAutoQueueHandler {
List parentsToCreate = new ArrayList<>();
ApplicationPlacementContext queueCandidateContext = parentContext;
-CSQueue existingQueueCandidate =
getQueue(queueCandidateContext.getQueue());
+CSQueue existingQueueCandidate = getQueue(
+queueCandidateContext.getFullQueuePath());
while (existingQueueCandidate == null) {
parentsToCreate.add(queueCandidateContext);
queueCandidateContext = CSQueueUtils.extractQueuePath(
queueCandidateContext.getParentQueue());
- existingQueueCandidate = getQueue(queueCandidateContext.getQueue());
+ existingQueueCandidate = getQueue(
+ queueCandidateContext.getFullQueuePath());
}
// Reverse the collection to to represent the hierarchy to be created
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerNewQueueAutoCreation.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerNewQueueAutoCreation.java
index 48dba82..d48bdc1 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerNewQueueAutoCreation.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerNewQueueAutoCreation.java
@@ -525,6 +525,25 @@ public class TestCapacitySchedulerNewQueueAutoCreation
user0LeafQueue.getMinimumAllocation()).getMemorySize(), 1e-6);
}
+ @Test
+ public void testAutoCreateQueueIfAmbiguousQueueNames() throws Exception {
+startScheduler();
+
+AbstractCSQueue b = (AbstractCSQueue) cs.getQueue("root.b");
+Assert.assertFalse(b.isDynamicQueue());
+
+createQueue("root.a.b.b");
+
+AbstractCSQueue bAutoParent = (AbstractCSQueue) cs.getQueue("root.a.b");
+Assert.assertTrue(bAutoParent.isDynamicQueue());
+Assert.assertTrue(bAutoParent.hasChildQueues());
+
+AbstractCSQueue bAutoLeafQueue =
+(AbstractCSQueue) cs.getQueue("root.a.b.b");
+Assert.assertTrue(bAutoLeafQueue.isDynamicQueue());
+Assert.assertFalse(bAutoLeafQueue.hasChildQueues());
+ }
+
private LeafQueue createQueue(String queuePath) throws YarnException {
return autoQueueHandler.autoCreateQueue(
CSQueueUtils.extractQueuePath(queuePath));
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10610. Add queuePath to RESTful API for CapacityScheduler consistent with FairScheduler queuePath. Contributed by Qi Zhu
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new c19326c YARN-10610. Add queuePath to RESTful API for
CapacityScheduler consistent with FairScheduler queuePath. Contributed by Qi Zhu
c19326c is described below
commit c19326c051103d0985c0f949654487cfa684335a
Author: Szilard Nemeth
AuthorDate: Fri Feb 5 17:34:22 2021 +0100
YARN-10610. Add queuePath to RESTful API for CapacityScheduler consistent
with FairScheduler queuePath. Contributed by Qi Zhu
---
.../webapp/dao/CapacitySchedulerInfo.java | 6 +
.../webapp/dao/CapacitySchedulerQueueInfo.java | 2 --
.../webapp/TestRMWebServicesCapacitySched.java | 26 +-
.../TestRMWebServicesForCSWithPartitions.java | 2 +-
4 files changed, 27 insertions(+), 9 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
index 9143817..fc18d4d 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
@@ -49,6 +49,7 @@ public class CapacitySchedulerInfo extends SchedulerInfo {
protected float weight;
protected float normalizedWeight;
protected String queueName;
+ private String queuePath;
protected CapacitySchedulerQueueInfoList queues;
protected QueueCapacitiesInfo capacities;
protected CapacitySchedulerHealthInfo health;
@@ -69,6 +70,7 @@ public class CapacitySchedulerInfo extends SchedulerInfo {
public CapacitySchedulerInfo(CSQueue parent, CapacityScheduler cs) {
this.queueName = parent.getQueueName();
+this.queuePath = parent.getQueuePath();
this.usedCapacity = parent.getUsedCapacity() * 100;
this.capacity = parent.getCapacity() * 100;
float max = parent.getMaximumCapacity();
@@ -134,6 +136,10 @@ public class CapacitySchedulerInfo extends SchedulerInfo {
return this.queueName;
}
+ public String getQueuePath() {
+return this.queuePath;
+ }
+
public ResourceInfo getMaximumAllocation() {
return maximumAllocation;
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java
index 9126566..3865cce 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java
@@ -57,9 +57,7 @@ public class CapacitySchedulerQueueInfo {
@XmlTransient
static final float EPSILON = 1e-8f;
- @XmlTransient
protected String queuePath;
-
protected float capacity;
protected float usedCapacity;
protected float maxCapacity;
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java
index 9ad75d6..a0a0884 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java
@@ -84,9 +84,14 @@ public class TestRMWebServicesCapacitySched extends
JerseyTestBase {
float absoluteUsedCapacity;
int numApplications;
String queueName;
+private String queuePath;
String state;
boolean isAbsoluteResource;
boolean autoCreateChildQueueEnabled;
+
+public
[hadoop] branch trunk updated: YARN-10428. Zombie applications in the YARN queue using FAIR + sizebasedweight. Contributed by Guang Yang, Andras Gyori
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 79a4659 YARN-10428. Zombie applications in the YARN queue using FAIR
+ sizebasedweight. Contributed by Guang Yang, Andras Gyori
79a4659 is described below
commit 79a46599f76e470527ad94b0894dacb28db01465
Author: Szilard Nemeth
AuthorDate: Fri Feb 5 17:08:16 2021 +0100
YARN-10428. Zombie applications in the YARN queue using FAIR +
sizebasedweight. Contributed by Guang Yang, Andras Gyori
---
.../scheduler/policy/FairOrderingPolicy.java | 6 ++--
.../scheduler/policy/TestFairOrderingPolicy.java | 42 ++
2 files changed, 46 insertions(+), 2 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FairOrderingPolicy.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FairOrderingPolicy.java
index 863d8e2..645492e 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FairOrderingPolicy.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FairOrderingPolicy.java
@@ -100,10 +100,12 @@ public class FairOrderingPolicy extends AbstractCom
private double getMagnitude(SchedulableEntity r) {
double mag = r.getSchedulingResourceUsage().getCachedUsed(
CommonNodeLabelsManager.ANY).getMemorySize();
-if (sizeBasedWeight) {
+if (sizeBasedWeight && mag != 0) {
double weight =
Math.log1p(r.getSchedulingResourceUsage().getCachedDemand(
CommonNodeLabelsManager.ANY).getMemorySize()) / Math.log(2);
- mag = mag / weight;
+ if (weight != 0) {
+mag = mag / weight;
+ }
}
return mag;
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/TestFairOrderingPolicy.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/TestFairOrderingPolicy.java
index d51f9f5..ac5caed 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/TestFairOrderingPolicy.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/TestFairOrderingPolicy.java
@@ -23,7 +23,9 @@ import static org.junit.Assert.assertEquals;
import java.util.*;
import org.apache.hadoop.yarn.api.records.NodeId;
+import org.apache.hadoop.yarn.api.records.Priority;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager;
import org.apache.hadoop.yarn.server.resourcemanager.MockRM;
import org.apache.hadoop.yarn.server.resourcemanager.MockRMAppSubmissionData;
import org.apache.hadoop.yarn.server.resourcemanager.MockRMAppSubmitter;
@@ -346,4 +348,44 @@ public class TestFairOrderingPolicy {
assertEquals("Comparator Output", 0,
policy.getComparator().compare(r1, r2));
}
+
+ @Test
+ public void testRemoveEntitiesWithSizeBasedWeightAsCompletedJobs() {
+FairOrderingPolicy policy =
+new FairOrderingPolicy();
+policy.setSizeBasedWeight(true);
+
+// Add 10 different schedulable entities
+List entities = new ArrayList<>(10);
+for (int i = 1; i <= 10; i++) {
+ MockSchedulableEntity r = new MockSchedulableEntity();
+ r.setApplicationPriority(Priority.newInstance(i));
+ r.setUsed(Resources.createResource(4 * i));
+ r.setPending(Resources.createResource(4 * i));
+ AbstractComparatorOrderingPolicy.updateSchedulingResourceUsage(
+ r.getSchedulingResourceUsage());
+ policy.addSchedulableEntity(r);
+ entities.add(r);
+}
+
+// Mark the first 5 entities as completed by setting
+// the resources to 0
+for (int i = 0; i < 5; i++) {
+ MockSchedulableEntity r = entities.get(i);
+ r.getSchedulingResourceUsage().setCachedUsed(
+ CommonNodeLabelsManager.ANY, Resources.createResource(0));
+ r.getSchedulingResourceUsage().setCachedPending(
+ CommonNodeLabelsManager.ANY, Resources.createResource(0));
+ policy.entityRequiresReordering(r);
+}
+
+
[hadoop] branch trunk updated: YARN-10612. Fix findbugs issue introduced in YARN-10585. Contributed by Gergely Pollak
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new c174141 YARN-10612. Fix findbugs issue introduced in YARN-10585.
Contributed by Gergely Pollak
c174141 is described below
commit c17414111ba10b142bae85691a7913dcdae404c1
Author: Szilard Nemeth
AuthorDate: Thu Feb 4 08:12:13 2021 +0100
YARN-10612. Fix findbugs issue introduced in YARN-10585. Contributed by
Gergely Pollak
---
.../capacity/placement/converter/LegacyMappingRuleToJson.java | 4
1 file changed, 4 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/converter/LegacyMappingRuleToJson.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/converter/LegacyMappingRuleToJson.java
index 113b08b..6aa4049 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/converter/LegacyMappingRuleToJson.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/converter/LegacyMappingRuleToJson.java
@@ -138,10 +138,6 @@ public class LegacyMappingRuleToJson {
* @return JSON Format of the provided mapping rules, null if no rules are
set
*/
public String convert() {
-if (userGroupMappingRules == null && applicationNameMappingRules == null) {
- return null;
-}
-
//creating the basic JSON config structure
ObjectNode rootNode = objectMapper.createObjectNode();
ArrayNode rulesNode = objectMapper.createArrayNode();
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10605. Add queue-mappings-override.enable property in FS2CS conversions. Contributed by Andras Gyori
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 9bf2ac0 YARN-10605. Add queue-mappings-override.enable property in
FS2CS conversions. Contributed by Andras Gyori
9bf2ac0 is described below
commit 9bf2ac07bb8451e5b7ee5aa84d3afcb7b9e8b312
Author: Szilard Nemeth
AuthorDate: Tue Feb 2 09:13:46 2021 +0100
YARN-10605. Add queue-mappings-override.enable property in FS2CS
conversions. Contributed by Andras Gyori
---
.../scheduler/fair/converter/FSConfigToCSConfigConverter.java | 1 +
.../scheduler/fair/converter/TestFSConfigToCSConfigConverter.java | 2 ++
2 files changed, 3 insertions(+)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
index 7a0b711..a760234 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
@@ -339,6 +339,7 @@ public class FSConfigToCSConfigConverter {
capacitySchedulerConfig.set(MAPPING_RULE_FORMAT,
MAPPING_RULE_FORMAT_JSON);
+ capacitySchedulerConfig.setOverrideWithQueueMappings(true);
if (!rulesToFile) {
String json =
((ByteArrayOutputStream)mappingRulesOutputStream)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java
index 6375fe9..5450d40 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java
@@ -668,6 +668,8 @@ public class TestFSConfigToCSConfigConverter {
any(FSConfigToCSConfigRuleHandler.class),
any(CapacitySchedulerConfiguration.class),
anyBoolean());
+assertTrue(converter.getCapacitySchedulerConfig().getBoolean(
+CapacitySchedulerConfiguration.ENABLE_QUEUE_MAPPING_OVERRIDE, false));
}
@Test
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10604. Support auto queue creation without mapping rules. Contributed by Andras Gyori
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 7d88953 YARN-10604. Support auto queue creation without mapping
rules. Contributed by Andras Gyori
7d88953 is described below
commit 7d88953768411ef3e1187eaca039a2b0255ad091
Author: Szilard Nemeth
AuthorDate: Tue Feb 2 09:07:25 2021 +0100
YARN-10604. Support auto queue creation without mapping rules. Contributed
by Andras Gyori
---
.../scheduler/capacity/CapacityScheduler.java | 18 +++--
.../TestCapacitySchedulerNewQueueAutoCreation.java | 30 +-
2 files changed, 45 insertions(+), 3 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
index 204fa73..1bb74a0 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
@@ -932,12 +932,26 @@ public class CapacityScheduler extends
boolean isRecovery) {
CSQueue queue = getQueue(queueName);
+ApplicationPlacementContext fallbackContext = placementContext;
if (queue == null) {
- if (placementContext != null && placementContext.hasParentQueue()) {
+ // Even if placement rules are turned off, we still have the opportunity
+ // to auto create a queue.
+ if (placementContext == null) {
+fallbackContext = CSQueueUtils.extractQueuePath(queueName);
+ }
+ if (fallbackContext.hasParentQueue()) {
try {
- return autoCreateLeafQueue(placementContext);
+ return autoCreateLeafQueue(fallbackContext);
} catch (YarnException | IOException e) {
+ // A null queue is expected if the placementContext is null. In order
+ // not to disrupt the control flow, if we fail to auto create a
queue,
+ // we fall back to the original logic.
+ if (placementContext == null) {
+LOG.error("Could not auto-create leaf queue " + queueName +
+" due to : ", e);
+return null;
+ }
if (isRecovery) {
if (!getConfiguration().shouldAppFailFast(getConfig())) {
LOG.error("Could not auto-create leaf queue " + queueName +
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerNewQueueAutoCreation.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerNewQueueAutoCreation.java
index 0c5375e..48dba82 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerNewQueueAutoCreation.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerNewQueueAutoCreation.java
@@ -18,14 +18,21 @@
package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.exceptions.YarnException;
import org.apache.hadoop.yarn.server.resourcemanager.MockNM;
import org.apache.hadoop.yarn.server.resourcemanager.MockRM;
import
org.apache.hadoop.yarn.server.resourcemanager.nodelabels.NullRMNodeLabelsManager;
import
org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager;
+import
org.apache.hadoop.yarn.server.resourcemanager.placement.ApplicationPlacementContext;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceScheduler;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerDynamicEditException;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.AppAddedSchedulerEvent;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.
[hadoop] branch trunk updated: YARN-10598. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information. Contributed by Benjamin Teke
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 9628aa8 YARN-10598. CS Flexible Auto Queue Creation: Modify RM
/scheduler endpoint to extend the creation type with additional information.
Contributed by Benjamin Teke
9628aa8 is described below
commit 9628aa87bf213c1752511679a31c3d483b93db34
Author: Szilard Nemeth
AuthorDate: Wed Jan 27 18:14:06 2021 +0100
YARN-10598. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint
to extend the creation type with additional information. Contributed by
Benjamin Teke
---
.../webapp/dao/CapacitySchedulerInfo.java | 5 +
.../webapp/dao/CapacitySchedulerQueueInfo.java | 5 +
.../dao/helper/CapacitySchedulerInfoHelper.java| 71 +---
.../webapp/TestRMWebServicesCapacitySched.java | 6 +-
...estRMWebServicesCapacitySchedDynamicConfig.java | 122 +++--
.../TestRMWebServicesForCSWithPartitions.java | 2 +-
6 files changed, 159 insertions(+), 52 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
index 548cfe8..9143817 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
@@ -58,6 +58,8 @@ public class CapacitySchedulerInfo extends SchedulerInfo {
protected String orderingPolicyInfo;
protected String mode;
protected String queueType;
+ protected String creationMethod;
+ protected String autoCreationEligibility;
@XmlTransient
static final float EPSILON = 1e-8f;
@@ -107,6 +109,9 @@ public class CapacitySchedulerInfo extends SchedulerInfo {
}
mode = CapacitySchedulerInfoHelper.getMode(parent);
queueType = CapacitySchedulerInfoHelper.getQueueType(parent);
+creationMethod = CapacitySchedulerInfoHelper.getCreationMethod(parent);
+autoCreationEligibility = CapacitySchedulerInfoHelper
+.getAutoCreationEligibility(parent);
}
public float getCapacity() {
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java
index dec666b..9126566 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java
@@ -91,6 +91,8 @@ public class CapacitySchedulerQueueInfo {
protected LeafQueueTemplateInfo leafQueueTemplate;
protected String mode;
protected String queueType;
+ protected String creationMethod;
+ protected String autoCreationEligibility;
CapacitySchedulerQueueInfo() {
};
@@ -137,6 +139,9 @@ public class CapacitySchedulerQueueInfo {
mode = CapacitySchedulerInfoHelper.getMode(q);
queueType = CapacitySchedulerInfoHelper.getQueueType(q);
+creationMethod = CapacitySchedulerInfoHelper.getCreationMethod(q);
+autoCreationEligibility = CapacitySchedulerInfoHelper
+.getAutoCreationEligibility(q);
ResourceUsage queueResourceUsage = q.getQueueResourceUsage();
populateQueueResourceUsage(queueResourceUsage);
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/helper/CapacitySchedulerInfoHelper.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/helper/CapacitySchedulerInfoHelper.java
index 52a5a5c..355528d7 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/helper/CapacitySchedulerInfoHelper.java
+++
b/hadoop-yarn-project/hadoop-yarn
[hadoop] branch trunk updated: YARN-10599. fs2cs should generate new 'auto-queue-creation-v2.enabled' properties for all parents. Contributed by Peter Bacsko
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 7c4ef42 YARN-10599. fs2cs should generate new
'auto-queue-creation-v2.enabled' properties for all parents. Contributed by
Peter Bacsko
7c4ef42 is described below
commit 7c4ef42837955a53145f824b27ca64c11cbd
Author: Szilard Nemeth
AuthorDate: Wed Jan 27 15:49:26 2021 +0100
YARN-10599. fs2cs should generate new 'auto-queue-creation-v2.enabled'
properties for all parents. Contributed by Peter Bacsko
---
.../weightconversion/WeightToWeightConverter.java | 12 +++--
.../fair/converter/TestFSQueueConverter.java | 29
.../TestWeightToWeightConverter.java | 31 --
.../weightconversion/WeightConverterTestBase.java | 10 ---
4 files changed, 75 insertions(+), 7 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/weightconversion/WeightToWeightConverter.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/weightconversion/WeightToWeightConverter.java
index e5ce0b7..6dcd05c 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/weightconversion/WeightToWeightConverter.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/weightconversion/WeightToWeightConverter.java
@@ -23,6 +23,7 @@ import static
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.C
import java.util.List;
import org.apache.hadoop.conf.Configuration;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue;
import org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueue;
public class WeightToWeightConverter
@@ -33,14 +34,21 @@ public class WeightToWeightConverter
Configuration csConfig) {
List children = queue.getChildQueues();
-children.forEach(fsQueue -> csConfig.set(
-getProperty(fsQueue), getWeightString(fsQueue)));
+if (queue instanceof FSParentQueue || !children.isEmpty()) {
+ children.forEach(fsQueue -> csConfig.set(
+ getProperty(fsQueue), getWeightString(fsQueue)));
+ csConfig.setBoolean(getAutoCreateV2EnabledProperty(queue), true);
+}
}
private String getProperty(FSQueue queue) {
return PREFIX + queue.getName() + ".capacity";
}
+ private String getAutoCreateV2EnabledProperty(FSQueue queue) {
+return PREFIX + queue.getName() + ".auto-queue-creation-v2.enabled";
+ }
+
private String getWeightString(FSQueue queue) {
return Float.toString(queue.getWeight()) + "w";
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java
index 7807092..e648c1c 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java
@@ -335,6 +335,35 @@ public class TestFSQueueConverter {
}
@Test
+ public void testAutoCreateV2FlagsInWeightMode() {
+converter = builder.withPercentages(false).build();
+
+converter.convertQueueHierarchy(rootQueue);
+
+assertTrue("root autocreate v2 flag",
+csConfig.getBoolean(
+PREFIX + "root.auto-queue-creation-v2.enabled", false));
+assertTrue("root.admins autocreate v2 flag",
+csConfig.getBoolean(
+PREFIX + "root.admins.auto-queue-creation-v2.enabled", false));
+assertTrue("root.users autocreate v2 flag",
+csConfig.getBoolean(
+PREFIX + "root.users.auto-queue-creation-v2.enabled", false));
+assertTrue("root.misc autocreate v2 flag",
+csConfig.getBoolean(
+PREFIX + "root.misc.auto-queue-creati
[hadoop] branch trunk updated: YARN-10585. Create a class which can convert from legacy mapping rule format to the new JSON format. Contributed by Gergely Pollak
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new e2a7008 YARN-10585. Create a class which can convert from legacy
mapping rule format to the new JSON format. Contributed by Gergely Pollak
e2a7008 is described below
commit e2a7008d50d7cf2ec031c98a2b1da8075c48e0ec
Author: Szilard Nemeth
AuthorDate: Tue Jan 26 18:31:39 2021 +0100
YARN-10585. Create a class which can convert from legacy mapping rule
format to the new JSON format. Contributed by Gergely Pollak
---
.../resourcemanager/placement/MappingRule.java | 6 +-
.../placement/MappingRuleActions.java | 5 +-
.../converter/LegacyMappingRuleToJson.java | 405 +
.../placement/TestCSMappingPlacementRule.java | 2 +-
.../placement/TestMappingRuleActions.java | 5 +-
.../converter/TestLegacyMappingRuleToJson.java | 240
6 files changed, 657 insertions(+), 6 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/MappingRule.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/MappingRule.java
index e61ad95..9d67d78 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/MappingRule.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/MappingRule.java
@@ -106,7 +106,11 @@ public class MappingRule {
switch (type) {
case USER_MAPPING:
- matcher = MappingRuleMatchers.createUserMatcher(source);
+ if (source.equals("%user")) {
+matcher = MappingRuleMatchers.createAllMatcher();
+ } else {
+matcher = MappingRuleMatchers.createUserMatcher(source);
+ }
break;
case GROUP_MAPPING:
matcher = MappingRuleMatchers.createUserGroupMatcher(source);
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/MappingRuleActions.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/MappingRuleActions.java
index 13cdbe8..35d7276 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/MappingRuleActions.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/MappingRuleActions.java
@@ -96,8 +96,9 @@ public final class MappingRuleActions {
@Override
public String toString() {
return "PlaceToQueueAction{" +
- "queueName='" + queuePattern + '\'' +
- '}';
+ "queueName='" + queuePattern + "'," +
+ "allowCreate=" + allowCreate +
+ "}";
}
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/converter/LegacyMappingRuleToJson.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/converter/LegacyMappingRuleToJson.java
new file mode 100644
index 000..113b08b
--- /dev/null
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/converter/LegacyMappingRuleToJson.java
@@ -0,0 +1,405 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY
[hadoop] branch trunk updated: YARN-10596. Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled. Contributed by Andras Gyori
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new f1766e5 YARN-10596. Allow static definition of childless ParentQueues
with auto-queue-creation-v2 enabled. Contributed by Andras Gyori
f1766e5 is described below
commit f1766e5bb4bccee9b6b88daa1483c01b50db330d
Author: Szilard Nemeth
AuthorDate: Tue Jan 26 16:22:35 2021 +0100
YARN-10596. Allow static definition of childless ParentQueues with
auto-queue-creation-v2 enabled. Contributed by Andras Gyori
---
.../capacity/CapacitySchedulerQueueManager.java| 9 -
.../TestCapacitySchedulerNewQueueAutoCreation.java | 40 ++
...estRMWebServicesCapacitySchedDynamicConfig.java | 2 --
3 files changed, 48 insertions(+), 3 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueManager.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueManager.java
index c5ce700..00d1cda 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueManager.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueManager.java
@@ -238,14 +238,21 @@ public class CapacitySchedulerQueueManager implements
SchedulerQueueManager<
boolean isReservableQueue = conf.isReservable(fullQueueName);
boolean isAutoCreateEnabled = conf.isAutoCreateChildQueueEnabled(
fullQueueName);
+// if a queue is eligible for auto queue creation v2
+// it must be a ParentQueue (even if it is empty)
+boolean isAutoQueueCreationV2Enabled = conf.isAutoQueueCreationV2Enabled(
+fullQueueName);
boolean isDynamicParent = false;
+// Auto created parent queues might not have static children, but they
+// must be kept as a ParentQueue
CSQueue oldQueue = oldQueues.get(fullQueueName);
if (oldQueue instanceof ParentQueue) {
isDynamicParent = ((ParentQueue) oldQueue).isDynamicQueue();
}
-if (childQueueNames.size() == 0 && !isDynamicParent) {
+if (childQueueNames.size() == 0 && !isDynamicParent &&
+!isAutoQueueCreationV2Enabled) {
if (null == parent) {
throw new IllegalStateException(
"Queue configuration missing child queue names for " + queueName);
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerNewQueueAutoCreation.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerNewQueueAutoCreation.java
index 98b6d3f..eada112 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerNewQueueAutoCreation.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerNewQueueAutoCreation.java
@@ -422,6 +422,46 @@ public class TestCapacitySchedulerNewQueueAutoCreation
Assert.assertTrue(user0.isDynamicQueue());
}
+ @Test
+ public void testChildlessParentQueueWhenAutoQueueCreationEnabled()
+ throws Exception {
+startScheduler();
+csConf.setQueues("root", new String[]{"a", "b", "empty-auto-parent"});
+csConf.setNonLabeledQueueWeight("root", 1f);
+csConf.setNonLabeledQueueWeight("root.a", 1f);
+csConf.setNonLabeledQueueWeight("root.b", 1f);
+csConf.setQueues("root.a", new String[]{"a1"});
+csConf.setNonLabeledQueueWeight("root.a.a1", 1f);
+csConf.setAutoQueueCreationV2Enabled("root", true);
+csConf.setAutoQueueCreationV2Enabled("root.a", true);
+cs.reinitialize(csConf, mockRM.getRMContext());
+
+CSQueue empty = cs.getQueue("root.empty-auto-parent");
+Assert.assertTrue("empty-auto-parent is not a LeafQueue",
+empty instanceof LeafQueue);
+empty.stopQueue();
+
+csConf.setQueues("root", new String
[hadoop] branch trunk updated: YARN-10515. Fix flaky test TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags. Contributed by Peter Bacsko
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 8620984 YARN-10515. Fix flaky test
TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags.
Contributed by Peter Bacsko
8620984 is described below
commit 8620984b8df53afbde420532920f5392d045f9b5
Author: Szilard Nemeth
AuthorDate: Thu Jan 21 20:58:57 2021 +0100
YARN-10515. Fix flaky test
TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags.
Contributed by Peter Bacsko
---
.../scheduler/capacity/TestCapacitySchedulerAutoCreatedQueueBase.java | 2 ++
1 file changed, 2 insertions(+)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAutoCreatedQueueBase.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAutoCreatedQueueBase.java
index 1dd639c..8320d82 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAutoCreatedQueueBase.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAutoCreatedQueueBase.java
@@ -52,6 +52,7 @@ import
org.apache.hadoop.yarn.server.resourcemanager.placement.QueueMapping.Queu
import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMApp;
import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppEventType;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ContainerUpdates;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics;
import org.apache.hadoop.yarn.server.resourcemanager.scheduler
.ResourceScheduler;
@@ -205,6 +206,7 @@ public class TestCapacitySchedulerAutoCreatedQueueBase {
@Before
public void setUp() throws Exception {
+QueueMetrics.clearQueueMetrics();
CapacitySchedulerConfiguration conf = setupSchedulerConfiguration();
setupQueueConfiguration(conf);
conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10490. yarn top command not quitting completely with ctrl+c. Contributed by Agshin Kazimli
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 1ccba37 YARN-10490. yarn top command not quitting completely with
ctrl+c. Contributed by Agshin Kazimli
1ccba37 is described below
commit 1ccba3734d1c46aaae1cd3dd4dd504e691771167
Author: Szilard Nemeth
AuthorDate: Thu Jan 21 20:30:19 2021 +0100
YARN-10490. yarn top command not quitting completely with ctrl+c.
Contributed by Agshin Kazimli
---
.../src/main/java/org/apache/hadoop/yarn/client/cli/TopCLI.java | 9 -
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/TopCLI.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/TopCLI.java
index 79b1406..882d2bf 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/TopCLI.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/TopCLI.java
@@ -444,6 +444,7 @@ public class TopCLI extends YarnCLI {
public static void main(String[] args) throws Exception {
TopCLI topImp = new TopCLI();
+topImp.addShutdownHook();
topImp.setSysOutPrintStream(System.out);
topImp.setSysErrPrintStream(System.err);
int res = ToolRunner.run(topImp, args);
@@ -492,7 +493,6 @@ public class TopCLI extends YarnCLI {
rmStartTime = getRMStartTime();
}
}
-clearScreen();
return 0;
}
@@ -1220,4 +1220,11 @@ public class TopCLI extends YarnCLI {
byte[] output = IOUtils.toByteArray(p.getInputStream());
return new String(output, "ASCII");
}
+
+ private void addShutdownHook() {
+//clear screen when the program exits
+Runtime.getRuntime().addShutdownHook(new Thread(() -> {
+ clearScreen();
+}));
+ }
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10579. CS Flexible AQC: Modify RM /scheduler endpoint to include weight values for queues. Contributed by Szilard Nemeth
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 06fef5e YARN-10579. CS Flexible AQC: Modify RM /scheduler endpoint to
include weight values for queues. Contributed by Szilard Nemeth
06fef5e is described below
commit 06fef5ee43d8f95860c24577951544d82775d96a
Author: Szilard Nemeth
AuthorDate: Thu Jan 21 09:23:11 2021 +0100
YARN-10579. CS Flexible AQC: Modify RM /scheduler endpoint to include
weight values for queues. Contributed by Szilard Nemeth
---
.../webapp/dao/CapacitySchedulerInfo.java | 6 +-
.../webapp/dao/CapacitySchedulerQueueInfo.java | 12
.../webapp/dao/PartitionQueueCapacitiesInfo.java | 21 ++
.../webapp/dao/QueueCapacitiesInfo.java| 5 ++
.../webapp/TestRMWebServicesCapacitySched.java | 6 +-
...estRMWebServicesCapacitySchedDynamicConfig.java | 74 +++---
.../TestRMWebServicesForCSWithPartitions.java | 2 +-
7 files changed, 113 insertions(+), 13 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
index 9794acd..a6fcfe6 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java
@@ -46,6 +46,8 @@ public class CapacitySchedulerInfo extends SchedulerInfo {
protected float capacity;
protected float usedCapacity;
protected float maxCapacity;
+ protected float weight;
+ protected float normalizedWeight;
protected String queueName;
protected CapacitySchedulerQueueInfoList queues;
protected QueueCapacitiesInfo capacities;
@@ -70,6 +72,8 @@ public class CapacitySchedulerInfo extends SchedulerInfo {
if (max < EPSILON || max > 1f)
max = 1f;
this.maxCapacity = max * 100;
+this.weight = parent.getQueueCapacities().getWeight();
+this.normalizedWeight = parent.getQueueCapacities().getNormalizedWeight();
capacities = new QueueCapacitiesInfo(parent.getQueueCapacities(),
parent.getQueueResourceQuotas(), false);
@@ -147,7 +151,7 @@ public class CapacitySchedulerInfo extends SchedulerInfo {
CapacityScheduler cs, CSQueue parent) {
CapacitySchedulerQueueInfoList queuesInfo =
new CapacitySchedulerQueueInfoList();
-// JAXB marashalling leads to situation where the "type" field injected
+// JAXB marshalling leads to situation where the "type" field injected
// for JSON changes from string to array depending on order of printing
// Issue gets fixed if all the leaf queues are marshalled before the
// non-leaf queues. See YARN-4785 for more details.
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java
index 6fb0290..5e45f53 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java
@@ -66,6 +66,8 @@ public class CapacitySchedulerQueueInfo {
protected float absoluteCapacity;
protected float absoluteMaxCapacity;
protected float absoluteUsedCapacity;
+ protected float weight;
+ protected float normalizedWeight;
protected int numApplications;
protected String queueName;
protected boolean isAbsoluteResource;
@@ -109,6 +111,8 @@ public class CapacitySchedulerQueueInfo {
cap(q.getAbsoluteMaximumCapacity(), 0f, 1f) * 100;
absoluteUsedCapacity =
cap(q.getAbsoluteUsedCapacity(), 0f, 1f) * 100;
+weight = q.getQueueCapacities().getWeight();
+normalizedWeight = q.getQueueCapacities().getNormalizedWeight();
numApplications = q.getNumApplications();
allocatedContainers = q.getMetrics().getAllocatedContainers();
pendin
[hadoop] branch trunk updated: YARN-10578. Fix Auto Queue Creation parent handling. Contributed by Andras Gyori
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new cfe6e1f YARN-10578. Fix Auto Queue Creation parent handling.
Contributed by Andras Gyori
cfe6e1f is described below
commit cfe6e1f7da95d2ae05e55e5244925d296df4bbd4
Author: Szilard Nemeth
AuthorDate: Wed Jan 20 15:22:44 2021 +0100
YARN-10578. Fix Auto Queue Creation parent handling. Contributed by Andras
Gyori
---
.../scheduler/capacity/AbstractCSQueue.java| 19
.../scheduler/capacity/CapacityScheduler.java | 25 +++---
.../CapacitySchedulerAutoQueueHandler.java | 5 +
.../TestCapacitySchedulerNewQueueAutoCreation.java | 3 +--
4 files changed, 14 insertions(+), 38 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
index c02e1d3..bc3ff22 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
@@ -154,11 +154,6 @@ public abstract class AbstractCSQueue implements CSQueue {
// is it a dynamic queue?
private boolean dynamicQueue = false;
- // When this queue has application submit to?
- // This property only applies to dynamic queue,
- // and will be used to check when the queue need to be removed.
- private long lastSubmittedTimestamp;
-
public AbstractCSQueue(CapacitySchedulerContext cs,
String queueName, CSQueue parent, CSQueue old) throws IOException {
this(cs, cs.getConfiguration(), queueName, parent, old);
@@ -1633,18 +1628,4 @@ public abstract class AbstractCSQueue implements CSQueue
{
writeLock.unlock();
}
}
-
- public long getLastSubmittedTimestamp() {
-return lastSubmittedTimestamp;
- }
-
- // "Tab" the queue, so this queue won't be removed because of idle timeout.
- public void signalToSubmitToQueue() {
-writeLock.lock();
-try {
- this.lastSubmittedTimestamp = System.currentTimeMillis();
-} finally {
- writeLock.unlock();
-}
- }
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
index 5ee557a..204fa73 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
@@ -343,7 +343,7 @@ public class CapacityScheduler extends
this.queueManager.setCapacitySchedulerContext(this);
this.autoQueueHandler = new CapacitySchedulerAutoQueueHandler(
- this.queueManager, this.conf);
+ this.queueManager);
this.workflowPriorityMappingsMgr = new WorkflowPriorityMappingsManager();
@@ -3380,26 +3380,25 @@ public class CapacityScheduler extends
if (!StringUtils.isEmpty(parentQueueName)) {
CSQueue parentQueue = getQueue(parentQueueName);
- if (parentQueue == null) {
-throw new SchedulerDynamicEditException(
-"Could not auto-create leaf queue for " + leafQueueName
-+ ". Queue mapping specifies an invalid parent queue "
-+ "which does not exist " + parentQueueName);
- }
-
- if (conf.isAutoCreateChildQueueEnabled(parentQueue.getQueuePath())) {
+ if (parentQueue != null &&
+ conf.isAutoCreateChildQueueEnabled(parentQueue.getQueuePath())) {
// Case 1: Handle ManagedParentQueue
-AutoCreatedLeafQueue autoCreatedLeafQueue = null;
ManagedParentQueue autoCreateEnabledParentQueue =
(ManagedParentQueue) parentQueue;
-autoCreatedLeafQueue = new AutoCreatedLeafQueue(this, leafQueueName,
-autoCreateEnabledParentQueue);
+AutoCre
[hadoop] branch trunk updated: YARN-10573. Enhance placement rule conversion in fs2cs in weight mode and enable it by default. Contributed by Peter Bacsko
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new a326f22 YARN-10573. Enhance placement rule conversion in fs2cs in
weight mode and enable it by default. Contributed by Peter Bacsko
a326f22 is described below
commit a326f2260665ba3bf863c803df85d0bae174
Author: Szilard Nemeth
AuthorDate: Tue Jan 19 09:42:40 2021 +0100
YARN-10573. Enhance placement rule conversion in fs2cs in weight mode and
enable it by default. Contributed by Peter Bacsko
---
.../FSConfigToCSConfigArgumentHandler.java | 10 +--
.../converter/FSConfigToCSConfigConverter.java | 2 +-
.../converter/FSConfigToCSConfigRuleHandler.java | 27
.../fair/converter/QueuePlacementConverter.java| 79 ++
.../TestFSConfigToCSConfigArgumentHandler.java | 9 +--
.../converter/TestFSConfigToCSConfigConverter.java | 38 ++-
.../TestFSConfigToCSConfigConverterMain.java | 2 -
.../TestFSConfigToCSConfigRuleHandler.java | 10 ---
.../converter/TestQueuePlacementConverter.java | 65 +++---
9 files changed, 155 insertions(+), 87 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java
index 6f235c2..084b67d 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java
@@ -102,13 +102,12 @@ public class FSConfigToCSConfigArgumentHandler {
"Disables checking whether a placement rule is terminal to maintain" +
" backward compatibility with configs that were made before
YARN-8967.",
false),
-CONVERT_PLACEMENT_RULES("convert placement rules",
-"m", "convert-placement-rules",
-"Convert Fair Scheduler placement rules to Capacity" +
-" Scheduler mapping rules", false),
SKIP_VERIFICATION("skip verification", "s",
"skip-verification",
"Skips the verification of the converted configuration", false),
+SKIP_PLACEMENT_RULES_CONVERSION("skip placement rules conversion",
+"sp", "skip-convert-placement-rules",
+"Do not convert placement rules", false),
ENABLE_ASYNC_SCHEDULER("enable asynchronous scheduler", "a",
"enable-async-scheduler",
"Enables the Asynchronous scheduler which decouples the
CapacityScheduler" +
" scheduling from Node Heartbeats.", false),
@@ -253,7 +252,8 @@ public class FSConfigToCSConfigArgumentHandler {
String outputDir =
cliParser.getOptionValue(CliOption.OUTPUT_DIR.shortSwitch);
boolean convertPlacementRules =
-cliParser.hasOption(CliOption.CONVERT_PLACEMENT_RULES.shortSwitch);
+!cliParser.hasOption(
+CliOption.SKIP_PLACEMENT_RULES_CONVERSION.shortSwitch);
checkFile(CliOption.YARN_SITE, yarnSiteXmlFile);
checkFile(CliOption.FAIR_SCHEDULER, fairSchedulerXmlFile);
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
index 603aaed..7a0b711 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
@@ -321,7 +321,7 @@ public class FSConfigToCSConfigConverter {
MappingRulesDescription desc =
placementConverter.convertPlacementPolicy(placementManager,
- ruleHandler
[hadoop] branch trunk updated: YARN-10535. Make queue placement in CapacityScheduler compliant with auto-queue-placement. Contributed by Gergely Pollak
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 6abdb14 YARN-10535. Make queue placement in CapacityScheduler
compliant with auto-queue-placement. Contributed by Gergely Pollak
6abdb14 is described below
commit 6abdb148e4c9c6628911cbbe1f8d83e6993d4179
Author: Szilard Nemeth
AuthorDate: Mon Jan 18 20:19:36 2021 +0100
YARN-10535. Make queue placement in CapacityScheduler compliant with
auto-queue-placement. Contributed by Gergely Pollak
---
.../placement/CSMappingPlacementRule.java | 80 +++
.../placement/MappingRuleValidationContext.java| 3 +-
.../MappingRuleValidationContextImpl.java | 238 +
.../placement/MappingRuleValidationHelper.java | 153 +
.../placement/MockQueueHierarchyBuilder.java | 30 ++-
.../TestMappingRuleValidationContextImpl.java | 63 +-
.../TestCapacitySchedulerAutoQueueCreation.java| 10 +-
7 files changed, 433 insertions(+), 144 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
index b1a733d..908498d 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
@@ -26,7 +26,12 @@ import
org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.exceptions.YarnException;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceScheduler;
-import org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.*;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueue;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerContext;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -250,46 +255,47 @@ public class CSMappingPlacementRule extends PlacementRule
{
private String validateAndNormalizeQueueWithParent(
String parent, String leaf, boolean allowCreate) throws YarnException {
-CSQueue parentQueue = queueManager.getQueue(parent);
-//we don't find the specified parent, so the placement rule is invalid
-//for this case
-if (parentQueue == null) {
- if (queueManager.isAmbiguous(parent)) {
-throw new YarnException("Mapping rule specified a parent queue '" +
-parent + "', but it is ambiguous.");
- } else {
-throw new YarnException("Mapping rule specified a parent queue '" +
-parent + "', but it does not exist.");
- }
-}
-
-//normalizing parent path
-String parentPath = parentQueue.getQueuePath();
-String fullPath = parentPath + DOT + leaf;
-
-//checking if the queue actually exists
-CSQueue queue = queueManager.getQueue(fullPath);
-//if we have a parent which is not a managed parent and the queue doesn't
-//then it is an invalid target, since the queue won't be auto-created
-if (!(parentQueue instanceof ManagedParentQueue) && queue == null) {
+String normalizedPath =
+MappingRuleValidationHelper.normalizeQueuePathRoot(
+queueManager, parent + DOT + leaf);
+MappingRuleValidationHelper.ValidationResult validity =
+MappingRuleValidationHelper.validateQueuePathAutoCreation(
+queueManager, normalizedPath);
+
+switch (validity) {
+case AMBIGUOUS_PARENT:
throw new YarnException("Mapping rule specified a parent queue '" +
- parent + "', but it is not a managed parent queue, " +
+ parent + "', but it is ambiguous.");
+case AMBIGUOUS_QUEUE:
+ throw new YarnException("Mapping rule specified a target queue '" +
+
[hadoop] branch trunk updated: YARN-10574. Fix the FindBugs warning introduced in YARN-10506. Contributed by Gergely Pollak
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new acea862 YARN-10574. Fix the FindBugs warning introduced in
YARN-10506. Contributed by Gergely Pollak
acea862 is described below
commit acea862d59123391f9096614f4ad22d3818f62ba
Author: Szilard Nemeth
AuthorDate: Mon Jan 18 16:04:45 2021 +0100
YARN-10574. Fix the FindBugs warning introduced in YARN-10506. Contributed
by Gergely Pollak
---
.../server/resourcemanager/scheduler/capacity/CapacityScheduler.java | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
index 89c1cf7..5ee557a 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
@@ -3387,8 +3387,7 @@ public class CapacityScheduler extends
+ "which does not exist " + parentQueueName);
}
- if (parentQueue != null &&
- conf.isAutoCreateChildQueueEnabled(parentQueue.getQueuePath())) {
+ if (conf.isAutoCreateChildQueueEnabled(parentQueue.getQueuePath())) {
// Case 1: Handle ManagedParentQueue
AutoCreatedLeafQueue autoCreatedLeafQueue = null;
ManagedParentQueue autoCreateEnabledParentQueue =
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-7200. SLS generates a realtimetrack.json file but that file is missing the closing ']'. Contributed by Agshin Kazimli
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 6cd540e YARN-7200. SLS generates a realtimetrack.json file but that
file is missing the closing ']'. Contributed by Agshin Kazimli
6cd540e is described below
commit 6cd540e96484bb0ce546c12a97cca05da3cb59fc
Author: Szilard Nemeth
AuthorDate: Fri Jan 15 22:32:30 2021 +0100
YARN-7200. SLS generates a realtimetrack.json file but that file is missing
the closing ']'. Contributed by Agshin Kazimli
---
.../main/java/org/apache/hadoop/yarn/sls/SLSRunner.java| 14 +-
.../hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java| 13 +
.../apache/hadoop/yarn/sls/scheduler/SLSFairScheduler.java | 13 +
.../apache/hadoop/yarn/sls/scheduler/SchedulerMetrics.java | 4
4 files changed, 39 insertions(+), 5 deletions(-)
diff --git
a/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java
b/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java
index b4d4a80..a8d2aa6 100644
---
a/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java
+++
b/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java
@@ -159,6 +159,10 @@ public class SLSRunner extends Configured implements Tool {
private TraceType inputType;
private SynthTraceJobProducer stjp;
+ public static int getRemainingApps() {
+return remainingApps;
+ }
+
public SLSRunner() throws ClassNotFoundException {
Configuration tempConf = new Configuration(false);
init(tempConf);
@@ -933,12 +937,12 @@ public class SLSRunner extends Configured implements Tool
{
public static void decreaseRemainingApps() {
remainingApps--;
+ }
-if (remainingApps == 0) {
- LOG.info("SLSRunner tears down.");
- if (exitAtTheFinish) {
-System.exit(0);
- }
+ public static void exitSLSRunner() {
+LOG.info("SLSRunner tears down.");
+if (exitAtTheFinish) {
+ System.exit(0);
}
}
diff --git
a/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java
b/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java
index 9b22390..f34a3d3 100644
---
a/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java
+++
b/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java
@@ -54,6 +54,8 @@ import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.SchedulerEv
import org.apache.hadoop.yarn.sls.SLSRunner;
import org.apache.hadoop.yarn.sls.conf.SLSConfiguration;
import org.apache.hadoop.yarn.util.resource.Resources;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import com.codahale.metrics.Timer;
@@ -75,6 +77,9 @@ public class SLSCapacityScheduler extends CapacityScheduler
implements
private boolean metricsON;
private Tracker tracker;
+ // logger
+ private static final Logger LOG =
LoggerFactory.getLogger(SLSCapacityScheduler.class);
+
public Tracker getTracker() {
return tracker;
}
@@ -218,6 +223,14 @@ public class SLSCapacityScheduler extends
CapacityScheduler implements
AppAttemptRemovedSchedulerEvent appRemoveEvent =
(AppAttemptRemovedSchedulerEvent) schedulerEvent;
appQueueMap.remove(appRemoveEvent.getApplicationAttemptID());
+if (SLSRunner.getRemainingApps() == 0) {
+ try {
+getSchedulerMetrics().tearDown();
+SLSRunner.exitSLSRunner();
+ } catch (Exception e) {
+LOG.error("Scheduler Metrics failed to tear down.", e);
+ }
+}
} else if (schedulerEvent.getType() ==
SchedulerEventType.APP_ATTEMPT_ADDED
&& schedulerEvent instanceof AppAttemptAddedSchedulerEvent) {
diff --git
a/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSFairScheduler.java
b/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSFairScheduler.java
index 317c05e..09ef2cf 100644
---
a/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSFairScheduler.java
+++
b/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSFairScheduler.java
@@ -44,6 +44,8 @@ import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedule
import org.apache.hadoop.yarn.sls.SLSRunner;
import org.apache.hadoop.yarn.sls.conf.SLSConfiguration;
import org.apache.hadoop.yarn.util.resource.Resources;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.util.HashSet;
@@ -63,6 +65,9 @@ public c
[hadoop] branch trunk updated: YARN-10525. Add weight mode conversion to fs2cs. Contributed by Peter Bacsko
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 836c630 YARN-10525. Add weight mode conversion to fs2cs. Contributed
by Peter Bacsko
836c630 is described below
commit 836c6304304b5c23a627e6a9459335e462fa5639
Author: Szilard Nemeth
AuthorDate: Wed Jan 13 14:24:12 2021 +0100
YARN-10525. Add weight mode conversion to fs2cs. Contributed by Peter Bacsko
---
.../FSConfigToCSConfigArgumentHandler.java | 6 +
.../converter/FSConfigToCSConfigConverter.java | 3 +
.../FSConfigToCSConfigConverterParams.java | 14 +-
.../scheduler/fair/converter/FSQueueConverter.java | 37 ++--
.../fair/converter/FSQueueConverterBuilder.java| 6 +
.../weightconversion/CapacityConverter.java| 26 +++
.../weightconversion/CapacityConverterFactory.java | 31
.../WeightToPercentConverter.java} | 104 ++-
.../weightconversion/WeightToWeightConverter.java | 47 +
.../TestFSConfigToCSConfigArgumentHandler.java | 37
.../TestFSConfigToCSConfigConverterMain.java | 36 +++-
.../fair/converter/TestFSQueueConverter.java | 41 -
.../TestWeightToCapacityConversionUtil.java| 194 -
.../TestWeightToPercentageConverter.java | 174 ++
.../TestWeightToWeightConverter.java | 72
.../weightconversion/WeightConverterTestBase.java | 55 ++
16 files changed, 609 insertions(+), 274 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java
index 8f217e2..6f235c2 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java
@@ -115,6 +115,10 @@ public class FSConfigToCSConfigArgumentHandler {
RULES_TO_FILE("rules to external file", "e", "rules-to-file",
"Generates the converted placement rules to an external JSON file " +
"called mapping-rules.json", false),
+CONVERT_PERCENTAGES("convert weights to percentages",
+"pc", "percentage",
+"Converts FS queue weights to percentages",
+false),
HELP("help", "h", "help", "Displays the list of options", false);
private final String name;
@@ -275,6 +279,8 @@ public class FSConfigToCSConfigArgumentHandler {
.withConvertPlacementRules(convertPlacementRules)
.withPlacementRulesToFile(
cliParser.hasOption(CliOption.RULES_TO_FILE.shortSwitch))
+.withUsePercentages(
+cliParser.hasOption(CliOption.CONVERT_PERCENTAGES.shortSwitch))
.build();
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
index efd7a81..603aaed 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
@@ -102,6 +102,7 @@ public class FSConfigToCSConfigConverter {
private boolean convertPlacementRules = true;
private String outputDirectory;
private boolean rulesToFile;
+ private boolean usePercentages;
public FSConfigToCSConfigConverter(FSConfigToCSConfigRuleHandler
ruleHandler, ConversionOptions conversionOptions) {
@@ -119,6 +120,7 @@ public class FSConfigToCSConfigConverter {
this.convertPlacementRules = params.isConvertPlacementRules();
this.outputDirectory = params
[hadoop] branch trunk updated: YARN-10563. Fix dependency exclusion problem in poms. Contributed by Peter Bacsko
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 85b1c01 YARN-10563. Fix dependency exclusion problem in poms.
Contributed by Peter Bacsko
85b1c01 is described below
commit 85b1c017eedff4fd42431355b50374f073504e48
Author: Szilard Nemeth
AuthorDate: Tue Jan 12 15:29:17 2021 +0100
YARN-10563. Fix dependency exclusion problem in poms. Contributed by Peter
Bacsko
---
hadoop-project/pom.xml | 38 --
.../hadoop-yarn-server-resourcemanager/pom.xml | 46 ++
2 files changed, 46 insertions(+), 38 deletions(-)
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index ff19abc..6f644eb 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -1755,44 +1755,6 @@
org.jsonschema2pojo
jsonschema2pojo-core
${jsonschema2pojo.version}
-
-
- com.fasterxml.jackson.core
- *
-
-
- com.fasterxml.jackson.dataformat
- *
-
-
- org.codehaus.jackson
- *
-
-
- com.google.code.gson
- gson
-
-
- com.google.code.findbugs
- annotations
-
-
- org.scala-lang
- scala-library
-
-
- org.jsonschema2pojo
- jsonschema2pojo-scalagen
-
-
- com.google.code.javaparser
- javaparser
-
-
- javax.validation
- validation-api
-
-
org.xerial.snappy
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
index 0c6a193..485da71 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
@@ -256,6 +256,52 @@
org.jsonschema2pojo
jsonschema2pojo-core
+
+
+ joda-time
+ joda-time
+
+
+ com.google.code.findbugs
+ jsr305
+
+
+ com.fasterxml.jackson.core
+ *
+
+
+ com.fasterxml.jackson.dataformat
+ *
+
+
+ org.codehaus.jackson
+ *
+
+
+ com.google.code.gson
+ gson
+
+
+ com.google.code.findbugs
+ annotations
+
+
+ org.scala-lang
+ scala-library
+
+
+ org.jsonschema2pojo
+ jsonschema2pojo-scalagen
+
+
+ com.google.code.javaparser
+ javaparser
+
+
+ javax.validation
+ validation-api
+
+
com.fasterxml.jackson.core
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10570. Remove experimental warning message from fs2cs. Contributed by Peter Bacsko
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 9c6e093 YARN-10570. Remove experimental warning message from fs2cs.
Contributed by Peter Bacsko
9c6e093 is described below
commit 9c6e093783b85e5b9c6590e77d46036630e567f1
Author: Szilard Nemeth
AuthorDate: Tue Jan 12 15:21:27 2021 +0100
YARN-10570. Remove experimental warning message from fs2cs. Contributed by
Peter Bacsko
---
.../scheduler/fair/converter/FSConfigToCSConfigConverter.java | 6 --
1 file changed, 6 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
index af89657..efd7a81 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
@@ -78,10 +78,6 @@ public class FSConfigToCSConfigConverter {
private static final String FAIR_SCHEDULER_XML =
"fair-scheduler.xml";
- public static final String WARNING_TEXT =
- "WARNING: This feature is experimental and not intended " +
- "for production use!";
-
private Resource clusterResource;
private boolean preemptionEnabled = false;
private int queueMaxAppsDefault;
@@ -225,8 +221,6 @@ public class FSConfigToCSConfigConverter {
@VisibleForTesting
void convert(Configuration inputYarnSiteConfig) throws Exception {
-System.out.println(WARNING_TEXT);
-
// initialize Fair Scheduler
RMContext ctx = new RMContextImpl();
PlacementManager placementManager = new PlacementManager();
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10507. Add the capability to fs2cs to write the converted placement rules inside capacity-scheduler.xml. Contributed by Peter Bacsko
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 4176759 YARN-10507. Add the capability to fs2cs to write the
converted placement rules inside capacity-scheduler.xml. Contributed by Peter
Bacsko
4176759 is described below
commit 41767599093f6a39e43be963336af24738616cb1
Author: Szilard Nemeth
AuthorDate: Fri Jan 8 13:40:33 2021 +0100
YARN-10507. Add the capability to fs2cs to write the converted placement
rules inside capacity-scheduler.xml. Contributed by Peter Bacsko
---
.../FSConfigToCSConfigArgumentHandler.java | 12
.../converter/FSConfigToCSConfigConverter.java | 78 +++---
.../converter/FSConfigToCSConfigConverterMain.java | 20 --
.../FSConfigToCSConfigConverterParams.java | 14 +++-
.../fair/converter/VerificationException.java | 3 +
.../TestFSConfigToCSConfigArgumentHandler.java | 37 +-
.../converter/TestFSConfigToCSConfigConverter.java | 8 +--
.../TestFSConfigToCSConfigConverterMain.java | 52 +++
8 files changed, 177 insertions(+), 47 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java
index b6730c7..8f217e2 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java
@@ -112,6 +112,9 @@ public class FSConfigToCSConfigArgumentHandler {
ENABLE_ASYNC_SCHEDULER("enable asynchronous scheduler", "a",
"enable-async-scheduler",
"Enables the Asynchronous scheduler which decouples the
CapacityScheduler" +
" scheduling from Node Heartbeats.", false),
+RULES_TO_FILE("rules to external file", "e", "rules-to-file",
+"Generates the converted placement rules to an external JSON file " +
+"called mapping-rules.json", false),
HELP("help", "h", "help", "Displays the list of options", false);
private final String name;
@@ -254,6 +257,13 @@ public class FSConfigToCSConfigArgumentHandler {
checkDirectory(CliOption.OUTPUT_DIR, outputDir);
checkOutputDirDoesNotContainXmls(yarnSiteXmlFile, outputDir);
+// check mapping-rules.json if we intend to generate it
+if (!cliParser.hasOption(CliOption.CONSOLE_MODE.shortSwitch) &&
+cliParser.hasOption(CliOption.RULES_TO_FILE.shortSwitch)) {
+ checkFileNotInOutputDir(new File(outputDir),
+ FSConfigToCSConfigConverter.MAPPING_RULES_JSON);
+}
+
return FSConfigToCSConfigConverterParams.Builder.create()
.withYarnSiteXmlConfig(yarnSiteXmlFile)
.withFairSchedulerXmlConfig(fairSchedulerXmlFile)
@@ -263,6 +273,8 @@ public class FSConfigToCSConfigArgumentHandler {
.withConsole(cliParser.hasOption(CliOption.CONSOLE_MODE.shortSwitch))
.withOutputDirectory(outputDir)
.withConvertPlacementRules(convertPlacementRules)
+.withPlacementRulesToFile(
+cliParser.hasOption(CliOption.RULES_TO_FILE.shortSwitch))
.build();
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
index ce05889..af89657 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
@@ -17,13 +17,18 @@
package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair
[hadoop] branch trunk updated: YARN-10528. maxAMShare should only be accepted for leaf queues, not parent queues. Contributed by Siddharth Ahuja
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 580a6a7 YARN-10528. maxAMShare should only be accepted for leaf
queues, not parent queues. Contributed by Siddharth Ahuja
580a6a7 is described below
commit 580a6a75a3e3d3b7918edeffd6e93fc211166884
Author: Szilard Nemeth
AuthorDate: Fri Jan 8 12:38:43 2021 +0100
YARN-10528. maxAMShare should only be accepted for leaf queues, not parent
queues. Contributed by Siddharth Ahuja
---
.../fair/allocation/AllocationFileQueueParser.java | 25 ++--
.../fair/TestAllocationFileLoaderService.java | 66 ++
2 files changed, 86 insertions(+), 5 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java
index 72c6c68..e89682d 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java
@@ -126,6 +126,7 @@ public class AllocationFileQueueParser {
NodeList fields = element.getChildNodes();
boolean isLeaf = true;
boolean isReservable = false;
+boolean isMaxAMShareSet = false;
for (int j = 0; j < fields.getLength(); j++) {
Node fieldNode = fields.item(j);
@@ -157,6 +158,7 @@ public class AllocationFileQueueParser {
float val = Float.parseFloat(text);
val = Math.min(val, 1.0f);
builder.queueMaxAMShares(queueName, val);
+isMaxAMShareSet = true;
} else if (MAX_CONTAINER_ALLOCATION.equals(field.getTagName())) {
String text = getTrimmedTextData(field);
ConfigurableResource val =
@@ -220,7 +222,6 @@ public class AllocationFileQueueParser {
isLeaf = false;
}
}
-
// if a leaf in the alloc file is marked as type='parent'
// then store it as a parent queue
if (isLeaf && !"parent".equals(element.getAttribute("type"))) {
@@ -230,10 +231,11 @@ public class AllocationFileQueueParser {
}
} else {
if (isReservable) {
-throw new AllocationConfigurationException("The configuration settings"
-+ " for " + queueName + " are invalid. A queue element that "
-+ "contains child queue elements or that has the type='parent' "
-+ "attribute cannot also include a reservation element.");
+throw new AllocationConfigurationException(
+getErrorString(queueName, RESERVATION));
+ } else if (isMaxAMShareSet) {
+throw new AllocationConfigurationException(
+getErrorString(queueName, MAX_AMSHARE));
}
builder.configuredQueues(FSQueueType.PARENT, queueName);
}
@@ -253,6 +255,19 @@ public class AllocationFileQueueParser {
builder.getMaxQueueResources(), queueName);
}
+ /**
+ * Set up the error string based on the supplied parent queueName and
element.
+ * @param parentQueueName the parent queue name.
+ * @param element the element that should not be present for the parent
queue.
+ * @return the error string.
+ */
+ private String getErrorString(String parentQueueName, String element) {
+return "The configuration settings"
++ " for " + parentQueueName + " are invalid. A queue element that "
++ "contains child queue elements or that has the type='parent' "
++ "attribute cannot also include a " + element + " element.";
+ }
+
private String getTrimmedTextData(Element element) {
return ((Text) element.getFirstChild()).getData().trim();
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java
index 0650027..9fb76cf 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/t
[hadoop] branch branch-3.3 updated: YARN-10528. maxAMShare should only be accepted for leaf queues, not parent queues. Contributed by Siddharth Ahuja
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new f6b9f82 YARN-10528. maxAMShare should only be accepted for leaf
queues, not parent queues. Contributed by Siddharth Ahuja
f6b9f82 is described below
commit f6b9f82b3fcb2151bf3b920419b9d67578d2698c
Author: Szilard Nemeth
AuthorDate: Fri Jan 8 12:41:17 2021 +0100
YARN-10528. maxAMShare should only be accepted for leaf queues, not parent
queues. Contributed by Siddharth Ahuja
---
.../fair/allocation/AllocationFileQueueParser.java | 25 ++--
.../fair/TestAllocationFileLoaderService.java | 66 ++
2 files changed, 86 insertions(+), 5 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java
index 72c6c68..e89682d 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java
@@ -126,6 +126,7 @@ public class AllocationFileQueueParser {
NodeList fields = element.getChildNodes();
boolean isLeaf = true;
boolean isReservable = false;
+boolean isMaxAMShareSet = false;
for (int j = 0; j < fields.getLength(); j++) {
Node fieldNode = fields.item(j);
@@ -157,6 +158,7 @@ public class AllocationFileQueueParser {
float val = Float.parseFloat(text);
val = Math.min(val, 1.0f);
builder.queueMaxAMShares(queueName, val);
+isMaxAMShareSet = true;
} else if (MAX_CONTAINER_ALLOCATION.equals(field.getTagName())) {
String text = getTrimmedTextData(field);
ConfigurableResource val =
@@ -220,7 +222,6 @@ public class AllocationFileQueueParser {
isLeaf = false;
}
}
-
// if a leaf in the alloc file is marked as type='parent'
// then store it as a parent queue
if (isLeaf && !"parent".equals(element.getAttribute("type"))) {
@@ -230,10 +231,11 @@ public class AllocationFileQueueParser {
}
} else {
if (isReservable) {
-throw new AllocationConfigurationException("The configuration settings"
-+ " for " + queueName + " are invalid. A queue element that "
-+ "contains child queue elements or that has the type='parent' "
-+ "attribute cannot also include a reservation element.");
+throw new AllocationConfigurationException(
+getErrorString(queueName, RESERVATION));
+ } else if (isMaxAMShareSet) {
+throw new AllocationConfigurationException(
+getErrorString(queueName, MAX_AMSHARE));
}
builder.configuredQueues(FSQueueType.PARENT, queueName);
}
@@ -253,6 +255,19 @@ public class AllocationFileQueueParser {
builder.getMaxQueueResources(), queueName);
}
+ /**
+ * Set up the error string based on the supplied parent queueName and
element.
+ * @param parentQueueName the parent queue name.
+ * @param element the element that should not be present for the parent
queue.
+ * @return the error string.
+ */
+ private String getErrorString(String parentQueueName, String element) {
+return "The configuration settings"
++ " for " + parentQueueName + " are invalid. A queue element that "
++ "contains child queue elements or that has the type='parent' "
++ "attribute cannot also include a " + element + " element.";
+ }
+
private String getTrimmedTextData(Element element) {
return ((Text) element.getFirstChild()).getData().trim();
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java
index 0650027..9fb76cf 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanage
[hadoop] branch branch-3.2 updated: YARN-10528. maxAMShare should only be accepted for leaf queues, not parent queues. Contributed by Siddharth Ahuja
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 59795ec YARN-10528. maxAMShare should only be accepted for leaf
queues, not parent queues. Contributed by Siddharth Ahuja
59795ec is described below
commit 59795ec3d6cc30327a1bc3c7c9de2337abd7c7c2
Author: Szilard Nemeth
AuthorDate: Fri Jan 8 12:49:58 2021 +0100
YARN-10528. maxAMShare should only be accepted for leaf queues, not parent
queues. Contributed by Siddharth Ahuja
---
.../fair/allocation/AllocationFileQueueParser.java | 25 ++--
.../fair/TestAllocationFileLoaderService.java | 71 ++
2 files changed, 91 insertions(+), 5 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java
index 72c6c68..e89682d 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java
@@ -126,6 +126,7 @@ public class AllocationFileQueueParser {
NodeList fields = element.getChildNodes();
boolean isLeaf = true;
boolean isReservable = false;
+boolean isMaxAMShareSet = false;
for (int j = 0; j < fields.getLength(); j++) {
Node fieldNode = fields.item(j);
@@ -157,6 +158,7 @@ public class AllocationFileQueueParser {
float val = Float.parseFloat(text);
val = Math.min(val, 1.0f);
builder.queueMaxAMShares(queueName, val);
+isMaxAMShareSet = true;
} else if (MAX_CONTAINER_ALLOCATION.equals(field.getTagName())) {
String text = getTrimmedTextData(field);
ConfigurableResource val =
@@ -220,7 +222,6 @@ public class AllocationFileQueueParser {
isLeaf = false;
}
}
-
// if a leaf in the alloc file is marked as type='parent'
// then store it as a parent queue
if (isLeaf && !"parent".equals(element.getAttribute("type"))) {
@@ -230,10 +231,11 @@ public class AllocationFileQueueParser {
}
} else {
if (isReservable) {
-throw new AllocationConfigurationException("The configuration settings"
-+ " for " + queueName + " are invalid. A queue element that "
-+ "contains child queue elements or that has the type='parent' "
-+ "attribute cannot also include a reservation element.");
+throw new AllocationConfigurationException(
+getErrorString(queueName, RESERVATION));
+ } else if (isMaxAMShareSet) {
+throw new AllocationConfigurationException(
+getErrorString(queueName, MAX_AMSHARE));
}
builder.configuredQueues(FSQueueType.PARENT, queueName);
}
@@ -253,6 +255,19 @@ public class AllocationFileQueueParser {
builder.getMaxQueueResources(), queueName);
}
+ /**
+ * Set up the error string based on the supplied parent queueName and
element.
+ * @param parentQueueName the parent queue name.
+ * @param element the element that should not be present for the parent
queue.
+ * @return the error string.
+ */
+ private String getErrorString(String parentQueueName, String element) {
+return "The configuration settings"
++ " for " + parentQueueName + " are invalid. A queue element that "
++ "contains child queue elements or that has the type='parent' "
++ "attribute cannot also include a " + element + " element.";
+ }
+
private String getTrimmedTextData(Element element) {
return ((Text) element.getFirstChild()).getData().trim();
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java
index ac30b23..b7cb249 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanage
[hadoop] branch branch-3.1 updated: YARN-10528. maxAMShare should only be accepted for leaf queues, not parent queues. Contributed by Siddharth Ahuja
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 770a5631 YARN-10528. maxAMShare should only be accepted for leaf
queues, not parent queues. Contributed by Siddharth Ahuja
770a5631 is described below
commit 770a56315e1e6af48a735bdd617d209cd9158064
Author: Szilard Nemeth
AuthorDate: Fri Jan 8 12:51:06 2021 +0100
YARN-10528. maxAMShare should only be accepted for leaf queues, not parent
queues. Contributed by Siddharth Ahuja
---
.../fair/allocation/AllocationFileQueueParser.java | 24 ++--
.../fair/TestAllocationFileLoaderService.java | 71 ++
2 files changed, 91 insertions(+), 4 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java
index ec7e4a4..ebd37fe 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/allocation/AllocationFileQueueParser.java
@@ -124,6 +124,7 @@ public class AllocationFileQueueParser {
NodeList fields = element.getChildNodes();
boolean isLeaf = true;
boolean isReservable = false;
+boolean isMaxAMShareSet = false;
for (int j = 0; j < fields.getLength(); j++) {
Node fieldNode = fields.item(j);
@@ -155,6 +156,7 @@ public class AllocationFileQueueParser {
float val = Float.parseFloat(text);
val = Math.min(val, 1.0f);
builder.queueMaxAMShares(queueName, val);
+isMaxAMShareSet = true;
} else if (WEIGHT.equals(field.getTagName())) {
String text = getTrimmedTextData(field);
double val = Double.parseDouble(text);
@@ -220,10 +222,11 @@ public class AllocationFileQueueParser {
builder.configuredQueues(FSQueueType.LEAF, queueName);
} else {
if (isReservable) {
-throw new AllocationConfigurationException("The configuration settings"
-+ " for " + queueName + " are invalid. A queue element that "
-+ "contains child queue elements or that has the type='parent' "
-+ "attribute cannot also include a reservation element.");
+throw new AllocationConfigurationException(
+getErrorString(queueName, RESERVATION));
+ } else if (isMaxAMShareSet) {
+throw new AllocationConfigurationException(
+getErrorString(queueName, MAX_AMSHARE));
}
builder.configuredQueues(FSQueueType.PARENT, queueName);
}
@@ -243,6 +246,19 @@ public class AllocationFileQueueParser {
builder.getMaxQueueResources(), queueName);
}
+ /**
+ * Set up the error string based on the supplied parent queueName and
element.
+ * @param parentQueueName the parent queue name.
+ * @param element the element that should not be present for the parent
queue.
+ * @return the error string.
+ */
+ private String getErrorString(String parentQueueName, String element) {
+return "The configuration settings"
++ " for " + parentQueueName + " are invalid. A queue element that "
++ "contains child queue elements or that has the type='parent' "
++ "attribute cannot also include a " + element + " element.";
+ }
+
private String getTrimmedTextData(Element element) {
return ((Text) element.getFirstChild()).getData().trim();
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java
index 4a7461d..cc28255 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/
[hadoop] branch trunk updated: YARN-10499. TestRouterWebServiceREST fails (#2490). Contributed by Akira Ajisaka
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new df7f1e5 YARN-10499. TestRouterWebServiceREST fails (#2490).
Contributed by Akira Ajisaka
df7f1e5 is described below
commit df7f1e5199eed917ff40618708e7641238684d24
Author: Akira Ajisaka
AuthorDate: Thu Dec 17 01:26:15 2020 +0900
YARN-10499. TestRouterWebServiceREST fails (#2490). Contributed by Akira
Ajisaka
---
.../server/resourcemanager/placement/CSMappingPlacementRule.java | 5 +
1 file changed, 5 insertions(+)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
index aff75ba..b1a733d 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/CSMappingPlacementRule.java
@@ -181,6 +181,11 @@ public class CSMappingPlacementRule extends PlacementRule {
return;
}
Set groupsSet = groups.getGroupsSet(user);
+if (groupsSet.isEmpty()) {
+ LOG.warn("There are no groups for user {}", user);
+ vctx.putExtraDataset("groups", groupsSet);
+ return;
+}
String secondaryGroup = null;
Iterator it = groupsSet.iterator();
String primaryGroup = it.next();
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10526. RMAppManager CS Placement ignores parent path. Contributed by Gergely Pollak
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new df35c7f YARN-10526. RMAppManager CS Placement ignores parent path.
Contributed by Gergely Pollak
df35c7f is described below
commit df35c7f51996b63a14a3323b40fb75a6f4c29c7d
Author: Szilard Nemeth
AuthorDate: Tue Dec 15 17:30:14 2020 +0100
YARN-10526. RMAppManager CS Placement ignores parent path. Contributed by
Gergely Pollak
---
.../yarn/server/resourcemanager/RMAppManager.java | 16 +-
.../scheduler/capacity/CapacityScheduler.java | 40 --
.../TestCapacitySchedulerAutoCreatedQueueBase.java | 2 +-
.../TestCapacitySchedulerAutoQueueCreation.java| 63 +-
4 files changed, 114 insertions(+), 7 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
index 13c2ec7..25ff384 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
@@ -500,11 +500,25 @@ public class RMAppManager implements
EventHandler,
}
}
+//In the case of capacity scheduler the queue name only means the name of
+// the leaf queue, but since YARN-9879, internal queue references should
+// use full path, so we get the queue and parent name from the placement
+// context instead of the submissionContext.
+String placementQueueName = submissionContext.getQueue();
+if (placementContext != null && scheduler instanceof CapacityScheduler) {
+ if (placementContext.hasParentQueue()) {
+placementQueueName = placementContext.getParentQueue() + "." +
+placementContext.getQueue();
+ } else {
+placementQueueName = placementContext.getQueue();
+ }
+}
+
// Create RMApp
RMAppImpl application =
new RMAppImpl(applicationId, rmContext, this.conf,
submissionContext.getApplicationName(), user,
-submissionContext.getQueue(),
+placementQueueName,
submissionContext, this.scheduler, this.masterService,
submitTime, submissionContext.getApplicationType(),
submissionContext.getApplicationTags(), amReqs, placementContext,
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
index 51df224..86f3023 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
@@ -1835,6 +1835,40 @@ public class CapacityScheduler extends
return assignment;
}
+ /**
+ * This method extracts the actual queue name from an app add event.
+ * Currently unfortunately ApplicationPlacementContext and
+ * ApplicationSubmissionContext are used in a quite erratic way, this method
+ * helps to get the proper placement path for the queue if placement context
+ * is provided
+ * @param appAddedEvent The application add event with details about the app
+ * @return The name of the queue the application should be added
+ */
+ private String getAddedAppQueueName(AppAddedSchedulerEvent appAddedEvent) {
+//appAddedEvent uses the queue from ApplicationSubmissionContext but in
+//the case of CS it may be only a leaf name due to legacy reasons
+String ret = appAddedEvent.getQueue();
+ApplicationPlacementContext placementContext =
+appAddedEvent.getPlacementContext();
+
+//If we have a placement context, it means a mapping rule made a decision
+//about the queue placement, so we use those data, it is supposed to be in
+//sync with the ApplicationSubmissionContext and appAddedEvent.getQueue,
but
+//because of the aforementioned legacy reasons these two may only contain
+//the leaf queue name.
+if
[hadoop] branch trunk updated: MAPREDUCE-7309. Improve performance of reading resource request for mapper/reducers from config. Contributed by Peter Bacsko & Wangda Tan
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new f813f14 MAPREDUCE-7309. Improve performance of reading resource
request for mapper/reducers from config. Contributed by Peter Bacsko & Wangda
Tan
f813f14 is described below
commit f813f14b5b8f04824c1a04a29110b352f3493b34
Author: Szilard Nemeth
AuthorDate: Tue Nov 24 17:02:35 2020 +0100
MAPREDUCE-7309. Improve performance of reading resource request for
mapper/reducers from config. Contributed by Peter Bacsko & Wangda Tan
---
.../hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java| 12 +++-
.../hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java| 7 +++
2 files changed, 18 insertions(+), 1 deletion(-)
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
index d832b3c..3943a3a 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
@@ -158,6 +158,9 @@ public abstract class TaskAttemptImpl implements
org.apache.hadoop.mapreduce.v2.app.job.TaskAttempt,
EventHandler {
+ @VisibleForTesting
+ protected final static Map RESOURCE_REQUEST_CACHE
+ = new HashMap<>();
static final Counters EMPTY_COUNTERS = new Counters();
private static final Logger LOG =
LoggerFactory.getLogger(TaskAttemptImpl.class);
@@ -172,7 +175,7 @@ public abstract class TaskAttemptImpl implements
private final Clock clock;
private final org.apache.hadoop.mapred.JobID oldJobId;
private final TaskAttemptListener taskAttemptListener;
- private final Resource resourceCapability;
+ private Resource resourceCapability;
protected Set dataLocalHosts;
protected Set dataLocalRacks;
private final List diagnostics = new ArrayList();
@@ -707,6 +710,10 @@ public abstract class TaskAttemptImpl implements
getResourceTypePrefix(taskType);
boolean memorySet = false;
boolean cpuVcoresSet = false;
+if (RESOURCE_REQUEST_CACHE.get(taskType) != null) {
+ resourceCapability = RESOURCE_REQUEST_CACHE.get(taskType);
+ return;
+}
if (resourceTypePrefix != null) {
List resourceRequests =
ResourceUtils.getRequestedResourcesFromConfig(conf,
@@ -767,6 +774,9 @@ public abstract class TaskAttemptImpl implements
if (!cpuVcoresSet) {
this.resourceCapability.setVirtualCores(getCpuRequired(conf, taskType));
}
+RESOURCE_REQUEST_CACHE.put(taskType, resourceCapability);
+LOG.info("Resource capability of task type {} is set to {}",
+taskType, resourceCapability);
}
private String getCpuVcoresKey(TaskType taskType) {
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java
index f20a7d3..4c03c78 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java
@@ -43,6 +43,7 @@ import
org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptFailEvent;
import
org.apache.hadoop.yarn.util.resource.CustomResourceTypesConfigurationProvider;
import org.junit.After;
import org.junit.Assert;
+import org.junit.Before;
import org.junit.BeforeClass;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
@@ -155,6 +156,11 @@ public class TestTaskAttempt{
ResourceUtils.resetResourceTypes(new Configuration());
}
+ @Before
+ public void before() {
+TaskAttemptImpl.RESOURCE_REQUEST_CACHE.clear();
+ }
+
@After
public void tearDown() {
ResourceUtils.resetResourceTypes(new Configuration());
@@ -1721,6 +1727,7 @@ public class TestTaskAttempt{
TestAppender testAppender = new TestAppender();
final Logger logger = Logger.getLogger(TaskAttemptImpl.class);
try {
+TaskAttemptImpl.RESOURCE_REQUEST_CACHE.clear();
logger.addAppender(testAppender);
EventHandler eventHandler =
[hadoop] branch trunk updated: YARN-10457. Add a configuration switch to change between legacy and JSON placement rule format. Contributed by Gergely Pollak
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 0d3155a YARN-10457. Add a configuration switch to change between
legacy and JSON placement rule format. Contributed by Gergely Pollak
0d3155a is described below
commit 0d3155a687fbffef89940e13867c82a84332a9f5
Author: Szilard Nemeth
AuthorDate: Wed Nov 18 17:02:47 2020 +0100
YARN-10457. Add a configuration switch to change between legacy and JSON
placement rule format. Contributed by Gergely Pollak
---
.../capacity/CapacitySchedulerConfiguration.java | 64 +++-
.../capacity/placement/MappingRuleCreator.java | 21 +++-
.../placement/TestCSMappingPlacementRule.java | 116 -
3 files changed, 195 insertions(+), 6 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
index 0ad4802..d0ee25d 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
@@ -23,6 +23,7 @@ import
org.apache.hadoop.thirdparty.com.google.common.base.Strings;
import org.apache.hadoop.thirdparty.com.google.common.collect.ImmutableSet;
import org.apache.hadoop.yarn.server.resourcemanager.placement.MappingRule;
import
org.apache.hadoop.yarn.server.resourcemanager.placement.QueuePlacementRuleUtils;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.placement.MappingRuleCreator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.hadoop.classification.InterfaceAudience.Private;
@@ -63,6 +64,7 @@ import
org.apache.hadoop.yarn.util.resource.ResourceCalculator;
import org.apache.hadoop.yarn.util.resource.ResourceUtils;
import org.apache.hadoop.yarn.util.resource.Resources;
+import java.io.IOException;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
@@ -391,6 +393,19 @@ public class CapacitySchedulerConfiguration extends
ReservationSchedulerConfigur
"allow-zero-capacity-sum";
public static final boolean DEFAULT_ALLOW_ZERO_CAPACITY_SUM = false;
+ public static final String MAPPING_RULE_FORMAT =
+ PREFIX + "mapping-rule-format";
+ public static final String MAPPING_RULE_JSON =
+ PREFIX + "mapping-rule-json";
+ public static final String MAPPING_RULE_JSON_FILE =
+ PREFIX + "mapping-rule-json-file";
+
+ public static final String MAPPING_RULE_FORMAT_LEGACY = "legacy";
+ public static final String MAPPING_RULE_FORMAT_JSON = "json";
+
+ public static final String MAPPING_RULE_FORMAT_DEFAULT =
+ MAPPING_RULE_FORMAT_LEGACY;
+
/**
* Different resource types supported.
*/
@@ -1168,7 +1183,7 @@ public class CapacitySchedulerConfiguration extends
ReservationSchedulerConfigur
return mappings;
}
- public List getMappingRules() {
+ public List parseLegacyMappingRules() {
List mappings = new ArrayList();
Collection mappingsString =
getTrimmedStringCollection(QUEUE_MAPPING);
@@ -1208,6 +1223,53 @@ public class CapacitySchedulerConfiguration extends
ReservationSchedulerConfigur
return mappings;
}
+ public List parseJSONMappingRules() throws IOException {
+String mappingJson = get(MAPPING_RULE_JSON, "");
+String mappingJsonFile = get(MAPPING_RULE_JSON_FILE, "");
+MappingRuleCreator creator = new MappingRuleCreator();
+
+if (!mappingJson.equals("")) {
+ LOG.info("Reading mapping rules from provided inline JSON '{}'.",
+ mappingJson);
+ try {
+return creator.getMappingRulesFromString(mappingJson);
+ } catch (IOException e) {
+LOG.error("Error parsing mapping rule inline JSON.");
+throw e;
+ }
+} else if (!mappingJsonFile.equals("")) {
+ LOG.info("Reading mapping rules from JSON file '{}'.",
+ mappingJsonFile);
+ try {
+return creator.getMappingRulesFromFile(mappingJsonFile.trim());
+ } catch (IOException e) {
+LOG.error("Error reading or parsing mapping rule JSON file
[hadoop] branch trunk updated: YARN-10486. FS-CS converter: handle case when weight=0 and allow more lenient capacity checks in Capacity Scheduler. Contributed by Peter Bacsko
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 5ff70a5 YARN-10486. FS-CS converter: handle case when weight=0 and
allow more lenient capacity checks in Capacity Scheduler. Contributed by Peter
Bacsko
5ff70a5 is described below
commit 5ff70a59c40fe91c067869adfa3b15cd8f12b0d0
Author: Szilard Nemeth
AuthorDate: Wed Nov 18 16:51:21 2020 +0100
YARN-10486. FS-CS converter: handle case when weight=0 and allow more
lenient capacity checks in Capacity Scheduler. Contributed by Peter Bacsko
---
.../capacity/CapacitySchedulerConfiguration.java | 13 ++
.../scheduler/capacity/ParentQueue.java| 48 -
.../scheduler/fair/converter/FSQueueConverter.java | 71 ++--
.../converter/WeightToCapacityConversionUtil.java | 144 +++
.../scheduler/capacity/TestParentQueue.java| 60 ++-
.../fair/converter/TestFSQueueConverter.java | 33 +++-
.../TestWeightToCapacityConversionUtil.java| 194 +
.../test/resources/fair-scheduler-conversion.xml | 9 +
8 files changed, 507 insertions(+), 65 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
index aa78c21..0ad4802 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
@@ -387,6 +387,10 @@ public class CapacitySchedulerConfiguration extends
ReservationSchedulerConfigur
public static final int DEFAULT_MAX_PARALLEL_APPLICATIONS =
Integer.MAX_VALUE;
+ public static final String ALLOW_ZERO_CAPACITY_SUM =
+ "allow-zero-capacity-sum";
+
+ public static final boolean DEFAULT_ALLOW_ZERO_CAPACITY_SUM = false;
/**
* Different resource types supported.
*/
@@ -1488,6 +1492,15 @@ public class CapacitySchedulerConfiguration extends
ReservationSchedulerConfigur
: defaultMaxParallelAppsForUser;
}
+ public boolean getAllowZeroCapacitySum(String queue) {
+return getBoolean(getQueuePrefix(queue)
++ ALLOW_ZERO_CAPACITY_SUM, DEFAULT_ALLOW_ZERO_CAPACITY_SUM);
+ }
+
+ public void setAllowZeroCapacitySum(String queue, boolean value) {
+setBoolean(getQueuePrefix(queue)
++ ALLOW_ZERO_CAPACITY_SUM, value);
+ }
private static final String PREEMPTION_CONFIG_PREFIX =
"yarn.resourcemanager.monitor.capacity.preemption.";
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
index 923e6875..7d82fae 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
@@ -95,6 +95,8 @@ public class ParentQueue extends AbstractCSQueue {
private int runnableApps;
+ private final boolean allowZeroCapacitySum;
+
public ParentQueue(CapacitySchedulerContext cs,
String queueName, CSQueue parent, CSQueue old) throws IOException {
super(cs, queueName, parent, old);
@@ -111,6 +113,8 @@ public class ParentQueue extends AbstractCSQueue {
}
this.childQueues = new ArrayList<>();
+this.allowZeroCapacitySum =
+cs.getConfiguration().getAllowZeroCapacitySum(getQueuePath());
setupQueueConfigs(cs.getClusterResource());
@@ -159,7 +163,8 @@ public class ParentQueue extends AbstractCSQueue {
+ aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
+ ", reservationsContinueLooking=" + reservationsContinueLooking
+ ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
- + ", priority=" + priority)
[hadoop] branch trunk updated: YARN-10420. Update CS MappingRule documentation with the new format and features. Contributed by Peter Bacsko
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 4890b85 YARN-10420. Update CS MappingRule documentation with the new
format and features. Contributed by Peter Bacsko
4890b85 is described below
commit 4890b855ece2362f05491b10d4546860014fb00c
Author: Szilard Nemeth
AuthorDate: Sat Oct 31 15:17:01 2020 +0100
YARN-10420. Update CS MappingRule documentation with the new format and
features. Contributed by Peter Bacsko
---
.../src/site/markdown/CapacityScheduler.md | 227 -
1 file changed, 222 insertions(+), 5 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
index 6a857e9..3e63c3a 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
@@ -116,6 +116,7 @@ Configuration
```
+
###Queue Properties
* Resource Allocation
@@ -172,6 +173,15 @@ Configuration
**Note:** An *ACL* is of the form *user1*,*user2* *space* *group1*,*group2*.
The special value of * implies *anyone*. The special value of *space* implies
*no one*. The default is * for the root queue if not specified.
+ * Queue lifetime for applications
+
+The `CapacityScheduler` supports the following parameters to lifetime of
an application:
+
+| Property | Description |
+|: |: |
+| `yarn.scheduler.capacity..maximum-application-lifetime` |
Maximum lifetime (in seconds) of an application which is submitted to a queue.
Any value less than or equal to zero will be considered as disabled. The
default is -1. If positive value is configured then any application submitted
to this queue will be killed after it exceeds the configured lifetime. User can
also specify lifetime per application in application submission context.
However, user lifetime will be overri [...]
+| `yarn.scheduler.capacity.root..default-application-lifetime` |
Default lifetime (in seconds) of an application which is submitted to a queue.
Any value less than or equal to zero will be considered as disabled. If the
user has not submitted application with lifetime value then this value will be
taken. It is point-in-time configuration. This feature can be set at any level
in the queue hierarchy. Child queues will inherit their parent's value unless
overridden at the child [...]
+
* Queue Mapping based on User or Group, Application Name or user defined
placement rules
The `CapacityScheduler` supports the following parameters to configure the
queue mapping based on user or group, user & group, or application name. User
can also define their own placement rule:
@@ -247,14 +257,221 @@ Below example covers single mapping separately. In case
of multiple mappings wit
```
- * Queue lifetime for applications
+###JSON-based queue mapping configuration
-The `CapacityScheduler` supports the following parameters to lifetime of
an application:
+ In order to make the queue mapping feature more versatile, a new format and
evaluation engine has been added to Capacity Scheduler. The new engine is fully
backwards compatible with the old one and adds several new features. Note that
it can also parse the old format, but the new features are only available if
you specify the mappings in JSON.
-| Property | Description |
+ * Syntax
+
+ Based on the current JSON schema, users can define mapping rules the
following way:
+
+```
+{
+ "rules": [
+{
+ "type": "...",
+ "matches": "...",
+ "policy": "...",
+ "parentQueue": "...",
+ "customPlacement": "...",
+ "fallbackResult":"...",
+ "create": true/false,
+ "value": "...",
+ "customPlacement": "..."
+},
+{
+ ... next rule ...
+}
+ ]
+}
+```
+
+Rules are evaluated from top to bottom. Compared to the legacy mapping rule
evaluator, it can be adjusted more flexibly what happens when the evaluation
stops and a given rule does not match.
+
+ * Rules
+
+ Each mapping rule can have the following settings:
+
+| Setting | Description |
|: |: |
-| `yarn.scheduler.capacity..maximum-application-lifetime` |
Maximum lifetime (in seconds) of an application which is submitted to a queue.
Any value less than or equal to zero will be considered as disabled. The
default is -1. If positive value is configured then any application submitted
to this queue will be killed after it exceeds the configured lifetime. User can
als
[hadoop] branch trunk updated: YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 31477a2 YARN-10460. Upgrading to JUnit 4.13 causes tests in
TestNodeStatusUpdater to fail. Contributed by Peter Bacsko
31477a2 is described below
commit 31477a224f2eddc78e19a9426f5ec6182193cc00
Author: Szilard Nemeth
AuthorDate: Mon Oct 19 13:17:51 2020 +0200
YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater
to fail. Contributed by Peter Bacsko
---
.../src/main/java/org/apache/hadoop/ipc/ClientCache.java | 8
.../src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine2.java | 5 +
.../hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java | 7 +++
3 files changed, 20 insertions(+)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ClientCache.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ClientCache.java
index a0720d4..9c02c45 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ClientCache.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ClientCache.java
@@ -29,6 +29,8 @@ import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.ObjectWritable;
import org.apache.hadoop.io.Writable;
+import com.google.common.annotations.VisibleForTesting;
+
/* Cache a client using its socket factory as the hash key */
@InterfaceAudience.LimitedPrivate({"HDFS", "MapReduce"})
@InterfaceStability.Evolving
@@ -114,4 +116,10 @@ public class ClientCache {
client.stop();
}
}
+
+ @VisibleForTesting
+ public void clearCache() {
+clients.values().forEach(c -> c.stop());
+clients.clear();
+ }
}
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine2.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine2.java
index fa1f106..5043051 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine2.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine2.java
@@ -352,6 +352,11 @@ public class ProtobufRpcEngine2 implements RpcEngine {
portRangeConfig, alignmentContext);
}
+ @VisibleForTesting
+ public static void clearClientCache() {
+CLIENTS.clearCache();
+ }
+
public static class Server extends RPC.Server {
static final ThreadLocal CURRENT_CALLBACK =
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java
index 28344f9..6625f96 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java
@@ -54,6 +54,7 @@ import org.apache.hadoop.io.DataOutputBuffer;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.retry.RetryPolicy;
import org.apache.hadoop.io.retry.RetryProxy;
+import org.apache.hadoop.ipc.ProtobufRpcEngine2;
import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
import org.apache.hadoop.net.NetUtils;
import org.apache.hadoop.net.ServerSocketUtil;
@@ -137,6 +138,12 @@ public class TestNodeStatusUpdater extends
NodeManagerTestBase {
private NodeManager nm;
private AtomicBoolean assertionFailedInThread = new AtomicBoolean(false);
+ @Before
+ public void before() {
+// to avoid threading issues with JUnit 4.13+
+ProtobufRpcEngine2.clearClientCache();
+ }
+
@After
public void tearDown() {
this.registeredNodes.clear();
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10454: Add applicationName policy. Contributed by Peter Bacsko
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 4235816 YARN-10454: Add applicationName policy. Contributed by Peter
Bacsko
4235816 is described below
commit 42358169f6739f439d9d2a11f6ef857ccadaad99
Author: Szilard Nemeth
AuthorDate: Tue Oct 13 10:11:10 2020 +0200
YARN-10454: Add applicationName policy. Contributed by Peter Bacsko
---
.../capacity/placement/MappingRuleCreator.java | 5 +
.../src/main/json_schema/MappingRulesDescription.json | 1 +
.../capacity/placement/TestMappingRuleCreator.java | 17 -
3 files changed, 22 insertions(+), 1 deletion(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/MappingRuleCreator.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/MappingRuleCreator.java
index 717e5bb..d042fa4 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/MappingRuleCreator.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/MappingRuleCreator.java
@@ -166,6 +166,11 @@ public class MappingRuleCreator {
getTargetQueue(rule.getParentQueue(),
"%user"), create);
break;
+case APPLICATION_NAME:
+ action = MappingRuleActions.createPlaceToQueueAction(
+ getTargetQueue(rule.getParentQueue(),
+ "%application"), create);
+ break;
case SET_DEFAULT_QUEUE:
String defaultQueue = rule.getValue();
checkArgument(defaultQueue != null, "default queue is undefined");
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/json_schema/MappingRulesDescription.json
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/json_schema/MappingRulesDescription.json
index 6955184..f74830d 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/json_schema/MappingRulesDescription.json
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/json_schema/MappingRulesDescription.json
@@ -40,6 +40,7 @@
"secondaryGroup",
"primaryGroupUser",
"secondaryGroupUser",
+ "applicationName",
"setDefaultQueue",
"custom" ] },
"parentQueue": { "type": "string" },
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/TestMappingRuleCreator.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/TestMappingRuleCreator.java
index d91e9bd..778adcb 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/TestMappingRuleCreator.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/TestMappingRuleCreator.java
@@ -43,7 +43,7 @@ public class TestMappingRuleCreator {
private static final String DEFAULT_QUEUE = "root.default";
private static final String SECONDARY_GROUP = "users";
private static final String PRIMARY_GROUP = "superuser";
- private static final String APPLICATION_NAME = "MapReduce";
+ private static final String APPLICATION_NAME = "testapplication";
private static final String SPECIFIED_QUEUE = "root.users.hadoop";
private static final String USER_NAME = "testuser";
@@ -317,6 +317,21 @@ public class TestMappingRuleCreator {
}
@Test
+ public void testApplicationNamePlacement() {
+rule.setPolicy(Policy.APPLICATION_NAME);
+
+verifyPlacementSucceeds(APPLICATION_NAME);
+ }
+
+ @Test
+ public void testApplicationNamePlacementWithParent() {
+
[hadoop] branch trunk updated: YARN-10413. Change fs2cs to generate mapping rules in the new format. Contributed by Peter Bacsko
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new a7a1f15 YARN-10413. Change fs2cs to generate mapping rules in the new
format. Contributed by Peter Bacsko
a7a1f15 is described below
commit a7a1f1541afe8ae9f98817d45bae2f30f78c1271
Author: Szilard Nemeth
AuthorDate: Tue Sep 29 20:18:26 2020 +0200
YARN-10413. Change fs2cs to generate mapping rules in the new format.
Contributed by Peter Bacsko
---
.../converter/FSConfigToCSConfigConverter.java | 144 -
.../converter/FSConfigToCSConfigRuleHandler.java | 63 ++--
.../scheduler/fair/converter/FSQueueConverter.java | 18 --
.../fair/converter/FSQueueConverterBuilder.java| 7 -
.../fair/converter/FSYarnSiteConverter.java| 20 --
.../fair/converter/QueuePlacementConverter.java| 233 +-
.../converter/TestFSConfigToCSConfigConverter.java | 137 +++--
.../TestFSConfigToCSConfigConverterMain.java | 6 +
.../TestFSConfigToCSConfigRuleHandler.java | 6 -
.../fair/converter/TestFSQueueConverter.java | 28 --
.../converter/TestQueuePlacementConverter.java | 335 +++--
.../test/resources/fair-scheduler-conversion.xml | 3 +
12 files changed, 534 insertions(+), 466 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
index a8862e9..2ee3c2e 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
@@ -27,11 +27,7 @@ import java.io.OutputStream;
import java.util.List;
import java.util.Map;
-import javax.xml.parsers.DocumentBuilder;
-import javax.xml.parsers.DocumentBuilderFactory;
-
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.security.authorize.AccessControlList;
import org.apache.hadoop.yarn.api.records.Resource;
@@ -43,22 +39,21 @@ import
org.apache.hadoop.yarn.server.resourcemanager.RMContext;
import org.apache.hadoop.yarn.server.resourcemanager.RMContextImpl;
import
org.apache.hadoop.yarn.server.resourcemanager.placement.PlacementManager;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.placement.schema.MappingRulesDescription;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationConfiguration;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationConfigurationException;
-import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.ConfigurableResource;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue;
import org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueue;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerConfiguration;
-import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.allocation.AllocationFileParser;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.policies.DominantResourceFairnessPolicy;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import org.w3c.dom.Document;
-import org.w3c.dom.Element;
-import org.w3c.dom.NodeList;
+import com.fasterxml.jackson.core.util.DefaultPrettyPrinter;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.ObjectWriter;
import com.google.common.annotations.VisibleForTesting;
/**
@@ -74,6 +69,8 @@ public class FSConfigToCSConfigConverter {
"capacity-scheduler.xml";
private static final String FAIR_SCHEDULER_XML =
"fair-scheduler.xml";
+ private static final String MAPPING_RULES_JSON =
+ "mapping-rules.json";
public static final String WARNING_TEXT =
"WARNING: This feature is experimental and not intended " +
@@ -86,23 +83,22 @@ public class FSCo
[hadoop] branch trunk updated: YARN-9333. TestFairSchedulerPreemption.testRelaxLocalityPreemptionWithNoLessAMInRemainingNodes fails intermittently. Contributed by Peter Bacsko
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new eacbe07 YARN-9333.
TestFairSchedulerPreemption.testRelaxLocalityPreemptionWithNoLessAMInRemainingNodes
fails intermittently. Contributed by Peter Bacsko
eacbe07 is described below
commit eacbe07b565f7ef6f05d0775dbafcf15eecb86f6
Author: Szilard Nemeth
AuthorDate: Thu Sep 17 19:57:19 2020 +0200
YARN-9333.
TestFairSchedulerPreemption.testRelaxLocalityPreemptionWithNoLessAMInRemainingNodes
fails intermittently. Contributed by Peter Bacsko
---
.../resourcemanager/scheduler/fair/TestFairSchedulerPreemption.java | 1 +
1 file changed, 1 insertion(+)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerPreemption.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerPreemption.java
index 1312b18..0240132 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerPreemption.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerPreemption.java
@@ -94,6 +94,7 @@ public class TestFairSchedulerPreemption extends
FairSchedulerTestBase {
conf.setBoolean(FairSchedulerConfiguration.PREEMPTION, true);
conf.setFloat(FairSchedulerConfiguration.PREEMPTION_THRESHOLD, 0f);
conf.setInt(FairSchedulerConfiguration.WAIT_TIME_BEFORE_KILL, 0);
+conf.setLong(FairSchedulerConfiguration.UPDATE_INTERVAL_MS, 60_000L);
setupCluster();
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10387. Implement logic which returns MappingRule objects based on mapping rules. Contributed by Peter Bacsko
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 773ac79 YARN-10387. Implement logic which returns MappingRule objects
based on mapping rules. Contributed by Peter Bacsko
773ac79 is described below
commit 773ac799c63b5a7caf53e9139008cc1a8e4bfa82
Author: Szilard Nemeth
AuthorDate: Wed Sep 9 15:12:58 2020 +0200
YARN-10387. Implement logic which returns MappingRule objects based on
mapping rules. Contributed by Peter Bacsko
---
.../capacity/placement/MappingRuleCreator.java | 218 ++
.../capacity/placement/TestMappingRuleCreator.java | 455 +
2 files changed, 673 insertions(+)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/MappingRuleCreator.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/MappingRuleCreator.java
new file mode 100644
index 000..717e5bb
--- /dev/null
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/MappingRuleCreator.java
@@ -0,0 +1,218 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.placement;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.yarn.server.resourcemanager.placement.MappingRule;
+import
org.apache.hadoop.yarn.server.resourcemanager.placement.MappingRuleAction;
+import
org.apache.hadoop.yarn.server.resourcemanager.placement.MappingRuleActions;
+import
org.apache.hadoop.yarn.server.resourcemanager.placement.MappingRuleMatcher;
+import
org.apache.hadoop.yarn.server.resourcemanager.placement.MappingRuleMatchers;
+
+// These are generated classes - use GeneratePojos class to create them
+// if they are missing
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.placement.schema.MappingRulesDescription;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.placement.schema.Rule;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.placement.schema.Rule.FallbackResult;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.placement.schema.Rule.Policy;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.placement.schema.Rule.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.annotations.VisibleForTesting;
+
+public class MappingRuleCreator {
+ private static final String ALL_USER = "*";
+ private static Logger LOG =
LoggerFactory.getLogger(MappingRuleCreator.class);
+
+ public MappingRulesDescription getMappingRulesFromJson(String jsonPath)
+ throws IOException {
+byte[] fileContents = Files.readAllBytes(Paths.get(jsonPath));
+return getMappingRulesFromJson(fileContents);
+ }
+
+ MappingRulesDescription getMappingRulesFromJson(byte[] contents)
+ throws IOException {
+ObjectMapper objectMapper = new ObjectMapper();
+return objectMapper.readValue(contents, MappingRulesDescription.class);
+ }
+
+ public List getMappingRules(String jsonPath) throws IOException
{
+MappingRulesDescription desc = getMappingRulesFromJson(jsonPath);
+return getMappingRules(desc);
+ }
+
+ @VisibleForTesting
+ List getMappingRules(MappingRulesDescription rules) {
+List mappingRules = new ArrayList<>();
+
+for (Rule rule : rules.getRules()) {
+ checkMandatoryParameters(rule);
+
+ MappingRule
[hadoop] branch trunk updated: Typos in YarnScheduler#allocate method's doc comment. Contributed by Siddharth Ahuja
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 696e4fe Typos in YarnScheduler#allocate method's doc comment.
Contributed by Siddharth Ahuja
696e4fe is described below
commit 696e4fe50e4282a996472cef7ae06b6a1c7be04e
Author: Szilard Nemeth
AuthorDate: Fri Sep 4 09:59:13 2020 +0200
Typos in YarnScheduler#allocate method's doc comment. Contributed by
Siddharth Ahuja
---
.../resourcemanager/scheduler/YarnScheduler.java | 27 +-
1 file changed, 16 insertions(+), 11 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java
index 5c49138..c386637 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java
@@ -127,17 +127,22 @@ public interface YarnScheduler extends
EventHandler {
public int getNumClusterNodes();
/**
- * The main api between the ApplicationMaster and the Scheduler.
- * The ApplicationMaster is updating his future resource requirements
- * and may release containers he doens't need.
- *
- * @param appAttemptId
- * @param ask
- * @param schedulingRequests
- * @param release
- * @param blacklistAdditions
- * @param blacklistRemovals
- * @param updateRequests @return the {@link Allocation} for the
application
+ * The main API between the ApplicationMaster and the Scheduler.
+ * The ApplicationMaster may request/update container resources,
+ * number of containers, node/rack preference for allocations etc.
+ * to the Scheduler.
+ * @param appAttemptId the id of the application attempt.
+ * @param ask the request made by an application to obtain various
allocations
+ * like host/rack, resource, number of containers, relaxLocality etc.,
+ * see {@link ResourceRequest}.
+ * @param schedulingRequests similar to ask, but with added ability to
specify
+ * allocation tags etc., see {@link SchedulingRequest}.
+ * @param release the list of containers to be released.
+ * @param blacklistAdditions places (node/rack) to be added to the blacklist.
+ * @param blacklistRemovals places (node/rack) to be removed from the
+ * blacklist.
+ * @param updateRequests container promotion/demotion updates.
+ * @return the {@link Allocation} for the application.
*/
@Public
@Stable
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: [UI1] Provide a way to hide Tools section in Web UIv1. Contributed by Andras Gyori
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 4ffe26f [UI1] Provide a way to hide Tools section in Web UIv1.
Contributed by Andras Gyori
4ffe26f is described below
commit 4ffe26f9b811ff505a7137a340105ad2ceb419fa
Author: Szilard Nemeth
AuthorDate: Mon Aug 31 16:00:39 2020 +0200
[UI1] Provide a way to hide Tools section in Web UIv1. Contributed by
Andras Gyori
---
.../hadoop/mapreduce/v2/hs/webapp/HsNavBlock.java | 22 +-
.../apache/hadoop/yarn/conf/YarnConfiguration.java | 2 ++
.../src/main/resources/yarn-default.xml| 6
.../applicationhistoryservice/webapp/NavBlock.java | 21 +
.../hadoop/yarn/server/webapp/WebPageUtils.java| 34 +-
.../yarn/server/nodemanager/webapp/NavBlock.java | 22 +++---
.../server/resourcemanager/webapp/NavBlock.java| 25 +++-
7 files changed, 101 insertions(+), 31 deletions(-)
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsNavBlock.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsNavBlock.java
index 9ef5a0f..fb077dd 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsNavBlock.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsNavBlock.java
@@ -18,8 +18,11 @@
package org.apache.hadoop.mapreduce.v2.hs.webapp;
+import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.mapreduce.v2.app.webapp.App;
import org.apache.hadoop.mapreduce.v2.util.MRApps;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.server.webapp.WebPageUtils;
import org.apache.hadoop.yarn.webapp.hamlet2.Hamlet;
import org.apache.hadoop.yarn.webapp.hamlet2.Hamlet.DIV;
import org.apache.hadoop.yarn.webapp.view.HtmlBlock;
@@ -31,8 +34,12 @@ import com.google.inject.Inject;
*/
public class HsNavBlock extends HtmlBlock {
final App app;
+ private Configuration conf;
- @Inject HsNavBlock(App app) { this.app = app; }
+ @Inject HsNavBlock(App app, Configuration conf) {
+this.app = app;
+this.conf = conf;
+ }
/*
* (non-Javadoc)
@@ -64,12 +71,11 @@ public class HsNavBlock extends HtmlBlock {
li().a(url("taskcounters", taskid), "Counters").__().__();
}
}
-nav.
- h3("Tools").
-ul().
- li().a("/conf", "Configuration").__().
- li().a("/logs", "Local logs").__().
- li().a("/stacks", "Server stacks").__().
- li().a("/jmx?qry=Hadoop:*", "Server metrics").__().__().__();
+
+Hamlet.UL> tools = WebPageUtils.appendToolSection(nav, conf);
+
+if (tools != null) {
+ tools.__().__();
+}
}
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index 6c1be0e..2dbbfbf 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -362,6 +362,8 @@ public class YarnConfiguration extends Configuration {
+ "webapp.ui2.war-file-path";
public static final String YARN_API_SERVICES_ENABLE = "yarn."
+ "webapp.api-service.enable";
+ public static final String YARN_WEBAPP_UI1_ENABLE_TOOLS = "yarn."
+ + "webapp.ui1.tools.enable";
@Private
public static final String DEFAULT_YARN_API_SYSTEM_SERVICES_CLASS =
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
index 47f1239..23c56e7 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
@@ -257,6 +257,12 @@
+Enable tools section in all ui1 webapp.
+yarn.webapp.ui1.tools.enable
+true
+
+
+
Explicitly provide WAR file path for ui2 if needed.
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/s
[hadoop] branch trunk updated: YARN-10373. Create Matchers for CS mapping rules. Contributed by Gergely Pollak
This is an automated email from the ASF dual-hosted git repository.
snemeth pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 56a5c36 YARN-10373. Create Matchers for CS mapping rules. Contributed
by Gergely Pollak
56a5c36 is described below
commit 56a5c360a101470b56e40a021891b73a26e6db73
Author: Szilard Nemeth
AuthorDate: Sat Aug 29 21:34:55 2020 +0200
YARN-10373. Create Matchers for CS mapping rules. Contributed by Gergely
Pollak
---
.../placement/MappingRuleMatcher.java | 28 +++
.../placement/MappingRuleMatchers.java | 239 +++
.../placement/TestMappingRuleMatchers.java | 255 +
3 files changed, 522 insertions(+)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/MappingRuleMatcher.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/MappingRuleMatcher.java
new file mode 100644
index 000..d2650e9
--- /dev/null
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/MappingRuleMatcher.java
@@ -0,0 +1,28 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.placement;
+
+public interface MappingRuleMatcher {
+ /**
+ * Returns true if the matcher matches the current context.
+ * @param variables The variable context, which contains all the variables
+ * @return true if this matcher matches to the provided variable set
+ */
+ boolean match(VariableContext variables);
+}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/MappingRuleMatchers.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/MappingRuleMatchers.java
new file mode 100644
index 000..fdc239f
--- /dev/null
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/MappingRuleMatchers.java
@@ -0,0 +1,239 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.placement;
+
+import java.util.Arrays;
+
+/**
+ * This class contains all the matcher and some helper methods to generate
them.
+ */
+public class MappingRuleMatchers {
+ /**
+ * Utility class, hiding constructor.
+ */
+ private MappingRuleMatchers() {}
+
+ /**
+ * MatchAllMatcher is a matcher which matches everything.
+ */
+ public static class MatchAllMatcher implements MappingRuleMatcher {
+/**
+ * The match will return true in all cases, to match all submissions.
+ * @param variables The variable context, which contains all the variables
+ * @return true
+ */
+@Override
+public boolean match(VariableContext variables) {
+ return true;
+}
+
+@Override
+public String toS
