[hadoop] branch trunk updated: YARN-7707. BackPort [GPG] Policy generator framework. (#5810)
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 6d32a06125f YARN-7707. BackPort [GPG] Policy generator framework.
(#5810)
6d32a06125f is described below
commit 6d32a06125f90485d3e90da2ebfa3d19cf6a818f
Author: slfan1989 <55643692+slfan1...@users.noreply.github.com>
AuthorDate: Fri Jul 28 13:41:27 2023 +0800
YARN-7707. BackPort [GPG] Policy generator framework. (#5810)
---
.../apache/hadoop/yarn/conf/YarnConfiguration.java | 37 +++
.../src/main/resources/yarn-default.xml| 42 +++
.../pom.xml| 18 ++
.../server/globalpolicygenerator/GPGContext.java | 4 +
.../globalpolicygenerator/GPGContextImpl.java | 10 +
.../globalpolicygenerator/GPGPolicyFacade.java | 222 ++
.../server/globalpolicygenerator/GPGUtils.java | 86 ++
.../GlobalPolicyGenerator.java | 38 ++-
.../policygenerator/GlobalPolicy.java | 78 +
.../NoOpGlobalPolicy.java} | 27 +-
.../policygenerator/PolicyGenerator.java | 268
.../UniformWeightedLocalityGlobalPolicy.java | 68 +
.../package-info.java} | 13 +-
.../globalpolicygenerator/TestGPGPolicyFacade.java | 202 +
.../policygenerator/TestPolicyGenerator.java | 336 +
.../src/test/resources/schedulerInfo1.json | 134
.../src/test/resources/schedulerInfo2.json | 196
17 files changed, 1751 insertions(+), 28 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index 648fddbbbe9..faa5c82d7e9 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -4390,6 +4390,43 @@ public class YarnConfiguration extends Configuration {
public static final String GPG_KERBEROS_PRINCIPAL_HOSTNAME_KEY =
FEDERATION_GPG_PREFIX +
"kerberos.principal.hostname";
+ public static final String FEDERATION_GPG_POLICY_PREFIX =
+ FEDERATION_GPG_PREFIX + "policy.generator.";
+
+ /** The interval at which the policy generator runs, default is one hour. */
+ public static final String GPG_POLICY_GENERATOR_INTERVAL =
+ FEDERATION_GPG_POLICY_PREFIX + "interval";
+ public static final long DEFAULT_GPG_POLICY_GENERATOR_INTERVAL =
TimeUnit.HOURS.toMillis(1);
+
+ /** The interval at which the policy generator runs, default is one hour.
+ * This is an deprecated property, We better set it
+ * `yarn.federation.gpg.policy.generator.interval`. */
+ public static final String GPG_POLICY_GENERATOR_INTERVAL_MS =
+ FEDERATION_GPG_POLICY_PREFIX + "interval-ms";
+
+ /**
+ * The configured policy generator class, runs NoOpGlobalPolicy by
+ * default.
+ */
+ public static final String GPG_GLOBAL_POLICY_CLASS =
FEDERATION_GPG_POLICY_PREFIX + "class";
+ public static final String DEFAULT_GPG_GLOBAL_POLICY_CLASS =
+ "org.apache.hadoop.yarn.server.globalpolicygenerator.policygenerator." +
+ "NoOpGlobalPolicy";
+
+ /**
+ * Whether or not the policy generator is running in read only (won't modify
+ * policies), default is false.
+ */
+ public static final String GPG_POLICY_GENERATOR_READONLY =
+ FEDERATION_GPG_POLICY_PREFIX + "readonly";
+ public static final boolean DEFAULT_GPG_POLICY_GENERATOR_READONLY = false;
+
+ /**
+ * Which sub-clusters the policy generator should blacklist.
+ */
+ public static final String GPG_POLICY_GENERATOR_BLACKLIST =
+ FEDERATION_GPG_POLICY_PREFIX + "blacklist";
+
/**
* Connection and Read timeout from the Router to RM.
*/
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
index f722af852f4..b643bd8d08d 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
@@ -5426,4 +5426,46 @@
100ms
+
+
+ The interval at which the policy generator runs, default is one hour
+
+yarn.federation.gpg.policy.generator.interval
+1h
+
+
+
+
+ The interval at which the policy generator runs, default is one hour.
+ This is an deprecated property, We better set it
+ `yarn.federation.gpg.policy.generator.interval`.
+
+
[hadoop] branch trunk updated: YARN-11520. Support capacity vector for AQCv2 dynamic templates. (#5871)
This is an automated email from the ASF dual-hosted git repository.
bteke pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 6dafd536264 YARN-11520. Support capacity vector for AQCv2 dynamic
templates. (#5871)
6dafd536264 is described below
commit 6dafd53626409a568464281f01af47a9ee85432b
Author: Benjamin Teke
AuthorDate: Thu Jul 27 17:33:19 2023 +0200
YARN-11520. Support capacity vector for AQCv2 dynamic templates. (#5871)
---
.../scheduler/capacity/AbstractCSQueue.java| 64 ++--
.../scheduler/capacity/AbstractLeafQueue.java | 4 +-
.../scheduler/capacity/AutoCreatedLeafQueue.java | 2 +-
...lerNewQueueAutoCreationWithCapacityVectors.java | 325 +
4 files changed, 358 insertions(+), 37 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
index 27a4ca6b432..26e14afc0fd 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
@@ -349,7 +349,7 @@ public abstract class AbstractCSQueue implements CSQueue {
this.acls = configuration.getAcls(getQueuePath());
if (isDynamicQueue() || this instanceof AbstractAutoCreatedLeafQueue) {
-setDynamicQueueProperties();
+parseAndSetDynamicTemplates();
setDynamicQueueACLProperties();
}
@@ -389,21 +389,14 @@ public abstract class AbstractCSQueue implements CSQueue {
this.queueNodeLabelsSettings.getConfiguredNodeLabels(),
QueueCapacityVector.newInstance());
- // Preserving the capacities set by Entitlements, see:
ReservationSystem.md
- if (this instanceof ReservationQueue ||
- this instanceof PlanQueue) {
-for (final String label :
queueNodeLabelsSettings.getConfiguredNodeLabels()) {
- setConfiguredMinCapacityVector(label,
- QueueCapacityVector.of(queueCapacities.getCapacity(label) * 100,
- QueueCapacityVector.ResourceUnitCapacityType.PERCENTAGE));
- setConfiguredMaxCapacityVector(label,
- QueueCapacityVector.of(queueCapacities.getMaximumCapacity(label)
* 100,
- QueueCapacityVector.ResourceUnitCapacityType.PERCENTAGE));
-}
- }
-
- // Re-adjust weight when mixed capacity type is used. 5w == [memory=5w,
vcores=5w]
for (final String label :
queueNodeLabelsSettings.getConfiguredNodeLabels()) {
+// Manually sets the capacity vector for:
+// 1. Dynamic queues that have no configured capacity vectors defined
by templates
+// 2. ReservationQueue and PlanQueue instances which have their
capacities set by entitlements and need to be
+//preserved during a restart, see: ReservationSystem.md
+overrideCapacityVectorsForSpecialQueues(label);
+
+// Re-adjust weight when mixed capacity type is used. 5w ==
[memory=5w, vcores=5w]
final QueueCapacityVector capacityVector =
configuredCapacityVectors.get(label);
final Set
definedCapacityTypes =
capacityVector.getDefinedCapacityTypes();
@@ -444,12 +437,10 @@ public abstract class AbstractCSQueue implements CSQueue {
/**
* Set properties specific to dynamic queues.
*/
- protected void setDynamicQueueProperties() {
-// Set properties from parent template
-if (parent instanceof AbstractParentQueue && isDynamicQueue()) {
+ protected void parseAndSetDynamicTemplates() {
+// Set the template properties from the parent to the queuepath of the
child
((AbstractParentQueue) parent).getAutoCreatedQueueTemplate()
- .setTemplateEntriesForChild(queueContext.getConfiguration(),
getQueuePath(),
- this instanceof AbstractLeafQueue);
+ .setTemplateEntriesForChild(queueContext.getConfiguration(),
getQueuePath(), this instanceof AbstractLeafQueue);
String parentTemplate = String.format("%s.%s", parent.getQueuePath(),
AutoCreatedQueueTemplate.AUTO_QUEUE_TEMPLATE_PREFIX);
@@ -459,28 +450,33 @@ public abstract class AbstractCSQueue implements CSQueue {
.getConfiguredNodeLabelsForAllQueues()
.getLabelsByQueue(parentTemplate);
- if (parentNodeLabels != null) {
-if (parentNodeLabels.size() > 1) {
+ if
[hadoop] branch trunk updated (b9712223722 -> d1c52bffbdd)
This is an automated email from the ASF dual-hosted git repository.
bteke pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
from b9712223722 HDFS-17120. Support snapshot diff based copylisting for
flat paths. (#5885)
add d1c52bffbdd YARN-11521. Parameterize tests runs with legacy/non-legacy
queue-mode. (#5883)
No new revisions were added by this update.
Summary of changes:
.../webapp/TestRMWebServicesCapacitySched.java | 18 +-
...estRMWebServicesCapacitySchedDynamicConfig.java | 37 ++-
...ebServicesCapacitySchedLegacyQueueCreation.java | 17 +-
...estRMWebServicesCapacitySchedulerMixedMode.java | 2 +-
.../TestRMWebServicesForCSWithPartitions.java | 83 +++
.../resourcemanager/webapp/TestWebServiceUtil.java | 58 -
.../webapp/dynamic-testAbsoluteMode-0.json | 272 ++---
...json => dynamic-testAbsoluteMode-legacy-0.json} | 0
.../webapp/dynamic-testPercentageMode-0.json | 68 +++---
...on => dynamic-testPercentageMode-legacy-0.json} | 0
.../resources/webapp/dynamic-testWeightMode-0.json | 68 +++---
.../webapp/dynamic-testWeightMode-16.json | 48 ++--
.../webapp/dynamic-testWeightMode-32.json | 48 ++--
.../webapp/dynamic-testWeightMode-after-aqc.json | 200 +++
.../webapp/dynamic-testWeightMode-before-aqc.json | 48 ++--
...0.json => dynamic-testWeightMode-legacy-0.json} | 0
json => dynamic-testWeightMode-legacy-16.json} | 0
json => dynamic-testWeightMode-legacy-32.json} | 0
...> dynamic-testWeightMode-legacy-after-aqc.json} | 0
... dynamic-testWeightMode-legacy-before-aqc.json} | 0
20 files changed, 520 insertions(+), 447 deletions(-)
copy
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/{dynamic-testAbsoluteMode-0.json
=> dynamic-testAbsoluteMode-legacy-0.json} (100%)
copy
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/{dynamic-testPercentageMode-0.json
=> dynamic-testPercentageMode-legacy-0.json} (100%)
copy
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/{dynamic-testWeightMode-0.json
=> dynamic-testWeightMode-legacy-0.json} (100%)
copy
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/{dynamic-testWeightMode-16.json
=> dynamic-testWeightMode-legacy-16.json} (100%)
copy
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/{dynamic-testWeightMode-32.json
=> dynamic-testWeightMode-legacy-32.json} (100%)
copy
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/{dynamic-testWeightMode-after-aqc.json
=> dynamic-testWeightMode-legacy-after-aqc.json} (100%)
copy
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/webapp/{dynamic-testWeightMode-before-aqc.json
=> dynamic-testWeightMode-legacy-before-aqc.json} (100%)
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated (b1fc00d4b22 -> b9712223722)
This is an automated email from the ASF dual-hosted git repository. umamahesh pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git from b1fc00d4b22 YARN-11539. Fix leaf-templates in Flexible AQC. (#5868) add b9712223722 HDFS-17120. Support snapshot diff based copylisting for flat paths. (#5885) No new revisions were added by this update. Summary of changes: .../org/apache/hadoop/tools/DistCpConstants.java | 4 ++ .../apache/hadoop/tools/GlobbedCopyListing.java| 2 +- .../org/apache/hadoop/tools/RegexCopyFilter.java | 1 + .../org/apache/hadoop/tools/SimpleCopyListing.java | 70 +++--- .../org/apache/hadoop/tools/TestCopyListing.java | 68 + 5 files changed, 122 insertions(+), 23 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated (1d09dcc614e -> b1fc00d4b22)
This is an automated email from the ASF dual-hosted git repository. bteke pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git from 1d09dcc614e HDFS-17117. Print reconstructionQueuesInitProgress periodically when BlockManager processMisReplicatesAsync. (#5877). Contributed by Haiyang Hu. add b1fc00d4b22 YARN-11539. Fix leaf-templates in Flexible AQC. (#5868) No new revisions were added by this update. Summary of changes: .../scheduler/capacity/AbstractCSQueue.java| 3 +- ...estRMWebServicesCapacitySchedDynamicConfig.java | 45 ++-- .../webapp/dynamic-testWeightMode-after-aqc.json | 299 +++-- .../webapp/dynamic-testWeightMode-before-aqc.json | 7 +- 4 files changed, 183 insertions(+), 171 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
