[ https://issues.apache.org/jira/browse/YARN-11511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benjamin Teke resolved YARN-11511. ---------------------------------- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Improve TestRMWebServices test config and data > ---------------------------------------------- > > Key: YARN-11511 > URL: https://issues.apache.org/jira/browse/YARN-11511 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler > Affects Versions: 3.4.0 > Reporter: Tamas Domok > Assignee: Bence Kosztolnik > Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > We found multiple configuration issues with > *TestRMWebServicesCapacitySched.java* and > *TestRMWebServicesCapacitySchedDynamicConfig.java*. > h3. 1. createMockRM created the RM with a non-intuitive queue config > (createMockRM was used from the TestRMWebServicesCapacitySchedDynamicConfig > where this was not expected) > Fix: > {code} > diff --git > a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java > > b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java > index ec65237fa6e..378f16e981a 100644 > --- > a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java > +++ > b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java > @@ -108,13 +108,13 @@ public TestRMWebServicesCapacitySched() { > @Override > public void setUp() throws Exception { > super.setUp(); > - rm = createMockRM(new CapacitySchedulerConfiguration( > - new Configuration(false))); > + rm = createMockRM(setupQueueConfiguration(new > CapacitySchedulerConfiguration( > + new Configuration(false)))); > GuiceServletConfig.setInjector( > Guice.createInjector(new WebServletModule(rm))); > } > - public static void setupQueueConfiguration( > + public CapacitySchedulerConfiguration setupQueueConfiguration( > CapacitySchedulerConfiguration config) { > // Define top-level queues > @@ -167,6 +167,8 @@ public static void setupQueueConfiguration( > config.setAutoCreateChildQueueEnabled(a1C, true); > config.setInt(PREFIX + a1C + DOT + > AUTO_CREATED_LEAF_QUEUE_TEMPLATE_PREFIX > + DOT + CAPACITY, 50); > + > + return config; > } > @Test > @@ -407,7 +409,6 @@ public static WebAppDescriptor createWebAppDescriptor() { > } > public static MockRM createMockRM(CapacitySchedulerConfiguration csConf) { > - setupQueueConfiguration(csConf); > YarnConfiguration conf = new YarnConfiguration(csConf); > conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class, > ResourceScheduler.class); > {code} > h3. 2. setupQueueConfiguration creates a mixed queue hierarchy (percentage > and absolute) > {code} > final String c = CapacitySchedulerConfiguration.ROOT + ".c"; > config.setCapacity(c, "[memory=1024]"); > {code} > root.c is configured in absolute mode while root.a and root.b are configured > in percentage > setupQueueConfiguration should be simplified, do the configuration like the > other tests (create a map with the string key value pairs) > h3. 3. createAbsoluteConfigLegacyAutoCreation does not set capacity for the > default queue > That makes it mixed (percentage + absolute) > h3. 4. initAutoQueueHandler is called with wrong units > The * GB is unnecessary, and the vcores should be configured too with a value > that makes sense. > h3. 5. CSConfigGenerator static class makes the tests hard to read. > The test cases should just have their configuration assembled in them. > h3. 6. testSchedulerResponseAbsoluteMode does not add any node > No cluster resource -> no effectiveMin/Max resource. > h1. Proposal > These tests need a rework, the configurations should be easy to follow and > the calculated effectiveMin/Max (and any other calculated value) should > result in reasonable numbers. The goal is to have a end to end like test > suite that verifies the queue hierarchy initialisation. > The queue hierarchies should be simple but at least 2 level, e.g.: > root.default > root.test_1.test_1_1 > root.test_1.test_1_2 > root.test_2 > The helper methods could be moved to a separate utility class from > TestRMWebServicesCapacitySched. > TestRMWebServicesCapacitySched can be kept for the basic tests (json/xml > slash at the end, apps, node-labels and resource-info). > AFAIK, the difference between the TestRMWebServicesCapacitySched and the > TestRMWebServicesCapacitySchedDynamicConfig is that the latter is a mutable > CS: > {code} > config.set(YarnConfiguration.SCHEDULER_CONFIGURATION_STORE_CLASS, > YarnConfiguration.MEMORY_CONFIGURATION_STORE); > {code} > The tests in the TestRMWebServicesCapacitySchedDynamicConfig could start with > zero resource then add some nodes and remove some nodes to verify the > calculations. > We should have tests for Absolute, Percentage, Weight mode. > Legacy AQC and Flexible AQC (multiple level queue creation, static child > queue) should be also tested. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org