Tamas Domok created YARN-11422: ---------------------------------- Summary: fs2cs failes to generate valid capacity-scheduler.xml Key: YARN-11422 URL: https://issues.apache.org/jira/browse/YARN-11422 Project: Hadoop YARN Issue Type: Bug Components: capacityscheduler Affects Versions: 3.4.0 Reporter: Tamas Domok Assignee: Tamas Domok Attachments: fair-scheduler-issue.xml, yarn-site.xml
Attachments: - [^fair-scheduler-issue.xml] - [^yarn-site.xml] Save them to /tmp/fair-scheduler.xml and /tmp/yarn-site.xml. Run the following command: {code} rm -rf /tmp/output/*; yarn fs2cs --cluster-resource memory-mb=36201037824,vcores=4914 --no-terminal-rule-check -y /tmp/yarn-site.xml -f /tmp/fair-scheduler.xml -o /tmp/output {code} The conversion fails with the following error: {code} 2023-01-20 13:37:42,250 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler failed in state STOPPED java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceStop(CapacityScheduler.java:428) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221) at org.apache.hadoop.service.AbstractService.close(AbstractService.java:248) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.ConvertedConfigValidator.validateConvertedConfig(ConvertedConfigValidator.java:74) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:179) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:47) 2023-01-20 13:37:42,251 ERROR converter.ConvertedConfigValidator: Could not start Capacity Scheduler org.apache.hadoop.yarn.exceptions.YarnException: Failed to initialize queues at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:807) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:318) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:412) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.ConvertedConfigValidator.validateConvertedConfig(ConvertedConfigValidator.java:70) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:179) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:47) Suppressed: java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceStop(CapacityScheduler.java:428) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221) at org.apache.hadoop.service.AbstractService.close(AbstractService.java:248) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.ConvertedConfigValidator.validateConvertedConfig(ConvertedConfigValidator.java:74) ... 2 more Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct queue ordering policy=fair queue=root.default at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:2053) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:172) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.<init>(ParentQueue.java:140) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.<init>(ParentQueue.java:115) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:276) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:283) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:176) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:798) ... 5 more Caused by: java.lang.ClassNotFoundException: fair at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:2049) ... 12 more 2023-01-20 13:37:42,251 ERROR converter.FSConfigToCSConfigArgumentHandler: Verification failed: Failed to initialize queues Verification failed: Failed to initialize queues 2023-01-20 13:37:42,251 ERROR converter.FSConfigToCSConfigConverterMain: Error while starting FS configuration conversion, see previous error messages for details! {code} With the following manual changes on the generated capacity-scheduler.xml: - add: yarn.scheduler.capacity.root.users.auto-queue-creation-v2.enabled=true - remove: yarn.scheduler.capacity.root.default.auto-queue-creation-v2.enabled=true - remove: yarn.scheduler.capacity.root.users.ordering-policy=fair RM can start up. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org