[ https://issues.apache.org/jira/browse/YARN-10082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Szilard Nemeth updated YARN-10082: ---------------------------------- Fix Version/s: 3.3.0 > FS-CS converter: disable terminal placement rule checking > --------------------------------------------------------- > > Key: YARN-10082 > URL: https://issues.apache.org/jira/browse/YARN-10082 > Project: Hadoop YARN > Issue Type: Sub-task > Reporter: Peter Bacsko > Assignee: Peter Bacsko > Priority: Critical > Fix For: 3.3.0 > > Attachments: YARN-10082-001.patch, YARN-10082-002.patch > > > Before YARN-8967, {{QueuePlacementRule}} class had a method called > {{isTerminal()}}. However, sometimes this method was hard-coded to return > false, accepting such configurations as: > {noformat} > <queuePlacementPolicy> > <rule name="specified" create="true"/> > <rule name="nestedUserQueue" create="true"> > <rule name="default" create="true" queue="users"/> > </rule> > <rule name="default"/> > </queuePlacementPolicy> > {noformat} > It's because {{NestedUserQueue.isTerminal()}} always returns {{false}}. > This changed after YARN-8967. Now, this configuration is not accepted because > {{QueuePlacementPolicy.fromXml()}} calculates a list of terminal rules > differently: > https://github.com/apache/hadoop/blob/5257f50abb71905ef3068fd45541d00ce9e8f355/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueuePlacementPolicy.java#L176-L183 > In order to migrate existing configuration that were created before > YARN-8967, we need a new switch (at least in migration mode) in FS to turn > off this validation, otherwise the tool will not be able to migrate these > configs and the following exception will be thrown: > {noformat} > ~$ ./yarn fs2cs -y /tmp/yarn-site.xml -f /tmp/fair-scheduler.xml -o /tmp > WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS. > 20/01/13 05:48:20 INFO converter.FSConfigToCSConfigConverter: Output > directory for yarn-site.xml and capacity-scheduler.xml is: /tmp > 20/01/13 05:48:20 INFO converter.FSConfigToCSConfigConverter: Conversion > rules file is not defined, using default conversion config! > 20/01/13 05:48:21 INFO converter.FSConfigToCSConfigConverter: Using > explicitly defined fair-scheduler.xml > WARNING: This feature is experimental and not intended for production use! > 20/01/13 05:48:21 INFO conf.Configuration: resource-types.xml not found > 20/01/13 05:48:21 INFO resource.ResourceUtils: Unable to find > 'resource-types.xml'. > 20/01/13 05:48:21 INFO security.YarnAuthorizationProvider: > org.apache.hadoop.yarn.security.ConfiguredYarnAuthorizer is instantiated. > 20/01/13 05:48:21 INFO scheduler.AbstractYarnScheduler: Minimum allocation = > <memory:1024, vCores:1> > 20/01/13 05:48:21 INFO scheduler.AbstractYarnScheduler: Maximum allocation = > <memory:8192, vCores:2> > 20/01/13 05:48:21 INFO placement.PlacementFactory: Creating PlacementRule > implementation: class > org.apache.hadoop.yarn.server.resourcemanager.placement.SpecifiedPlacementRule > 20/01/13 05:48:21 INFO placement.PlacementFactory: Creating PlacementRule > implementation: class > org.apache.hadoop.yarn.server.resourcemanager.placement.UserPlacementRule > 20/01/13 05:48:21 INFO fair.AllocationFileLoaderService: Loading allocation > file file:/tmp/fair-scheduler.xml > 20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule > implementation: class > org.apache.hadoop.yarn.server.resourcemanager.placement.SpecifiedPlacementRule > 20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule > implementation: class > org.apache.hadoop.yarn.server.resourcemanager.placement.UserPlacementRule > 20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule > implementation: class > org.apache.hadoop.yarn.server.resourcemanager.placement.DefaultPlacementRule > 20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule > implementation: class > org.apache.hadoop.yarn.server.resourcemanager.placement.DefaultPlacementRule > 20/01/13 05:48:22 INFO service.AbstractService: Service > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler > failed in state INITED > java.io.IOException: Failed to initialize FairScheduler > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1438) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.serviceInit(FairScheduler.java:1479) > at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:206) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:101) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:116) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:44) > Caused by: > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationConfigurationException: > Rules after rule 2 in queue placement policy can never be reached > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy.updateRuleSet(QueuePlacementPolicy.java:110) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy.fromXml(QueuePlacementPolicy.java:187) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.getQueuePlacementPolicy(AllocationFileLoaderService.java:276) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.reloadAllocations(AllocationFileLoaderService.java:257) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1436) > ... 6 more > 20/01/13 05:48:22 ERROR converter.FSConfigToCSConfigConverterMain: Error > while starting FS configuration conversion! > org.apache.hadoop.service.ServiceStateException: java.io.IOException: Failed > to initialize FairScheduler > at > org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105) > at org.apache.hadoop.service.AbstractService.init(AbstractService.java:173) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:206) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:101) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:116) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:44) > Caused by: java.io.IOException: Failed to initialize FairScheduler > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1438) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.serviceInit(FairScheduler.java:1479) > at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > ... 4 more > Caused by: > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationConfigurationException: > Rules after rule 2 in queue placement policy can never be reached > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy.updateRuleSet(QueuePlacementPolicy.java:110) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy.fromXml(QueuePlacementPolicy.java:187) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.getQueuePlacementPolicy(AllocationFileLoaderService.java:276) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.reloadAllocations(AllocationFileLoaderService.java:257) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1436) > ... 6 more > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org