[
https://issues.apache.org/jira/browse/YARN-10958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10958:
------------------------------
Component/s: capacity scheduler
Hadoop Flags: Reviewed
Target Version/s: 3.4.0
Affects Version/s: 3.4.0
> Use correct configuration for Group service init in CSMappingPlacementRule
> --------------------------------------------------------------------------
>
> Key: YARN-10958
> URL: https://issues.apache.org/jira/browse/YARN-10958
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
> Affects Versions: 3.4.0
> Reporter: Peter Bacsko
> Assignee: Szilard Nemeth
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> There is a potential problem in {{CSMappingPlacementRule.java}}:
> {noformat}
> if (groups == null) {
> groups = Groups.getUserToGroupsMappingService(conf);
> }
> {noformat}
> The problem is, we're supposed to pass {{scheduler.getConf()}}. The "conf"
> object is the config for capacity scheduler, which does not include the
> property which selects the group service provider. Therefore, the current
> code just works by chance, because Group mapping service is already
> initialized at this point. See the original fix in YARN-10053.
> Also, need a unit test to verify it.
> Idea:
> # Create a Configuration object in which the property
> "hadoop.security.group.mapping" refers to an existing a test implementation.
> # Add a new method to {{Groups}} which nulls out the singleton instance, eg.
> {{Groups.reset()}}.
> # Create a mock CapacityScheduler where {{getConf()}} and
> {{getConfiguration()}} contain different settings for
> "hadoop.security.group.mapping". Since {{getConf()}} is the service config,
> this should return the config object created in step #1.
> # Create an instance of {{CSMappingPlacementRule}} with a single primary
> group rule.
> # Run the placement evaluation.
> # Expected: returned queue matches what is supposed to be coming from the
> test group mapping service ("testuser" --> "testqueue").
> # Modify "hadoop.security.group.mapping" in the config object created in
> step #1.
> # Call {{Groups.refresh()}} which changes the group mapping ("testuser" -->
> "testqueue2"). This requires that the test group mapping service implement
> {{GroupMappingServiceProvider.cacheGroupsRefresh()}}.
> # Create a new instance of {{CSMappingPlacementRule}}.
> # Run the placement evaluation again
> # Expected: with the same user, the target queue has changed.
> This looks convoluted, but these steps make sure that:
> # {{CSMappingPlacementRule}} will force the initialization of groups.
> # We select the correct configuration for group service init.
> # We don't create a new {{Groups}} instance if the singleton is initialized,
> so we cover the original problem described in YARN-10597.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
