[jira] [Created] (YARN-10148) Add Unit test for queue ACL for both FS and CS
Kinga Marton created YARN-10148: --- Summary: Add Unit test for queue ACL for both FS and CS Key: YARN-10148 URL: https://issues.apache.org/jira/browse/YARN-10148 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Reporter: Kinga Marton Assignee: Kinga Marton Add some unit tests covering the queue ACL evaluation for both FS and CS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10117) FS-CS converter: adjust queue ACL to have the same output with CS as for FS has
[ https://issues.apache.org/jira/browse/YARN-10117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton resolved YARN-10117. - Resolution: Not A Problem > FS-CS converter: adjust queue ACL to have the same output with CS as for FS > has > --- > > Key: YARN-10117 > URL: https://issues.apache.org/jira/browse/YARN-10117 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Kinga Marton >Priority: Major > > Both FS and CS seems to check the ACL recursively: from the leaf via the > parent(s) to the root (inclusive). However there are some differences in > evaluating them, what cause to have different results with the two schedulers. > Some examples are the following ones: > ||Tested scenario||FS output||CS output|| > |Root - “ ” > C - * > C1 - *|Denied by root ACL|OK| > |Root - “ ” > C - “ ” > C1 - “ ”|Denied by root ACL|OK| > |Root - “ ” > C - * > C1 - “ ”|Denied by root ACL|OK| > |Root - “ ” > C - “ ” > C1 - *|Denied by root ACL|OK| > Note: I have set the same value for both submit application and administer > queue ACLs -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-10117) FS-CS converter: adjust queue ACL to have the same output with CS as for FS has
Kinga Marton created YARN-10117: --- Summary: FS-CS converter: adjust queue ACL to have the same output with CS as for FS has Key: YARN-10117 URL: https://issues.apache.org/jira/browse/YARN-10117 Project: Hadoop YARN Issue Type: Sub-task Reporter: Kinga Marton Both FS and CS seems to check the ACL recursively: from the leaf via the parent(s) to the root (inclusive). However there are some differences in evaluating them, what cause to have different results with the two schedulers. Some examples are the following ones: ||Tested scenario||FS output||CS output|| |Root - “ ” C - * C1 - *|Denied by root ACL|OK| |Root - “ ” C - “ ” C1 - “ ”|Denied by root ACL|OK| |Root - “ ” C - * C1 - “ ”|Denied by root ACL|OK| |Root - “ ” C - “ ” C1 - *|Denied by root ACL|OK| Note: I have set the same value for both submit application and administer queue ACLs -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-10100) [CS] Separate config validation steps from the update part
Kinga Marton created YARN-10100: --- Summary: [CS] Separate config validation steps from the update part Key: YARN-10100 URL: https://issues.apache.org/jira/browse/YARN-10100 Project: Hadoop YARN Issue Type: Improvement Components: capacity scheduler Reporter: Kinga Marton Assignee: Kinga Marton In Capacity Scheduler initialisation/reinitialisation there are a lot of validation steps performed. Some of this steps are deeply hidden. Having this implementation is really hard to figure out what a valid configuration means. Let's figure out what are exactly the validation steps and separate them from the update part. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-10070) NPE if no rule is defined and application-tag-based-placement is enabled
Kinga Marton created YARN-10070:
---
Summary: NPE if no rule is defined and
application-tag-based-placement is enabled
Key: YARN-10070
URL: https://issues.apache.org/jira/browse/YARN-10070
Project: Hadoop YARN
Issue Type: Bug
Reporter: Kinga Marton
Assignee: Kinga Marton
If there is no rule defined for a user NPE is thrown by the following line.
{code:java}
String queue = placementManager
.placeApplication(context, usernameUsedForPlacement).getQueue();{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
Kinga Marton created YARN-10022: --- Summary: Create RM Rest API to validate a CapacityScheduler Configuration Key: YARN-10022 URL: https://issues.apache.org/jira/browse/YARN-10022 Project: Hadoop YARN Issue Type: New Feature Reporter: Kinga Marton Assignee: Kinga Marton RMWebService should expose a new api which gets a CapacityScheduler Configuration as an input, validates it and returns success / failure. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-9890) [UI2] Add Application tag to the app table and app detail page.
Kinga Marton created YARN-9890: -- Summary: [UI2] Add Application tag to the app table and app detail page. Key: YARN-9890 URL: https://issues.apache.org/jira/browse/YARN-9890 Project: Hadoop YARN Issue Type: Sub-task Reporter: Kinga Marton Assignee: Kinga Marton Right now AFAIK there is no possibility to filter the applications based on the application tag in the UI. Adding this new column to the app table will make this filtering possible as well. >From the UI2 this information is missing from the application detail page as >well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-9889) [UI] Add Application Tag column to RM All Applications table
Kinga Marton created YARN-9889: -- Summary: [UI] Add Application Tag column to RM All Applications table Key: YARN-9889 URL: https://issues.apache.org/jira/browse/YARN-9889 Project: Hadoop YARN Issue Type: Sub-task Reporter: Kinga Marton Assignee: Kinga Marton Right now AFAIK there is no possibility to filter the applications based on the application tag in the UI. Adding this new column to the app table will make this filtering possible as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-9886) Queue mapping based on userid passed through application tag
Kinga Marton created YARN-9886: -- Summary: Queue mapping based on userid passed through application tag Key: YARN-9886 URL: https://issues.apache.org/jira/browse/YARN-9886 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Reporter: Kinga Marton Assignee: Kinga Marton There are situations when the real submitting user differs from the user what arrives to YARN. For example in case of a Hive application when Hive impersonation is turned off, the hive queries will run as Hive user and the mapping is done based on this username. Unfortunately in this case YARN doesn't have any information about the real user and there are cases when the customer may want to map this applications to the real submitting user's queue instead of the Hive one. For this cases if they would pass the username in the application tag we may read it and use that one during the queue mapping, if that user has rights to run on the real user's queue. [~sunilg] please correct me if I missed something. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-9793) Duplicate information in TimelineServiceV2.md
Julia Kinga Marton created YARN-9793: Summary: Duplicate information in TimelineServiceV2.md Key: YARN-9793 URL: https://issues.apache.org/jira/browse/YARN-9793 Project: Hadoop YARN Issue Type: Bug Components: docs Reporter: Julia Kinga Marton Assignee: Julia Kinga Marton In the documentation of the ATSv2, TimelineEntity objects description part there is a duplication: * configs: A map from a string (config name) to a string (config value) representing all configs associated with the entity. Users can post the whole config or a part of it in the configs field. *Supported for application and generic entities. Supported for application and generic entities.* -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-9784) TestLeafQueue is flaky
Julia Kinga Marton created YARN-9784:
Summary: TestLeafQueue is flaky
Key: YARN-9784
URL: https://issues.apache.org/jira/browse/YARN-9784
Project: Hadoop YARN
Issue Type: Bug
Components: test
Affects Versions: 3.3.0
Reporter: Julia Kinga Marton
Assignee: Julia Kinga Marton
There are some test cases in TestLeafQueue which are failing intermittently.
>From 100 runs, there were 16 failures.
Some failure examples are the following ones:
{code:java}
2019-08-26 13:18:13 [ERROR] Errors:
2019-08-26 13:18:13 [ERROR] TestLeafQueue.setUp:144->setUpInternal:221
WrongTypeOfReturnValue
2019-08-26 13:18:13 YarnConfigu...
2019-08-26 13:18:13 [ERROR] TestLeafQueue.setUp:144->setUpInternal:221
WrongTypeOfReturnValue
2019-08-26 13:18:13 YarnConfigu...
2019-08-26 13:18:13 [INFO]
2019-08-26 13:18:13 [ERROR] Tests run: 36, Failures: 0, Errors: 2, Skipped: 0
{code}
{code:java}
2019-08-26 13:18:09 [ERROR] Failures:
2019-08-26 13:18:09 [ERROR] TestLeafQueue.testHeadroomWithMaxCap:1373
expected:<2048> but was:<0>
2019-08-26 13:18:09 [INFO]
2019-08-26 13:18:09 [ERROR] Tests run: 36, Failures: 1, Errors: 0, Skipped: 0
{code}
{code:java}
2019-08-26 13:18:18 [ERROR] Errors:
2019-08-26 13:18:18 [ERROR] TestLeafQueue.setUp:144->setUpInternal:221
WrongTypeOfReturnValue
2019-08-26 13:18:18 YarnConfigu...
2019-08-26 13:18:18 [ERROR] TestLeafQueue.testHeadroomWithMaxCap:1307 ?
ClassCast org.apache.hadoop.yarn.c...
2019-08-26 13:18:18 [INFO]
2019-08-26 13:18:18 [ERROR] Tests run: 36, Failures: 0, Errors: 2, Skipped: 0
{code}
{code:java}
2019-08-26 13:18:10 [ERROR] Failures:
2019-08-26 13:18:10 [ERROR] TestLeafQueue.testDRFUserLimits:847 Verify user_0
got resources
2019-08-26 13:18:10 [INFO]
2019-08-26 13:18:10 [ERROR] Tests run: 36, Failures: 1, Errors: 0, Skipped: 0
{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
