[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926257#comment-16926257
]
Jonathan Hung commented on YARN-8948:
-
FYI I have pushed this to branch-3.2.
> PlacementRule interface should be for all YarnSchedulers
>
>
> Key: YARN-8948
> URL: https://issues.apache.org/jira/browse/YARN-8948
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Fix For: 3.3.0, 3.2.2
>
> Attachments: YARN-8948.001.patch, YARN-8948.002.patch,
> YARN-8948.003.patch, YARN-8948.004.patch, YARN-8948.005.patch,
> YARN-8948.006.patch
>
>
> *Issue 1:*
> YARN-3635 intention was to add PlacementRule interface common for all
> YarnSchedules.
> {code}
> 33 public abstract boolean initialize(
> 34 CapacitySchedulerContext schedulerContext) throws IOException;
> {code}
> PlacementRule initialization is done using CapacitySchedulerContext binding
> to CapacityScheduler
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703450#comment-16703450
]
Hudson commented on YARN-8948:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15526 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15526/])
YARN-8948. PlacementRule interface should be for all YarnSchedulers.
(bibinchundatt: rev a68d766e876631d7ee2e1a6504d4120ba628d178)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/AppNameMappingPlacementRule.java
> PlacementRule interface should be for all YarnSchedulers
>
>
> Key: YARN-8948
> URL: https://issues.apache.org/jira/browse/YARN-8948
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-8948.001.patch, YARN-8948.002.patch,
> YARN-8948.003.patch, YARN-8948.004.patch, YARN-8948.005.patch,
> YARN-8948.006.patch
>
>
> *Issue 1:*
> YARN-3635 intention was to add PlacementRule interface common for all
> YarnSchedules.
> {code}
> 33 public abstract boolean initialize(
> 34 CapacitySchedulerContext schedulerContext) throws IOException;
> {code}
> PlacementRule initialization is done using CapacitySchedulerContext binding
> to CapacityScheduler
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703010#comment-16703010
]
Bibin A Chundatt commented on YARN-8948:
Based on +1 from [~suma.shivaprasad].If no one complains i will commit it by
EOD.
> PlacementRule interface should be for all YarnSchedulers
>
>
> Key: YARN-8948
> URL: https://issues.apache.org/jira/browse/YARN-8948
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-8948.001.patch, YARN-8948.002.patch,
> YARN-8948.003.patch, YARN-8948.004.patch, YARN-8948.005.patch,
> YARN-8948.006.patch
>
>
> *Issue 1:*
> YARN-3635 intention was to add PlacementRule interface common for all
> YarnSchedules.
> {code}
> 33 public abstract boolean initialize(
> 34 CapacitySchedulerContext schedulerContext) throws IOException;
> {code}
> PlacementRule initialization is done using CapacitySchedulerContext binding
> to CapacityScheduler
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703009#comment-16703009
]
Bibin A Chundatt commented on YARN-8948:
Based on +1 from [~suma.shivaprasad].If no one complains i will commit it by
EOD.
> PlacementRule interface should be for all YarnSchedulers
>
>
> Key: YARN-8948
> URL: https://issues.apache.org/jira/browse/YARN-8948
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-8948.001.patch, YARN-8948.002.patch,
> YARN-8948.003.patch, YARN-8948.004.patch, YARN-8948.005.patch,
> YARN-8948.006.patch
>
>
> *Issue 1:*
> YARN-3635 intention was to add PlacementRule interface common for all
> YarnSchedules.
> {code}
> 33 public abstract boolean initialize(
> 34 CapacitySchedulerContext schedulerContext) throws IOException;
> {code}
> PlacementRule initialization is done using CapacitySchedulerContext binding
> to CapacityScheduler
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701519#comment-16701519
]
Bibin A Chundatt commented on YARN-8948:
Thanks [~suma.shivaprasad] for review
TestLeaderElectorService timeout with out patch also..
> PlacementRule interface should be for all YarnSchedulers
>
>
> Key: YARN-8948
> URL: https://issues.apache.org/jira/browse/YARN-8948
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-8948.001.patch, YARN-8948.002.patch,
> YARN-8948.003.patch, YARN-8948.004.patch, YARN-8948.005.patch,
> YARN-8948.006.patch
>
>
> *Issue 1:*
> YARN-3635 intention was to add PlacementRule interface common for all
> YarnSchedules.
> {code}
> 33 public abstract boolean initialize(
> 34 CapacitySchedulerContext schedulerContext) throws IOException;
> {code}
> PlacementRule initialization is done using CapacitySchedulerContext binding
> to CapacityScheduler
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700582#comment-16700582
]
Hadoop QA commented on YARN-8948:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 20 unchanged - 1 fixed = 20 total (was 21) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 26s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}105m 2s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}164m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8948 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949672/YARN-8948.006.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c0eb35a750eb 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 96c104d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/22728/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22728/testReport/ |
| Max. process+thread count | 893 (vs. ulimit of
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700453#comment-16700453
]
Suma Shivaprasad commented on YARN-8948:
Thanks [~bibinchundatt]. +1. Looks like UT failures are unrelated?
> PlacementRule interface should be for all YarnSchedulers
>
>
> Key: YARN-8948
> URL: https://issues.apache.org/jira/browse/YARN-8948
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-8948.001.patch, YARN-8948.002.patch,
> YARN-8948.003.patch, YARN-8948.004.patch, YARN-8948.005.patch,
> YARN-8948.006.patch
>
>
> *Issue 1:*
> YARN-3635 intention was to add PlacementRule interface common for all
> YarnSchedules.
> {code}
> 33 public abstract boolean initialize(
> 34 CapacitySchedulerContext schedulerContext) throws IOException;
> {code}
> PlacementRule initialization is done using CapacitySchedulerContext binding
> to CapacityScheduler
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700411#comment-16700411
]
Hadoop QA commented on YARN-8948:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 36s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}102m 50s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}155m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8948 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949655/YARN-8948.005.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 596575c25b40 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2730ead |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/22726/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22726/testReport/ |
| Max. process+thread count | 966 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700348#comment-16700348
]
Bibin A Chundatt commented on YARN-8948:
Thank [~suma.shivaprasad] for review.
Attached updated patch.
> PlacementRule interface should be for all YarnSchedulers
>
>
> Key: YARN-8948
> URL: https://issues.apache.org/jira/browse/YARN-8948
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-8948.001.patch, YARN-8948.002.patch,
> YARN-8948.003.patch, YARN-8948.004.patch, YARN-8948.005.patch,
> YARN-8948.006.patch
>
>
> *Issue 1:*
> YARN-3635 intention was to add PlacementRule interface common for all
> YarnSchedules.
> {code}
> 33 public abstract boolean initialize(
> 34 CapacitySchedulerContext schedulerContext) throws IOException;
> {code}
> PlacementRule initialization is done using CapacitySchedulerContext binding
> to CapacityScheduler
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700263#comment-16700263
]
Suma Shivaprasad commented on YARN-8948:
Thanks [~bibinchundatt]. Can we add the instance type validation for
CapacityScheduler back which was removed in the latter patches.
{noformat}
if(!(scheduler instanceof CapacityScheduler)){
70throw new IOException(
71"AppNameMappingPlacementRule can be configured only for "
72+ "CapacityScheduler");
73 }
74
{noformat}
> PlacementRule interface should be for all YarnSchedulers
>
>
> Key: YARN-8948
> URL: https://issues.apache.org/jira/browse/YARN-8948
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-8948.001.patch, YARN-8948.002.patch,
> YARN-8948.003.patch, YARN-8948.004.patch, YARN-8948.005.patch
>
>
> *Issue 1:*
> YARN-3635 intention was to add PlacementRule interface common for all
> YarnSchedules.
> {code}
> 33 public abstract boolean initialize(
> 34 CapacitySchedulerContext schedulerContext) throws IOException;
> {code}
> PlacementRule initialization is done using CapacitySchedulerContext binding
> to CapacityScheduler
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700170#comment-16700170
]
Bibin A Chundatt commented on YARN-8948:
[~wilfreds]
Attached patch removing unused imports too.
> PlacementRule interface should be for all YarnSchedulers
>
>
> Key: YARN-8948
> URL: https://issues.apache.org/jira/browse/YARN-8948
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-8948.001.patch, YARN-8948.002.patch,
> YARN-8948.003.patch, YARN-8948.004.patch, YARN-8948.005.patch
>
>
> *Issue 1:*
> YARN-3635 intention was to add PlacementRule interface common for all
> YarnSchedules.
> {code}
> 33 public abstract boolean initialize(
> 34 CapacitySchedulerContext schedulerContext) throws IOException;
> {code}
> PlacementRule initialization is done using CapacitySchedulerContext binding
> to CapacityScheduler
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695727#comment-16695727
]
Wilfred Spiegelenburg commented on YARN-8948:
-
Thank you [~bibinchundatt] for the updated patch:
We should remove the unused import in {{PlacementRule.java}} but beside that
the patch looks good, nice and simple.
test4tests -1 is OK, no new tests are needed.
The junit test failure is a known ZK issue and already tracked
> PlacementRule interface should be for all YarnSchedulers
>
>
> Key: YARN-8948
> URL: https://issues.apache.org/jira/browse/YARN-8948
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-8948.001.patch, YARN-8948.002.patch,
> YARN-8948.003.patch, YARN-8948.004.patch
>
>
> *Issue 1:*
> YARN-3635 intention was to add PlacementRule interface common for all
> YarnSchedules.
> {code}
> 33 public abstract boolean initialize(
> 34 CapacitySchedulerContext schedulerContext) throws IOException;
> {code}
> PlacementRule initialization is done using CapacitySchedulerContext binding
> to CapacityScheduler
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695233#comment-16695233
]
Hadoop QA commented on YARN-8948:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 46s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 20 unchanged - 0 fixed = 21 total (was 20) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 24s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}102m 32s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}160m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8948 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949084/YARN-8948.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 14346a6eb061 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 34b6fa7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/22668/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695027#comment-16695027
]
Bibin A Chundatt commented on YARN-8948:
[~wilfreds]/[~suma.shivaprasad]/[~sunilg]
Attached patch.
> PlacementRule interface should be for all YarnSchedulers
>
>
> Key: YARN-8948
> URL: https://issues.apache.org/jira/browse/YARN-8948
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-8948.001.patch, YARN-8948.002.patch,
> YARN-8948.003.patch, YARN-8948.004.patch
>
>
> *Issue 1:*
> YARN-3635 intention was to add PlacementRule interface common for all
> YarnSchedules.
> {code}
> 33 public abstract boolean initialize(
> 34 CapacitySchedulerContext schedulerContext) throws IOException;
> {code}
> PlacementRule initialization is done using CapacitySchedulerContext binding
> to CapacityScheduler
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684917#comment-16684917
]
Bibin A Chundatt commented on YARN-8948:
Thank you [~wilfreds] for review
We can change to ResoureScheduler, no issues . Initial implementation in
YARN-3635 was with rmcontext, so kept rmcontext.
Will change in next patch.
About second issue will move to another jira.
> PlacementRule interface should be for all YarnSchedulers
>
>
> Key: YARN-8948
> URL: https://issues.apache.org/jira/browse/YARN-8948
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-8948.001.patch, YARN-8948.002.patch,
> YARN-8948.003.patch
>
>
> *Issue 1:*
> YARN-3635 intention was to add PlacementRule interface common for all
> YarnSchedules.
> {code}
> 33 public abstract boolean initialize(
> 34 CapacitySchedulerContext schedulerContext) throws IOException;
> {code}
> PlacementRule initialization is done using CapacitySchedulerContext binding
> to CapacityScheduler
> *Issue 2:*
> {{yarn.scheduler.queue-placement-rules}} doesn't work as expected in Capacity
> Scheduler
> {quote}
> * **Queue Mapping Interface based on Default or User Defined Placement
> Rules** - This feature allows users to map a job to a specific queue based on
> some default placement rule. For instance based on user & group, or
> application name. User can also define their own placement rule.
> {quote}
> As per current UserGroupMapping is always added in placementRule.
> {{CapacityScheduler#updatePlacementRules}}
> {code}
> // Initialize placement rules
> Collection placementRuleStrs = conf.getStringCollection(
> YarnConfiguration.QUEUE_PLACEMENT_RULES);
> List placementRules = new ArrayList<>();
> ...
> // add UserGroupMappingPlacementRule if absent
> distingushRuleSet.add(YarnConfiguration.USER_GROUP_PLACEMENT_RULE);
> {code}
> PlacementRule configuration order is not maintained
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684892#comment-16684892
]
Wilfred Spiegelenburg commented on YARN-8948:
-
Hi [~bibinchundatt], thank you for the patch.
I have a question: why use the RMContext and not the {{ResourceScheduler}} as
the input for the PlacementRule interface? You pass in the {{RMContext}} and
really only use it to get the scheduler out It would remove the need for the
changes in the {{CapacityScheduler}}. The change would become much simpler and
the interface will then be usable for all scheduler. Using the
{{ResourcScheduler}} would also make using the interface in the FS much easier
The second change that you have added is really a different problem and is not
linked to the Interface change. Should that not be a seperate jira? It does not
rely on the Interface change at all.
> PlacementRule interface should be for all YarnSchedulers
>
>
> Key: YARN-8948
> URL: https://issues.apache.org/jira/browse/YARN-8948
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-8948.001.patch, YARN-8948.002.patch,
> YARN-8948.003.patch
>
>
> *Issue 1:*
> YARN-3635 intention was to add PlacementRule interface common for all
> YarnSchedules.
> {code}
> 33 public abstract boolean initialize(
> 34 CapacitySchedulerContext schedulerContext) throws IOException;
> {code}
> PlacementRule initialization is done using CapacitySchedulerContext binding
> to CapacityScheduler
> *Issue 2:*
> {{yarn.scheduler.queue-placement-rules}} doesn't work as expected in Capacity
> Scheduler
> {quote}
> * **Queue Mapping Interface based on Default or User Defined Placement
> Rules** - This feature allows users to map a job to a specific queue based on
> some default placement rule. For instance based on user & group, or
> application name. User can also define their own placement rule.
> {quote}
> As per current UserGroupMapping is always added in placementRule.
> {{CapacityScheduler#updatePlacementRules}}
> {code}
> // Initialize placement rules
> Collection placementRuleStrs = conf.getStringCollection(
> YarnConfiguration.QUEUE_PLACEMENT_RULES);
> List placementRules = new ArrayList<>();
> ...
> // add UserGroupMappingPlacementRule if absent
> distingushRuleSet.add(YarnConfiguration.USER_GROUP_PLACEMENT_RULE);
> {code}
> PlacementRule configuration order is not maintained
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681647#comment-16681647
]
Bibin A Chundatt commented on YARN-8948:
[~suma.shivaprasad]/[~sunilg] please review patch attached.
> PlacementRule interface should be for all YarnSchedulers
>
>
> Key: YARN-8948
> URL: https://issues.apache.org/jira/browse/YARN-8948
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-8948.001.patch, YARN-8948.002.patch,
> YARN-8948.003.patch
>
>
> *Issue 1:*
> YARN-3635 intention was to add PlacementRule interface common for all
> YarnSchedules.
> {code}
> 33 public abstract boolean initialize(
> 34 CapacitySchedulerContext schedulerContext) throws IOException;
> {code}
> PlacementRule initialization is done using CapacitySchedulerContext binding
> to CapacityScheduler
> *Issue 2:*
> {{yarn.scheduler.queue-placement-rules}} doesn't work as expected in Capacity
> Scheduler
> {quote}
> * **Queue Mapping Interface based on Default or User Defined Placement
> Rules** - This feature allows users to map a job to a specific queue based on
> some default placement rule. For instance based on user & group, or
> application name. User can also define their own placement rule.
> {quote}
> As per current UserGroupMapping is always added in placementRule.
> {{CapacityScheduler#updatePlacementRules}}
> {code}
> // Initialize placement rules
> Collection placementRuleStrs = conf.getStringCollection(
> YarnConfiguration.QUEUE_PLACEMENT_RULES);
> List placementRules = new ArrayList<>();
> ...
> // add UserGroupMappingPlacementRule if absent
> distingushRuleSet.add(YarnConfiguration.USER_GROUP_PLACEMENT_RULE);
> {code}
> PlacementRule configuration order is not maintained
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673037#comment-16673037
]
Hadoop QA commented on YARN-8948:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 870 unchanged - 1 fixed = 870 total (was 871)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 39s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 28s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}162m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueManagementDynamicEditPolicy
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8948 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946649/YARN-8948.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 67af53276c3d 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d16d5f7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/22406/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22406/testReport/ |
| Max. process+thread count
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673035#comment-16673035
]
Bibin A Chundatt commented on YARN-8948:
[~sunilg] Updating the severity . Not required to be a blocker.
> PlacementRule interface should be for all YarnSchedulers
>
>
> Key: YARN-8948
> URL: https://issues.apache.org/jira/browse/YARN-8948
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Attachments: YARN-8948.001.patch, YARN-8948.002.patch,
> YARN-8948.003.patch
>
>
> *Issue 1:*
> YARN-3635 intention was to add PlacementRule interface common for all
> YarnSchedules.
> {code}
> 33 public abstract boolean initialize(
> 34 CapacitySchedulerContext schedulerContext) throws IOException;
> {code}
> PlacementRule initialization is done using CapacitySchedulerContext binding
> to CapacityScheduler
> *Issue 2:*
> {{yarn.scheduler.queue-placement-rules}} doesn't work as expected in Capacity
> Scheduler
> {quote}
> * **Queue Mapping Interface based on Default or User Defined Placement
> Rules** - This feature allows users to map a job to a specific queue based on
> some default placement rule. For instance based on user & group, or
> application name. User can also define their own placement rule.
> {quote}
> As per current UserGroupMapping is always added in placementRule.
> {{CapacityScheduler#updatePlacementRules}}
> {code}
> // Initialize placement rules
> Collection placementRuleStrs = conf.getStringCollection(
> YarnConfiguration.QUEUE_PLACEMENT_RULES);
> List placementRules = new ArrayList<>();
> ...
> // add UserGroupMappingPlacementRule if absent
> distingushRuleSet.add(YarnConfiguration.USER_GROUP_PLACEMENT_RULE);
> {code}
> PlacementRule configuration order is not maintained
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673027#comment-16673027
]
Wilfred Spiegelenburg commented on YARN-8948:
-
[~bibinchundatt] Thank you for picking this up. I had just started looking at
this to move the FairScheduler over to the same interface. I have opened a new
jira for that already and will start with that as soon as this jira is finished.
> PlacementRule interface should be for all YarnSchedulers
>
>
> Key: YARN-8948
> URL: https://issues.apache.org/jira/browse/YARN-8948
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Attachments: YARN-8948.001.patch, YARN-8948.002.patch,
> YARN-8948.003.patch
>
>
> *Issue 1:*
> YARN-3635 intention was to add PlacementRule interface common for all
> YarnSchedules.
> {code}
> 33 public abstract boolean initialize(
> 34 CapacitySchedulerContext schedulerContext) throws IOException;
> {code}
> PlacementRule initialization is done using CapacitySchedulerContext binding
> to CapacityScheduler
> *Issue 2:*
> {{yarn.scheduler.queue-placement-rules}} doesn't work as expected in Capacity
> Scheduler
> {quote}
> * **Queue Mapping Interface based on Default or User Defined Placement
> Rules** - This feature allows users to map a job to a specific queue based on
> some default placement rule. For instance based on user & group, or
> application name. User can also define their own placement rule.
> {quote}
> As per current UserGroupMapping is always added in placementRule.
> {{CapacityScheduler#updatePlacementRules}}
> {code}
> // Initialize placement rules
> Collection placementRuleStrs = conf.getStringCollection(
> YarnConfiguration.QUEUE_PLACEMENT_RULES);
> List placementRules = new ArrayList<>();
> ...
> // add UserGroupMappingPlacementRule if absent
> distingushRuleSet.add(YarnConfiguration.USER_GROUP_PLACEMENT_RULE);
> {code}
> PlacementRule configuration order is not maintained
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672961#comment-16672961
]
Sunil Govindan commented on YARN-8948:
--
On this note, this is not an interface level compatibility problem.
Ideally we should have done for all schedulers, however this is not affecting
the interface, and hence i suggest to mark this as Major and get this in all
respective versions and next release can take this.
If you feel this is really a blocker to 3.2, kindly let me know.
> PlacementRule interface should be for all YarnSchedulers
>
>
> Key: YARN-8948
> URL: https://issues.apache.org/jira/browse/YARN-8948
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Attachments: YARN-8948.001.patch, YARN-8948.002.patch,
> YARN-8948.003.patch
>
>
> *Issue 1:*
> YARN-3635 intention was to add PlacementRule interface common for all
> YarnSchedules.
> {code}
> 33 public abstract boolean initialize(
> 34 CapacitySchedulerContext schedulerContext) throws IOException;
> {code}
> PlacementRule initialization is done using CapacitySchedulerContext binding
> to CapacityScheduler
> *Issue 2:*
> {{yarn.scheduler.queue-placement-rules}} doesn't work as expected in Capacity
> Scheduler
> {quote}
> * **Queue Mapping Interface based on Default or User Defined Placement
> Rules** - This feature allows users to map a job to a specific queue based on
> some default placement rule. For instance based on user & group, or
> application name. User can also define their own placement rule.
> {quote}
> As per current UserGroupMapping is always added in placementRule.
> {{CapacityScheduler#updatePlacementRules}}
> {code}
> // Initialize placement rules
> Collection placementRuleStrs = conf.getStringCollection(
> YarnConfiguration.QUEUE_PLACEMENT_RULES);
> List placementRules = new ArrayList<>();
> ...
> // add UserGroupMappingPlacementRule if absent
> distingushRuleSet.add(YarnConfiguration.USER_GROUP_PLACEMENT_RULE);
> {code}
> PlacementRule configuration order is not maintained
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672922#comment-16672922
]
Bibin A Chundatt commented on YARN-8948:
[~sunilg]
{code}
cud u pls explain how this is breaking wire compatibility
{code}
Not breaking wire compatability since nothing is transmitted between process.
In jira i mentioned the *interface is supposed be for all schedulers.* shouldnt
be for capacityscheduler alone. Else next version might have to change again??
{code}
If this is not breaking any compatibility, i would like to degrade severity to
Minor so that 3.2 release can go on.
{code}
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Compatibility.html
Since we havnt marked {{PlacementRule}} as public interface level compatability
might not be requried.
YARN-8016 made the placementRule refined and made it usable.
Another issue seems currently UserGroup*Rule is added always and the order is
not maintained for {{distingushRuleSet}} custom configuration will not work as
expected. Consider as functional issue.
Lets take a call based on response from [~leftnoteasy] and [~suma.shivaprasad]
too.
> PlacementRule interface should be for all YarnSchedulers
>
>
> Key: YARN-8948
> URL: https://issues.apache.org/jira/browse/YARN-8948
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Attachments: YARN-8948.001.patch, YARN-8948.002.patch,
> YARN-8948.003.patch
>
>
> *Issue 1:*
> YARN-3635 intention was to add PlacementRule interface common for all
> YarnSchedules.
> {code}
> 33 public abstract boolean initialize(
> 34 CapacitySchedulerContext schedulerContext) throws IOException;
> {code}
> PlacementRule initialization is done using CapacitySchedulerContext binding
> to CapacityScheduler
> *Issue 2:*
> {{yarn.scheduler.queue-placement-rules}} doesn't work as expected in Capacity
> Scheduler
> {quote}
> * **Queue Mapping Interface based on Default or User Defined Placement
> Rules** - This feature allows users to map a job to a specific queue based on
> some default placement rule. For instance based on user & group, or
> application name. User can also define their own placement rule.
> {quote}
> As per current UserGroupMapping is always added in placementRule.
> {{CapacityScheduler#updatePlacementRules}}
> {code}
> // Initialize placement rules
> Collection placementRuleStrs = conf.getStringCollection(
> YarnConfiguration.QUEUE_PLACEMENT_RULES);
> List placementRules = new ArrayList<>();
> ...
> // add UserGroupMappingPlacementRule if absent
> distingushRuleSet.add(YarnConfiguration.USER_GROUP_PLACEMENT_RULE);
> {code}
> PlacementRule configuration order is not maintained
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672883#comment-16672883
]
Sunil Govindan commented on YARN-8948:
--
[~bibinchundatt]
Few questions:
# cud u pls explain how this is breaking wire compatibility b/w old hadoop
version to new one?
# These jiras are already in hadoop-3.1 and other released? Hence is this
already broken by these version?
# any workarounds if u upgrade to hadoop-3.2 and then use old clients/apps by
which we ll not have any breakage?
If this is not breaking any compatibility, i would like to degrade severity to
Minor so that 3.2 release can go on.
cc [~leftnoteasy] [~suma.shivaprasad]
> PlacementRule interface should be for all YarnSchedulers
>
>
> Key: YARN-8948
> URL: https://issues.apache.org/jira/browse/YARN-8948
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Attachments: YARN-8948.001.patch, YARN-8948.002.patch,
> YARN-8948.003.patch
>
>
> *Issue 1:*
> YARN-3635 intention was to add PlacementRule interface common for all
> YarnSchedules.
> {code}
> 33 public abstract boolean initialize(
> 34 CapacitySchedulerContext schedulerContext) throws IOException;
> {code}
> PlacementRule initialization is done using CapacitySchedulerContext binding
> to CapacityScheduler
> *Issue 2:*
> {{yarn.scheduler.queue-placement-rules}} doesn't work as expected in Capacity
> Scheduler
> {quote}
> * **Queue Mapping Interface based on Default or User Defined Placement
> Rules** - This feature allows users to map a job to a specific queue based on
> some default placement rule. For instance based on user & group, or
> application name. User can also define their own placement rule.
> {quote}
> As per current UserGroupMapping is always added in placementRule.
> {{CapacityScheduler#updatePlacementRules}}
> {code}
> // Initialize placement rules
> Collection placementRuleStrs = conf.getStringCollection(
> YarnConfiguration.QUEUE_PLACEMENT_RULES);
> List placementRules = new ArrayList<>();
> ...
> // add UserGroupMappingPlacementRule if absent
> distingushRuleSet.add(YarnConfiguration.USER_GROUP_PLACEMENT_RULE);
> {code}
> PlacementRule configuration order is not maintained
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672686#comment-16672686
]
Bibin A Chundatt commented on YARN-8948:
[~sunil.gov...@gmail.com]
Could you look into this issue..
> PlacementRule interface should be for all YarnSchedulers
>
>
> Key: YARN-8948
> URL: https://issues.apache.org/jira/browse/YARN-8948
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Critical
> Attachments: YARN-8948.001.patch, YARN-8948.002.patch
>
>
> *Issue 1:*
> YARN-3635 intention was to add PlacementRule interface common for all
> YarnSchedules.
> {code}
> 33 public abstract boolean initialize(
> 34 CapacitySchedulerContext schedulerContext) throws IOException;
> {code}
> PlacementRule initialization is done using CapacitySchedulerContext binding
> to CapacityScheduler
> *Issue 2:*
> {{yarn.scheduler.queue-placement-rules}} doesn't work as expected in Capacity
> Scheduler
> {quote}
> * **Queue Mapping Interface based on Default or User Defined Placement
> Rules** - This feature allows users to map a job to a specific queue based on
> some default placement rule. For instance based on user & group, or
> application name. User can also define their own placement rule.
> {quote}
> As per current UserGroupMapping is always added in placementRule.
> {{CapacityScheduler#updatePlacementRules}}
> {code}
> // Initialize placement rules
> Collection placementRuleStrs = conf.getStringCollection(
> YarnConfiguration.QUEUE_PLACEMENT_RULES);
> List placementRules = new ArrayList<>();
> ...
> // add UserGroupMappingPlacementRule if absent
> distingushRuleSet.add(YarnConfiguration.USER_GROUP_PLACEMENT_RULE);
> {code}
> PlacementRule configuration order is not maintained
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667339#comment-16667339
]
Hadoop QA commented on YARN-8948:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 869 unchanged - 1 fixed = 869 total (was 870)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 27s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}102m 5s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueManagementDynamicEditPolicy
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8948 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946035/YARN-8948.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 242eb21a3a0e 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 78ea897 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/22367/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22367/testReport/ |
| Max. process+thread count | 959
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1235#comment-1235
]
Hadoop QA commented on YARN-8948:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 105 unchanged - 1 fixed = 105 total (was 106)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}106m 0s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}165m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueState |
| | hadoop.yarn.server.resourcemanager.webapp.TestNodesPage |
| | hadoop.yarn.server.resourcemanager.resourcetracker.TestNMReconnect |
| | hadoop.yarn.server.resourcemanager.reservation.TestReservationSystem |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueStateManager |
| | hadoop.yarn.server.resourcemanager.webapp.TestAppPage |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebApp |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
| | hadoop.yarn.server.resourcemanager.webapp.TestRedirectionErrorPage |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueMappings |
\\
\\
|| Subsystem ||
[jira] [Commented] (YARN-8948) PlacementRule interface should be for all YarnSchedulers
[
https://issues.apache.org/jira/browse/YARN-8948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1173#comment-1173
]
Bibin A Chundatt commented on YARN-8948:
cc: [~sunil.gov...@gmail.com]/[~wangda]
> PlacementRule interface should be for all YarnSchedulers
>
>
> Key: YARN-8948
> URL: https://issues.apache.org/jira/browse/YARN-8948
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-8948.001.patch
>
>
> YARN-3635 intention was to add PlacementRule interface common for all
> YarnSchedules.
> {code}
> 33 public abstract boolean initialize(
> 34 CapacitySchedulerContext schedulerContext) throws IOException;
> {code}
> PlacementRule initialization is done using CapacitySchedulerContext binding
> to CapacityScheduler
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
