[jira] [Commented] (YARN-7443) Add native FPGA module support to do isolation with cgroups
[
https://issues.apache.org/jira/browse/YARN-7443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251041#comment-16251041
]
Hadoop QA commented on YARN-7443:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
46s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
27m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 23s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 24m 52s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 66m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.TestLocalDirsHandlerService |
| | hadoop.yarn.server.nodemanager.TestLinuxContainerExecutorWithMocks |
| | hadoop.yarn.server.nodemanager.TestNodeManagerReboot |
| |
hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor |
| | hadoop.yarn.server.nodemanager.TestNodeHealthService |
| |
hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService
|
| |
hadoop.yarn.server.nodemanager.containermanager.scheduler.TestContainerSchedulerQueuing
|
| | hadoop.yarn.server.nodemanager.webapp.TestNMWebServices |
| | hadoop.yarn.server.nodemanager.containermanager.TestContainerManager |
| |
hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService
|
| | hadoop.yarn.server.nodemanager.TestNodeManagerResync |
| | hadoop.yarn.server.nodemanager.TestNodeManagerShutdown |
| |
hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch |
| Timed out junit tests |
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestContainerManagerRecovery
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7443 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12897465/YARN-7443-trunk.002.patch
|
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux fd9f4a6307c4 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 4f40cd3 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/18479/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test
[jira] [Updated] (YARN-7464) Allow fiters on Nodes page
[ https://issues.apache.org/jira/browse/YARN-7464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vasudevan Skm updated YARN-7464: Attachment: YARN-7464.002.patch [~sunil.gov...@gmail.com]DOne the requested copy change > Allow fiters on Nodes page > -- > > Key: YARN-7464 > URL: https://issues.apache.org/jira/browse/YARN-7464 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm > Attachments: Screen Shot 2017-11-08 at 4.56.04 PM.png, Screen Shot > 2017-11-08 at 4.56.12 PM.png, YARN-7464.001.patch, YARN-7464.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7443) Add native FPGA module support to do isolation with cgroups
[
https://issues.apache.org/jira/browse/YARN-7443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251034#comment-16251034
]
Hadoop QA commented on YARN-7443:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
25m 48s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 17m
16s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7443 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12897465/YARN-7443-trunk.002.patch
|
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux c1db06cac760 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 4f40cd3 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/18478/testReport/ |
| Max. process+thread count | 403 (vs. ulimit of 5000) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/18478/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add native FPGA module support to do isolation with cgroups
> ---
>
> Key: YARN-7443
> URL: https://issues.apache.org/jira/browse/YARN-7443
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
> Attachments: YARN-7443-trunk.001.patch, YARN-7443-trunk.002.patch
>
>
> Only support one major number devices configured in c-e.cfg for now. So
> almost same with GPU native module
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For
[jira] [Commented] (YARN-6918) Remove acls after queue delete to avoid memory leak
[ https://issues.apache.org/jira/browse/YARN-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251021#comment-16251021 ] Bibin A Chundatt commented on YARN-6918: Thank you [~subru] and [~sunilg] for looking into issue..Lets get this in for 2.9.1 and 3.0.0 then. > Remove acls after queue delete to avoid memory leak > --- > > Key: YARN-6918 > URL: https://issues.apache.org/jira/browse/YARN-6918 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: YARN-6918.001.patch > > > Acl for deleted queue need to removed from allAcls to avoid leak > (Priority,YarnAuthorizer) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7462) Render outstanding resource requests on application details page
[
https://issues.apache.org/jira/browse/YARN-7462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250997#comment-16250997
]
Hadoop QA commented on YARN-7462:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
27m 44s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 20s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7462 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12897454/YARN-7462.003.patch |
| Optional Tests | asflicense shadedclient |
| uname | Linux 447200171ee5 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 4f40cd3 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 297 (vs. ulimit of 5000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/18477/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Render outstanding resource requests on application details page
>
>
> Key: YARN-7462
> URL: https://issues.apache.org/jira/browse/YARN-7462
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-ui-v2
>Reporter: Vasudevan Skm
>Assignee: Vasudevan Skm
> Attachments: Screen Shot 2017-11-08 at 3.24.30 PM.png, Screen Shot
> 2017-11-08 at 3.38.48 PM.png, YARN-7462.001.patch, YARN-7462.002.patch,
> YARN-7462.003.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7464) Allow fiters on Nodes page
[ https://issues.apache.org/jira/browse/YARN-7464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250995#comment-16250995 ] Sunil G commented on YARN-7464: --- Please rename "Node Managers" to "Nodes Status" > Allow fiters on Nodes page > -- > > Key: YARN-7464 > URL: https://issues.apache.org/jira/browse/YARN-7464 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm > Attachments: Screen Shot 2017-11-08 at 4.56.04 PM.png, Screen Shot > 2017-11-08 at 4.56.12 PM.png, YARN-7464.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7336) Unsafe cast from long to int Resource.hashCode() method
[
https://issues.apache.org/jira/browse/YARN-7336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-7336:
---
Attachment: YARN-7336.branch-3.0.001.patch
Adding branch-3.0 patch from resource-types branch prior to merge.
> Unsafe cast from long to int Resource.hashCode() method
> ---
>
> Key: YARN-7336
> URL: https://issues.apache.org/jira/browse/YARN-7336
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.0.0-beta1, 3.1.0
>Reporter: Daniel Templeton
>Assignee: Miklos Szegedi
>Priority: Critical
> Labels: ready-to-commit
> Fix For: 3.0.0, 3.1.0
>
> Attachments: YARN-7336.000.patch, YARN-7336.001.patch,
> YARN-7336.branch-3.0.001.patch
>
>
> For example:
> {code}
> final int prime = 47;
> long result = 0;
> for (ResourceInformation entry : resources) {
> result = prime * result + entry.hashCode();
> }
> return (int) result;
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7336) Unsafe cast from long to int Resource.hashCode() method
[
https://issues.apache.org/jira/browse/YARN-7336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-7336:
---
Fix Version/s: 3.0.0
> Unsafe cast from long to int Resource.hashCode() method
> ---
>
> Key: YARN-7336
> URL: https://issues.apache.org/jira/browse/YARN-7336
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.0.0-beta1, 3.1.0
>Reporter: Daniel Templeton
>Assignee: Miklos Szegedi
>Priority: Critical
> Labels: ready-to-commit
> Fix For: 3.0.0, 3.1.0
>
> Attachments: YARN-7336.000.patch, YARN-7336.001.patch,
> YARN-7336.branch-3.0.001.patch
>
>
> For example:
> {code}
> final int prime = 47;
> long result = 0;
> for (ResourceInformation entry : resources) {
> result = prime * result + entry.hashCode();
> }
> return (int) result;
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6927) Add support for individual resource types requests in MapReduce
[ https://issues.apache.org/jira/browse/YARN-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6927: --- Fix Version/s: 3.0.0 > Add support for individual resource types requests in MapReduce > --- > > Key: YARN-6927 > URL: https://issues.apache.org/jira/browse/YARN-6927 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Daniel Templeton >Assignee: Gergo Repas > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-6927.000.patch, YARN-6927.001.patch, > YARN-6927.002.patch, YARN-6927.003.patch, YARN-6927.004.patch, > YARN-6927.005.patch, YARN-6927.006.patch, YARN-6927.007.patch, > YARN-6927.008.patch, YARN-6927.009.patch, YARN-6927.010.patch, > YARN-6927.branch-3.0.001.patch > > > YARN-6504 adds support for resource profiles in MapReduce jobs, but resource > profiles don't give users much flexibility in their resource requests. To > satisfy users' needs, MapReduce should also allow users to specify arbitrary > resource requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6927) Add support for individual resource types requests in MapReduce
[ https://issues.apache.org/jira/browse/YARN-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6927: --- Attachment: YARN-6927.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > Add support for individual resource types requests in MapReduce > --- > > Key: YARN-6927 > URL: https://issues.apache.org/jira/browse/YARN-6927 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Daniel Templeton >Assignee: Gergo Repas > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-6927.000.patch, YARN-6927.001.patch, > YARN-6927.002.patch, YARN-6927.003.patch, YARN-6927.004.patch, > YARN-6927.005.patch, YARN-6927.006.patch, YARN-6927.007.patch, > YARN-6927.008.patch, YARN-6927.009.patch, YARN-6927.010.patch, > YARN-6927.branch-3.0.001.patch > > > YARN-6504 adds support for resource profiles in MapReduce jobs, but resource > profiles don't give users much flexibility in their resource requests. To > satisfy users' needs, MapReduce should also allow users to specify arbitrary > resource requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7374) Improve performance of DRF comparisons for resource types in fair scheduler
[ https://issues.apache.org/jira/browse/YARN-7374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-7374: --- Attachment: YARN-7374.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > Improve performance of DRF comparisons for resource types in fair scheduler > --- > > Key: YARN-7374 > URL: https://issues.apache.org/jira/browse/YARN-7374 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-7374.001.patch, YARN-7374.002.patch, > YARN-7374.003.patch, YARN-7374.004.patch, YARN-7374.005.patch, > YARN-7374.branch-3.0.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7374) Improve performance of DRF comparisons for resource types in fair scheduler
[ https://issues.apache.org/jira/browse/YARN-7374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-7374: --- Fix Version/s: 3.0.0 > Improve performance of DRF comparisons for resource types in fair scheduler > --- > > Key: YARN-7374 > URL: https://issues.apache.org/jira/browse/YARN-7374 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-7374.001.patch, YARN-7374.002.patch, > YARN-7374.003.patch, YARN-7374.004.patch, YARN-7374.005.patch, > YARN-7374.branch-3.0.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7397) Reduce lock contention in FairScheduler#getAppWeight()
[
https://issues.apache.org/jira/browse/YARN-7397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-7397:
---
Fix Version/s: 3.0.0
> Reduce lock contention in FairScheduler#getAppWeight()
> --
>
> Key: YARN-7397
> URL: https://issues.apache.org/jira/browse/YARN-7397
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Fix For: 3.0.0, 3.1.0
>
> Attachments: YARN-7397.001.patch, YARN-7397.branch-3.0.001.patch
>
>
> In profiling the fair scheduler, a large amount of time is spent waiting to
> get the lock in {{FairScheduler.getAppWeight()}}, when the lock isn't
> actually needed. This patch reduces the scope of the lock to eliminate that
> contention.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7397) Reduce lock contention in FairScheduler#getAppWeight()
[
https://issues.apache.org/jira/browse/YARN-7397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-7397:
---
Attachment: YARN-7397.branch-3.0.001.patch
Adding branch-3.0 patch from resource-types branch prior to merge.
> Reduce lock contention in FairScheduler#getAppWeight()
> --
>
> Key: YARN-7397
> URL: https://issues.apache.org/jira/browse/YARN-7397
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Fix For: 3.0.0, 3.1.0
>
> Attachments: YARN-7397.001.patch, YARN-7397.branch-3.0.001.patch
>
>
> In profiling the fair scheduler, a large amount of time is spent waiting to
> get the lock in {{FairScheduler.getAppWeight()}}, when the lock isn't
> actually needed. This patch reduces the scope of the lock to eliminate that
> contention.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6984) DominantResourceCalculator.isAnyMajorResourceZero() should test all resources
[
https://issues.apache.org/jira/browse/YARN-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-6984:
---
Fix Version/s: 3.0.0
> DominantResourceCalculator.isAnyMajorResourceZero() should test all resources
> -
>
> Key: YARN-6984
> URL: https://issues.apache.org/jira/browse/YARN-6984
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Sunil G
> Fix For: 3.0.0, 3.1.0
>
> Attachments: YARN-6984.001.patch, YARN-6984.002.patch,
> YARN-6984.branch-3.0.001.patch
>
>
> The method currently tests only memory and CPU. It looks to me like it
> should test all resources, i.e. it should do what {{isInvalidDivisor()}} does
> and should, in fact, replace that method. [~sunilg], since you wrote the
> method originally, can you comment on what its intended semantics are?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7270) Fix unsafe casting from long to int for class Resource and its sub-classes
[
https://issues.apache.org/jira/browse/YARN-7270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-7270:
---
Attachment: YARN-7270.branch-3.0.addendum.001.patch
> Fix unsafe casting from long to int for class Resource and its sub-classes
> --
>
> Key: YARN-7270
> URL: https://issues.apache.org/jira/browse/YARN-7270
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 2.9.0, 3.0.0, 3.1.0
>
> Attachments: YARN-7270.001.patch, YARN-7270.002.patch,
> YARN-7270.003.patch, YARN-7270.004.patch, YARN-7270.005.patch,
> YARN-7270.branch-3.0.addendum.001.patch, YARN-7270.branch-3.0.patch
>
>
> Class {{Resource}} has three sub classes(FixedValueResource,
> LightWeightResource, and ResourcePBImpl). Only FixedValueResource handle
> long-to-int casting nicely. The other two didn't. This bug is introduced by
> resource type feature and causes several unit test failures. For example:
> {code}
> Error Message
> expected:<> but was:<>
> Stacktrace
> java.lang.AssertionError: expected:<> but
> was:<>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppAttempt.testHeadroomWithBlackListedNodes(TestFSAppAttempt.java:325)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6984) DominantResourceCalculator.isAnyMajorResourceZero() should test all resources
[
https://issues.apache.org/jira/browse/YARN-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-6984:
---
Attachment: YARN-6984.branch-3.0.001.patch
Adding branch-3.0 patch from resource-types branch prior to merge.
> DominantResourceCalculator.isAnyMajorResourceZero() should test all resources
> -
>
> Key: YARN-6984
> URL: https://issues.apache.org/jira/browse/YARN-6984
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Sunil G
> Fix For: 3.0.0, 3.1.0
>
> Attachments: YARN-6984.001.patch, YARN-6984.002.patch,
> YARN-6984.branch-3.0.001.patch
>
>
> The method currently tests only memory and CPU. It looks to me like it
> should test all resources, i.e. it should do what {{isInvalidDivisor()}} does
> and should, in fact, replace that method. [~sunilg], since you wrote the
> method originally, can you comment on what its intended semantics are?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7172) ResourceCalculator.fitsIn() should not take a cluster resource parameter
[
https://issues.apache.org/jira/browse/YARN-7172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-7172:
---
Attachment: YARN-7172.branch-3.0.001.patch
Adding branch-3.0 patch from resource-types branch prior to merge.
> ResourceCalculator.fitsIn() should not take a cluster resource parameter
>
>
> Key: YARN-7172
> URL: https://issues.apache.org/jira/browse/YARN-7172
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Sen Zhao
> Labels: newbie
> Fix For: 3.0.0, 3.1.0
>
> Attachments: YARN-7172.001.patch, YARN-7172.002.patch,
> YARN-7172.003.patch, YARN-7172.004.patch, YARN-7172.005.patch,
> YARN-7172.006.patch, YARN-7172.branch-3.0.001.patch
>
>
> There are numerous calls to {{ClusterNodeTracker.getClusterResource()}}
> (which involves a lock) to get a value to pass as the cluster resource
> parameter to {{Resources.fitsIn()}}, but the parameter is (quite reasonably)
> ignored. We should remove the parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7172) ResourceCalculator.fitsIn() should not take a cluster resource parameter
[
https://issues.apache.org/jira/browse/YARN-7172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-7172:
---
Fix Version/s: 3.0.0
> ResourceCalculator.fitsIn() should not take a cluster resource parameter
>
>
> Key: YARN-7172
> URL: https://issues.apache.org/jira/browse/YARN-7172
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Sen Zhao
> Labels: newbie
> Fix For: 3.0.0, 3.1.0
>
> Attachments: YARN-7172.001.patch, YARN-7172.002.patch,
> YARN-7172.003.patch, YARN-7172.004.patch, YARN-7172.005.patch,
> YARN-7172.006.patch, YARN-7172.branch-3.0.001.patch
>
>
> There are numerous calls to {{ClusterNodeTracker.getClusterResource()}}
> (which involves a lock) to get a value to pass as the cluster resource
> parameter to {{Resources.fitsIn()}}, but the parameter is (quite reasonably)
> ignored. We should remove the parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6612) Update fair scheduler policies to be aware of resource types
[ https://issues.apache.org/jira/browse/YARN-6612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6612: --- Fix Version/s: 3.0.0 > Update fair scheduler policies to be aware of resource types > > > Key: YARN-6612 > URL: https://issues.apache.org/jira/browse/YARN-6612 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-6612.015.patch, YARN-6612.016.patch, > YARN-6612.YARN-3926.001.patch, YARN-6612.YARN-3926.002.patch, > YARN-6612.YARN-3926.004.patch, YARN-6612.YARN-3926.005.patch, > YARN-6612.YARN-3926.006.patch, YARN-6612.YARN-3926.007.patch, > YARN-6612.YARN-3926.008.patch, YARN-6612.YARN-3926.009.patch, > YARN-6612.YARN-3926.010.patch, YARN-6612.YARN-3926.011.patch, > YARN-6612.YARN-3926.012.patch, YARN-6612.YARN-3926.013.patch, > YARN-6612.YARN-3926.014.patch, YARN-6612.branch-3.0.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7137) Move newly added APIs to unstable in YARN-3926 branch
[ https://issues.apache.org/jira/browse/YARN-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-7137: --- Fix Version/s: 3.0.0 > Move newly added APIs to unstable in YARN-3926 branch > - > > Key: YARN-7137 > URL: https://issues.apache.org/jira/browse/YARN-7137 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-7137.YARN-3926.001.patch, > YARN-7137.YARN-3926.002.patch, YARN-7137.branch-3.0.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6612) Update fair scheduler policies to be aware of resource types
[ https://issues.apache.org/jira/browse/YARN-6612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6612: --- Attachment: YARN-6612.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > Update fair scheduler policies to be aware of resource types > > > Key: YARN-6612 > URL: https://issues.apache.org/jira/browse/YARN-6612 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-6612.015.patch, YARN-6612.016.patch, > YARN-6612.YARN-3926.001.patch, YARN-6612.YARN-3926.002.patch, > YARN-6612.YARN-3926.004.patch, YARN-6612.YARN-3926.005.patch, > YARN-6612.YARN-3926.006.patch, YARN-6612.YARN-3926.007.patch, > YARN-6612.YARN-3926.008.patch, YARN-6612.YARN-3926.009.patch, > YARN-6612.YARN-3926.010.patch, YARN-6612.YARN-3926.011.patch, > YARN-6612.YARN-3926.012.patch, YARN-6612.YARN-3926.013.patch, > YARN-6612.YARN-3926.014.patch, YARN-6612.branch-3.0.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7137) Move newly added APIs to unstable in YARN-3926 branch
[ https://issues.apache.org/jira/browse/YARN-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-7137: --- Attachment: YARN-7137.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > Move newly added APIs to unstable in YARN-3926 branch > - > > Key: YARN-7137 > URL: https://issues.apache.org/jira/browse/YARN-7137 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Fix For: 3.1.0 > > Attachments: YARN-7137.YARN-3926.001.patch, > YARN-7137.YARN-3926.002.patch, YARN-7137.branch-3.0.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7136) Additional Performance Improvement for Resource Profile Feature
[ https://issues.apache.org/jira/browse/YARN-7136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-7136: --- Fix Version/s: 3.0.0 > Additional Performance Improvement for Resource Profile Feature > --- > > Key: YARN-7136 > URL: https://issues.apache.org/jira/browse/YARN-7136 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-7136.001.patch, YARN-7136.YARN-3926.001.patch, > YARN-7136.YARN-3926.002.patch, YARN-7136.YARN-3926.003.patch, > YARN-7136.YARN-3926.004.patch, YARN-7136.YARN-3926.005.patch, > YARN-7136.YARN-3926.006.patch, YARN-7136.YARN-3926.007.patch, > YARN-7136.YARN-3926.008.patch, YARN-7136.YARN-3926.009.patch, > YARN-7136.YARN-3926.010.patch, YARN-7136.YARN-3926.011.patch, > YARN-7136.YARN-3926.012.patch, YARN-7136.YARN-3926.013.patch, > YARN-7136.YARN-3926.014.patch, YARN-7136.YARN-3926.015.patch, > YARN-7136.YARN-3926.016.patch, YARN-7136.branch-3.0.001.patch > > > This JIRA is plan to add following misc perf improvements: > 1) Use final int in Resources/ResourceCalculator to cache > #known-resource-types. (Significant improvement). > 2) Catch Java's ArrayOutOfBound Exception instead of checking array.length > every time. (Significant improvement). > 3) Avoid setUnit validation (which is a HashSet lookup) when initialize > default Memory/VCores ResourceInformation (Significant improvement). > 4) Avoid unnecessary loop array in Resource#toString/hashCode. (Some > improvement). > 5) Removed readOnlyResources in BaseResource. (Minor improvement). > 6) Removed enum: MandatoryResources, use final integer instead. (Minor > improvement). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7136) Additional Performance Improvement for Resource Profile Feature
[ https://issues.apache.org/jira/browse/YARN-7136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-7136: --- Attachment: YARN-7136.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > Additional Performance Improvement for Resource Profile Feature > --- > > Key: YARN-7136 > URL: https://issues.apache.org/jira/browse/YARN-7136 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-7136.001.patch, YARN-7136.YARN-3926.001.patch, > YARN-7136.YARN-3926.002.patch, YARN-7136.YARN-3926.003.patch, > YARN-7136.YARN-3926.004.patch, YARN-7136.YARN-3926.005.patch, > YARN-7136.YARN-3926.006.patch, YARN-7136.YARN-3926.007.patch, > YARN-7136.YARN-3926.008.patch, YARN-7136.YARN-3926.009.patch, > YARN-7136.YARN-3926.010.patch, YARN-7136.YARN-3926.011.patch, > YARN-7136.YARN-3926.012.patch, YARN-7136.YARN-3926.013.patch, > YARN-7136.YARN-3926.014.patch, YARN-7136.YARN-3926.015.patch, > YARN-7136.YARN-3926.016.patch, YARN-7136.branch-3.0.001.patch > > > This JIRA is plan to add following misc perf improvements: > 1) Use final int in Resources/ResourceCalculator to cache > #known-resource-types. (Significant improvement). > 2) Catch Java's ArrayOutOfBound Exception instead of checking array.length > every time. (Significant improvement). > 3) Avoid setUnit validation (which is a HashSet lookup) when initialize > default Memory/VCores ResourceInformation (Significant improvement). > 4) Avoid unnecessary loop array in Resource#toString/hashCode. (Some > improvement). > 5) Removed readOnlyResources in BaseResource. (Minor improvement). > 6) Removed enum: MandatoryResources, use final integer instead. (Minor > improvement). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7056) Document Resource Profiles feature
[ https://issues.apache.org/jira/browse/YARN-7056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-7056: --- Attachment: YARN-7056.branch-3.0.001.patch > Document Resource Profiles feature > -- > > Key: YARN-7056 > URL: https://issues.apache.org/jira/browse/YARN-7056 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Fix For: 3.1.0 > > Attachments: YARN-7056.YARN-3926.001.patch, > YARN-7056.YARN-3926.002.patch, YARN-7056.branch-3.0.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6933) ResourceUtils.DISALLOWED_NAMES check is duplicated
[
https://issues.apache.org/jira/browse/YARN-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-6933:
---
Attachment: YARN-6933.branch-3.0.001.patch
Adding branch-3.0 patch from resource-types branch prior to merge.
> ResourceUtils.DISALLOWED_NAMES check is duplicated
> --
>
> Key: YARN-6933
> URL: https://issues.apache.org/jira/browse/YARN-6933
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Manikandan R
> Labels: newbie++
> Fix For: 3.0.0, 3.1.0
>
> Attachments: YARN-6933-YARN-3926.001.patch,
> YARN-6933-YARN-3926.002.patch, YARN-6933-YARN-3926.003.patch,
> YARN-6933-YARN-3926.004.patch, YARN-6933-YARN-3926.005.patch,
> YARN-6933-YARN-3926.006.patch, YARN-6933-YARN-3926.007.patch,
> YARN-6933-YARN-3926.008.patch, YARN-6933-YARN-3926.009.patch,
> YARN-6933.branch-3.0.001.patch
>
>
> Both are used to check that the mandatory resources were not redefined. Only
> one check is needed. I would recommend dropping {{DISALLOWED_RESOURCES}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6933) ResourceUtils.DISALLOWED_NAMES check is duplicated
[
https://issues.apache.org/jira/browse/YARN-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-6933:
---
Fix Version/s: 3.0.0
> ResourceUtils.DISALLOWED_NAMES check is duplicated
> --
>
> Key: YARN-6933
> URL: https://issues.apache.org/jira/browse/YARN-6933
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Manikandan R
> Labels: newbie++
> Fix For: 3.0.0, 3.1.0
>
> Attachments: YARN-6933-YARN-3926.001.patch,
> YARN-6933-YARN-3926.002.patch, YARN-6933-YARN-3926.003.patch,
> YARN-6933-YARN-3926.004.patch, YARN-6933-YARN-3926.005.patch,
> YARN-6933-YARN-3926.006.patch, YARN-6933-YARN-3926.007.patch,
> YARN-6933-YARN-3926.008.patch, YARN-6933-YARN-3926.009.patch,
> YARN-6933.branch-3.0.001.patch
>
>
> Both are used to check that the mandatory resources were not redefined. Only
> one check is needed. I would recommend dropping {{DISALLOWED_RESOURCES}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7093) Improve log message in ResourceUtils
[ https://issues.apache.org/jira/browse/YARN-7093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-7093: --- Attachment: YARN-7093.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > Improve log message in ResourceUtils > > > Key: YARN-7093 > URL: https://issues.apache.org/jira/browse/YARN-7093 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Sunil G >Assignee: Sunil G >Priority: Trivial > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-7093.YARN-3926.001.patch, > YARN-7093.branch-3.0.001.patch > > > Improve log message ResourceUtils class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7093) Improve log message in ResourceUtils
[ https://issues.apache.org/jira/browse/YARN-7093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-7093: --- Fix Version/s: 3.0.0 > Improve log message in ResourceUtils > > > Key: YARN-7093 > URL: https://issues.apache.org/jira/browse/YARN-7093 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Sunil G >Assignee: Sunil G >Priority: Trivial > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-7093.YARN-3926.001.patch, > YARN-7093.branch-3.0.001.patch > > > Improve log message ResourceUtils class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7039) Fix javac and javadoc errors in YARN-3926 branch
[ https://issues.apache.org/jira/browse/YARN-7039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-7039: --- Attachment: YARN-7039.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > Fix javac and javadoc errors in YARN-3926 branch > > > Key: YARN-7039 > URL: https://issues.apache.org/jira/browse/YARN-7039 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: YARN-3926 >Reporter: Sunil G >Assignee: Sunil G > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-7039.YARN-3926.001.patch, > YARN-7039.YARN-3926.002.patch, YARN-7039.branch-3.0.001.patch > > > Fix javac and doc errors. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7039) Fix javac and javadoc errors in YARN-3926 branch
[ https://issues.apache.org/jira/browse/YARN-7039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-7039: --- Fix Version/s: 3.0.0 > Fix javac and javadoc errors in YARN-3926 branch > > > Key: YARN-7039 > URL: https://issues.apache.org/jira/browse/YARN-7039 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: YARN-3926 >Reporter: Sunil G >Assignee: Sunil G > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-7039.YARN-3926.001.patch, > YARN-7039.YARN-3926.002.patch, YARN-7039.branch-3.0.001.patch > > > Fix javac and doc errors. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7067) Optimize ResourceType information display in UI
[
https://issues.apache.org/jira/browse/YARN-7067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-7067:
---
Attachment: YARN-7067.branch-3.0.001.patch
Adding branch-3.0 patch from resource-types branch prior to merge.
> Optimize ResourceType information display in UI
> ---
>
> Key: YARN-7067
> URL: https://issues.apache.org/jira/browse/YARN-7067
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Critical
> Fix For: 3.0.0, 3.1.0
>
> Attachments: YARN-7067.001.patch, YARN-7067.YARN-3926.001.patch,
> YARN-7067.branch-3.0.001.patch
>
>
> Existing resource type info shown as:
> {code}
> [name: memory-mb, units: Mi, type: COUNTABLE, value: 1024, minimum
> allocation: 1024, maximum allocation: 8192, name: vcores, units: , type:
> COUNTABLE, value: 1, minimum allocation: 1, maximum allocation: 4, name:
> resource1, units: G, type: COUNTABLE, value: 0, minimum allocation: 0,
> maximum allocation: 9223372036854775807]
> {code}
> We need to optimize this a little bit.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7067) Optimize ResourceType information display in UI
[
https://issues.apache.org/jira/browse/YARN-7067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-7067:
---
Fix Version/s: 3.0.0
> Optimize ResourceType information display in UI
> ---
>
> Key: YARN-7067
> URL: https://issues.apache.org/jira/browse/YARN-7067
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Critical
> Fix For: 3.0.0, 3.1.0
>
> Attachments: YARN-7067.001.patch, YARN-7067.YARN-3926.001.patch,
> YARN-7067.branch-3.0.001.patch
>
>
> Existing resource type info shown as:
> {code}
> [name: memory-mb, units: Mi, type: COUNTABLE, value: 1024, minimum
> allocation: 1024, maximum allocation: 8192, name: vcores, units: , type:
> COUNTABLE, value: 1, minimum allocation: 1, maximum allocation: 4, name:
> resource1, units: G, type: COUNTABLE, value: 0, minimum allocation: 0,
> maximum allocation: 9223372036854775807]
> {code}
> We need to optimize this a little bit.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7043) Cleanup ResourceProfileManager
[ https://issues.apache.org/jira/browse/YARN-7043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-7043: --- Attachment: YARN-7043.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > Cleanup ResourceProfileManager > -- > > Key: YARN-7043 > URL: https://issues.apache.org/jira/browse/YARN-7043 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Fix For: 3.1.0 > > Attachments: YARN-7043.YARN-3926.001.patch, > YARN-7043.YARN-3926.002.patch, YARN-7043.YARN-3926.003.patch, > YARN-7043.YARN-3926.004.patch, YARN-7043.branch-3.0.001.patch > > > Several cleanups we can do for ResourceProfileManager: > 1) Move GetResourceTypesInfo from profile manager to ResourceUtils. > 2) Move logics to check profile enabled, etc. from ClientRMService to > ResourceUtils. > 3) Throw exception when resource profile is disabled and method accessed by > other modules. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6781) ResourceUtils#initializeResourcesMap takes an unnecessary Map parameter
[
https://issues.apache.org/jira/browse/YARN-6781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-6781:
---
Attachment: YARN-6781.branch-3.0.001.patch
Adding branch-3.0 patch from resource-types branch prior to merge.
> ResourceUtils#initializeResourcesMap takes an unnecessary Map parameter
> ---
>
> Key: YARN-6781
> URL: https://issues.apache.org/jira/browse/YARN-6781
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Yu-Tang Lin
>Priority: Minor
> Labels: newbie
> Fix For: 3.0.0, 3.1.0
>
> Attachments: YARN-6781-YARN-3926.002.patch,
> YARN-6781-YARN-3926.003.patch, YARN-6781-YARN-3926.004.patch,
> YARN-6781.001.patch, YARN-6781.branch-3.0.001.patch
>
>
> The {{resourceInformationMap}} parameter is always passed in as a new {{Map}}
> object, and it's never referenced again after the call. The parameter can be
> eliminated. Instead the {{Map}} can be created inside the
> {{initializeResourcesMap()}} method.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6781) ResourceUtils#initializeResourcesMap takes an unnecessary Map parameter
[
https://issues.apache.org/jira/browse/YARN-6781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-6781:
---
Fix Version/s: 3.0.0
> ResourceUtils#initializeResourcesMap takes an unnecessary Map parameter
> ---
>
> Key: YARN-6781
> URL: https://issues.apache.org/jira/browse/YARN-6781
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Yu-Tang Lin
>Priority: Minor
> Labels: newbie
> Fix For: 3.0.0, 3.1.0
>
> Attachments: YARN-6781-YARN-3926.002.patch,
> YARN-6781-YARN-3926.003.patch, YARN-6781-YARN-3926.004.patch,
> YARN-6781.001.patch, YARN-6781.branch-3.0.001.patch
>
>
> The {{resourceInformationMap}} parameter is always passed in as a new {{Map}}
> object, and it's never referenced again after the call. The parameter can be
> eliminated. Instead the {{Map}} can be created inside the
> {{initializeResourcesMap()}} method.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6789) Add Client API to get all supported resource types from RM
[ https://issues.apache.org/jira/browse/YARN-6789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6789: --- Attachment: YARN-6789.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > Add Client API to get all supported resource types from RM > -- > > Key: YARN-6789 > URL: https://issues.apache.org/jira/browse/YARN-6789 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Fix For: 3.1.0 > > Attachments: YARN-6789-YARN-3926.001.patch, > YARN-6789-YARN-3926.002_incomplete_.patch, YARN-6789-YARN-3926.003.patch, > YARN-6789-YARN-3926.004.patch, YARN-6789-YARN-3926.005.patch, > YARN-6789-YARN-3926.006.patch, YARN-6789.branch-3.0.001.patch > > > It will be better to provide an api to get all supported resource types from > RM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7042) Clean up unit tests after YARN-6610
[ https://issues.apache.org/jira/browse/YARN-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-7042: --- Attachment: YARN-7042.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > Clean up unit tests after YARN-6610 > --- > > Key: YARN-7042 > URL: https://issues.apache.org/jira/browse/YARN-7042 > Project: Hadoop YARN > Issue Type: Sub-task > Components: test >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-7042.YARN-3926.001.patch, > YARN-7042.branch-3.0.001.patch > > > Some of the unit tests in YARN-6610 weren't quite testing what they were > supposed to be testing. This patch fixes that. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7042) Clean up unit tests after YARN-6610
[ https://issues.apache.org/jira/browse/YARN-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-7042: --- Fix Version/s: 3.0.0 > Clean up unit tests after YARN-6610 > --- > > Key: YARN-7042 > URL: https://issues.apache.org/jira/browse/YARN-7042 > Project: Hadoop YARN > Issue Type: Sub-task > Components: test >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-7042.YARN-3926.001.patch, > YARN-7042.branch-3.0.001.patch > > > Some of the unit tests in YARN-6610 weren't quite testing what they were > supposed to be testing. This patch fixes that. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7030) Performance optimizations in Resource and ResourceUtils class
[ https://issues.apache.org/jira/browse/YARN-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-7030: --- Attachment: YARN-7030.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > Performance optimizations in Resource and ResourceUtils class > - > > Key: YARN-7030 > URL: https://issues.apache.org/jira/browse/YARN-7030 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-7030.YARN-3926.001.patch, > YARN-7030.YARN-3926.002.patch, YARN-7030.branch-3.0.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7030) Performance optimizations in Resource and ResourceUtils class
[ https://issues.apache.org/jira/browse/YARN-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-7030: --- Fix Version/s: 3.0.0 > Performance optimizations in Resource and ResourceUtils class > - > > Key: YARN-7030 > URL: https://issues.apache.org/jira/browse/YARN-7030 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-7030.YARN-3926.001.patch, > YARN-7030.YARN-3926.002.patch, YARN-7030.branch-3.0.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6610) DominantResourceCalculator#getResourceAsValue dominant param is updated to handle multiple resources
[
https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-6610:
---
Attachment: YARN-6610.branch-3.0.001.patch
Adding branch-3.0 patch from resource-types branch prior to merge.
> DominantResourceCalculator#getResourceAsValue dominant param is updated to
> handle multiple resources
>
>
> Key: YARN-6610
> URL: https://issues.apache.org/jira/browse/YARN-6610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Fix For: 3.0.0, 3.1.0
>
> Attachments: YARN-6610.001.patch, YARN-6610.YARN-3926.002.patch,
> YARN-6610.YARN-3926.003.patch, YARN-6610.YARN-3926.004.patch,
> YARN-6610.YARN-3926.005.patch, YARN-6610.YARN-3926.006-rebased.patch,
> YARN-6610.YARN-3926.006.patch, YARN-6610.YARN-3926.007.patch,
> YARN-6610.YARN-3926.perf-rebased.patch,
> YARN-6610.YARN-3926.perf-wangda-001.patch,
> YARN-6610.YARN-3926.perf-wangda-002.patch, YARN-6610.YARN-3926.perf.patch,
> YARN-6610.branch-3.0.001.patch
>
>
> The {{dominant}} param assumes there are only two resources, i.e. true means
> to compare the dominant, and false means to compare the subordinate. Now
> that there are _n_ resources, this parameter no longer makes sense.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6610) DominantResourceCalculator#getResourceAsValue dominant param is updated to handle multiple resources
[
https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-6610:
---
Fix Version/s: 3.0.0
> DominantResourceCalculator#getResourceAsValue dominant param is updated to
> handle multiple resources
>
>
> Key: YARN-6610
> URL: https://issues.apache.org/jira/browse/YARN-6610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Fix For: 3.0.0, 3.1.0
>
> Attachments: YARN-6610.001.patch, YARN-6610.YARN-3926.002.patch,
> YARN-6610.YARN-3926.003.patch, YARN-6610.YARN-3926.004.patch,
> YARN-6610.YARN-3926.005.patch, YARN-6610.YARN-3926.006-rebased.patch,
> YARN-6610.YARN-3926.006.patch, YARN-6610.YARN-3926.007.patch,
> YARN-6610.YARN-3926.perf-rebased.patch,
> YARN-6610.YARN-3926.perf-wangda-001.patch,
> YARN-6610.YARN-3926.perf-wangda-002.patch, YARN-6610.YARN-3926.perf.patch,
> YARN-6610.branch-3.0.001.patch
>
>
> The {{dominant}} param assumes there are only two resources, i.e. true means
> to compare the dominant, and false means to compare the subordinate. Now
> that there are _n_ resources, this parameter no longer makes sense.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6892) Improve API implementation in Resources and DominantResourceCalculator class
[ https://issues.apache.org/jira/browse/YARN-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250970#comment-16250970 ] Daniel Templeton commented on YARN-6892: Adding branch-3.0 patch from resource-types branch prior to merge. > Improve API implementation in Resources and DominantResourceCalculator class > > > Key: YARN-6892 > URL: https://issues.apache.org/jira/browse/YARN-6892 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-6892-YARN-3926.001.patch, > YARN-6892-YARN-3926.002.patch, YARN-6892-YARN-3926.003.patch, > YARN-6892-YARN-3926.004.patch, YARN-6892.branch-3.0.001.patch > > > In YARN-3926, apis in Resources and DRC spents significant cpu cycles in most > of its api. For better performance, its better to improve the apis as > resource types order is defined in system level (ResourceUtils class ensures > this post YARN-6788) > This work is preceding to YARN-6788 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6892) Improve API implementation in Resources and DominantResourceCalculator class
[ https://issues.apache.org/jira/browse/YARN-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6892: --- Attachment: YARN-6892.branch-3.0.001.patch > Improve API implementation in Resources and DominantResourceCalculator class > > > Key: YARN-6892 > URL: https://issues.apache.org/jira/browse/YARN-6892 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-6892-YARN-3926.001.patch, > YARN-6892-YARN-3926.002.patch, YARN-6892-YARN-3926.003.patch, > YARN-6892-YARN-3926.004.patch, YARN-6892.branch-3.0.001.patch > > > In YARN-3926, apis in Resources and DRC spents significant cpu cycles in most > of its api. For better performance, its better to improve the apis as > resource types order is defined in system level (ResourceUtils class ensures > this post YARN-6788) > This work is preceding to YARN-6788 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6994) Remove last uses of Long from resource types code
[ https://issues.apache.org/jira/browse/YARN-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6994: --- Fix Version/s: 3.0.0 > Remove last uses of Long from resource types code > - > > Key: YARN-6994 > URL: https://issues.apache.org/jira/browse/YARN-6994 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Minor > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-6994.YARN-3926.001.patch, > YARN-6994.branch-3.0.001.patch > > > Most of the uses have been removed over the last few patches. There's only > one left that I see. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6892) Improve API implementation in Resources and DominantResourceCalculator class
[ https://issues.apache.org/jira/browse/YARN-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6892: --- Fix Version/s: 3.0.0 > Improve API implementation in Resources and DominantResourceCalculator class > > > Key: YARN-6892 > URL: https://issues.apache.org/jira/browse/YARN-6892 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-6892-YARN-3926.001.patch, > YARN-6892-YARN-3926.002.patch, YARN-6892-YARN-3926.003.patch, > YARN-6892-YARN-3926.004.patch, YARN-6892.branch-3.0.001.patch > > > In YARN-3926, apis in Resources and DRC spents significant cpu cycles in most > of its api. For better performance, its better to improve the apis as > resource types order is defined in system level (ResourceUtils class ensures > this post YARN-6788) > This work is preceding to YARN-6788 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6994) Remove last uses of Long from resource types code
[ https://issues.apache.org/jira/browse/YARN-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6994: --- Attachment: YARN-6994.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > Remove last uses of Long from resource types code > - > > Key: YARN-6994 > URL: https://issues.apache.org/jira/browse/YARN-6994 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Minor > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-6994.YARN-3926.001.patch, > YARN-6994.branch-3.0.001.patch > > > Most of the uses have been removed over the last few patches. There's only > one left that I see. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6788) Improve performance of resource profile branch
[ https://issues.apache.org/jira/browse/YARN-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6788: --- Attachment: YARN-6788.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > Improve performance of resource profile branch > -- > > Key: YARN-6788 > URL: https://issues.apache.org/jira/browse/YARN-6788 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Sunil G >Assignee: Sunil G >Priority: Blocker > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-6788-YARN-3926.001.patch, > YARN-6788-YARN-3926.002.patch, YARN-6788-YARN-3926.003.patch, > YARN-6788-YARN-3926.004.patch, YARN-6788-YARN-3926.005.patch, > YARN-6788-YARN-3926.006.patch, YARN-6788-YARN-3926.007.patch, > YARN-6788-YARN-3926.008.patch, YARN-6788-YARN-3926.009.patch, > YARN-6788-YARN-3926.010.patch, YARN-6788-YARN-3926.011.patch, > YARN-6788-YARN-3926.012.patch, YARN-6788-YARN-3926.013.patch, > YARN-6788-YARN-3926.014.patch, YARN-6788-YARN-3926.015.patch, > YARN-6788-YARN-3926.016.patch, YARN-6788-YARN-3926.017.patch, > YARN-6788-YARN-3926.018.patch, YARN-6788-YARN-3926.019.patch, > YARN-6788-YARN-3926.020.patch, YARN-6788-YARN-3926.021.patch, > YARN-6788-YARN-3926.022.patch, YARN-6788-YARN-3926.022.patch, > YARN-6788.branch-3.0.001.patch > > > Currently we could see a 15% performance delta with this branch. > Few performance improvements to improve the same. > Also this patch will handle > [comments|https://issues.apache.org/jira/browse/YARN-6761?focusedCommentId=16075418=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16075418] > from [~leftnoteasy]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6788) Improve performance of resource profile branch
[ https://issues.apache.org/jira/browse/YARN-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6788: --- Fix Version/s: 3.0.0 > Improve performance of resource profile branch > -- > > Key: YARN-6788 > URL: https://issues.apache.org/jira/browse/YARN-6788 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Sunil G >Assignee: Sunil G >Priority: Blocker > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-6788-YARN-3926.001.patch, > YARN-6788-YARN-3926.002.patch, YARN-6788-YARN-3926.003.patch, > YARN-6788-YARN-3926.004.patch, YARN-6788-YARN-3926.005.patch, > YARN-6788-YARN-3926.006.patch, YARN-6788-YARN-3926.007.patch, > YARN-6788-YARN-3926.008.patch, YARN-6788-YARN-3926.009.patch, > YARN-6788-YARN-3926.010.patch, YARN-6788-YARN-3926.011.patch, > YARN-6788-YARN-3926.012.patch, YARN-6788-YARN-3926.013.patch, > YARN-6788-YARN-3926.014.patch, YARN-6788-YARN-3926.015.patch, > YARN-6788-YARN-3926.016.patch, YARN-6788-YARN-3926.017.patch, > YARN-6788-YARN-3926.018.patch, YARN-6788-YARN-3926.019.patch, > YARN-6788-YARN-3926.020.patch, YARN-6788-YARN-3926.021.patch, > YARN-6788-YARN-3926.022.patch, YARN-6788-YARN-3926.022.patch, > YARN-6788.branch-3.0.001.patch > > > Currently we could see a 15% performance delta with this branch. > Few performance improvements to improve the same. > Also this patch will handle > [comments|https://issues.apache.org/jira/browse/YARN-6761?focusedCommentId=16075418=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16075418] > from [~leftnoteasy]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6761) Fix build for YARN-3926 branch
[ https://issues.apache.org/jira/browse/YARN-6761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6761: --- Attachment: YARN-6761.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > Fix build for YARN-3926 branch > -- > > Key: YARN-6761 > URL: https://issues.apache.org/jira/browse/YARN-6761 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-6761-YARN-3926.001.patch, > YARN-6761.branch-3.0.001.patch > > > After rebasing to trunk, due to the addition of YARN-6679, compilation of the > YARN-3926 branch is broken. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6761) Fix build for YARN-3926 branch
[ https://issues.apache.org/jira/browse/YARN-6761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6761: --- Fix Version/s: 3.0.0 > Fix build for YARN-3926 branch > -- > > Key: YARN-6761 > URL: https://issues.apache.org/jira/browse/YARN-6761 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-6761-YARN-3926.001.patch, > YARN-6761.branch-3.0.001.patch > > > After rebasing to trunk, due to the addition of YARN-6679, compilation of the > YARN-3926 branch is broken. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6786) ResourcePBImpl imports cleanup
[ https://issues.apache.org/jira/browse/YARN-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6786: --- Attachment: YARN-6786.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > ResourcePBImpl imports cleanup > -- > > Key: YARN-6786 > URL: https://issues.apache.org/jira/browse/YARN-6786 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Yeliang Cang >Priority: Trivial > Labels: newbie > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-6786-YARN-3926.001.patch, > YARN-6786.branch-3.0.001.patch > > > There is an unused import and an import *. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6786) ResourcePBImpl imports cleanup
[ https://issues.apache.org/jira/browse/YARN-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6786: --- Fix Version/s: 3.0.0 > ResourcePBImpl imports cleanup > -- > > Key: YARN-6786 > URL: https://issues.apache.org/jira/browse/YARN-6786 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Yeliang Cang >Priority: Trivial > Labels: newbie > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-6786-YARN-3926.001.patch, > YARN-6786.branch-3.0.001.patch > > > There is an unused import and an import *. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6445) [YARN-3926] Performance improvements in resource profile branch with respect to SLS
[ https://issues.apache.org/jira/browse/YARN-6445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6445: --- Fix Version/s: 3.0.0 > [YARN-3926] Performance improvements in resource profile branch with respect > to SLS > --- > > Key: YARN-6445 > URL: https://issues.apache.org/jira/browse/YARN-6445 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-6445-YARN-3926.001.patch, > YARN-6445-YARN-3926.002.patch, YARN-6445-YARN-3926.003.patch, > YARN-6445-YARN-3926.004.patch, YARN-6445.branch-3.0.001.patch > > > As part of the SLS runs on YARN-3926, we discovered a bunch of bottlenecks > around object creation and garbage collection. This JIRA is to apply a fix > for those bottlenecks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6445) [YARN-3926] Performance improvements in resource profile branch with respect to SLS
[ https://issues.apache.org/jira/browse/YARN-6445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6445: --- Attachment: YARN-6445.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > [YARN-3926] Performance improvements in resource profile branch with respect > to SLS > --- > > Key: YARN-6445 > URL: https://issues.apache.org/jira/browse/YARN-6445 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-6445-YARN-3926.001.patch, > YARN-6445-YARN-3926.002.patch, YARN-6445-YARN-3926.003.patch, > YARN-6445-YARN-3926.004.patch, YARN-6445.branch-3.0.001.patch > > > As part of the SLS runs on YARN-3926, we discovered a bunch of bottlenecks > around object creation and garbage collection. This JIRA is to apply a fix > for those bottlenecks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5588) Add support for resource profiles in distributed shell
[ https://issues.apache.org/jira/browse/YARN-5588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-5588: --- Attachment: YARN-5588.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > Add support for resource profiles in distributed shell > -- > > Key: YARN-5588 > URL: https://issues.apache.org/jira/browse/YARN-5588 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.1.0 > > Attachments: YARN-5588-YARN-3926.001.patch, > YARN-5588-YARN-3926.002.patch, YARN-5588-YARN-3926.003.patch, > YARN-5588-YARN-3926.004.patch, YARN-5588.branch-3.0.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7443) Add native FPGA module support to do isolation with cgroups
[ https://issues.apache.org/jira/browse/YARN-7443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated YARN-7443: --- Attachment: YARN-7443-trunk.002.patch fix same type define function in FPGA module when building in Redhat 6.8 > Add native FPGA module support to do isolation with cgroups > --- > > Key: YARN-7443 > URL: https://issues.apache.org/jira/browse/YARN-7443 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Zhankun Tang >Assignee: Zhankun Tang > Attachments: YARN-7443-trunk.001.patch, YARN-7443-trunk.002.patch > > > Only support one major number devices configured in c-e.cfg for now. So > almost same with GPU native module -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6232) Update resource usage and preempted resource calculations to take into account all resource types
[ https://issues.apache.org/jira/browse/YARN-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6232: --- Attachment: YARN-6232.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > Update resource usage and preempted resource calculations to take into > account all resource types > - > > Key: YARN-6232 > URL: https://issues.apache.org/jira/browse/YARN-6232 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-6232-YARN-3926.001.patch, > YARN-6232-YARN-3926.002.patch, YARN-6232-YARN-3926.003.patch, > YARN-6232.branch-3.0.001.patch > > > The chargeback calculations that take place on the RM should be updated to > take all resource types into account. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5587) Add support for resource profiles
[ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-5587: --- Attachment: YARN-5587.branch-3.0.001.patch > Add support for resource profiles > - > > Key: YARN-5587 > URL: https://issues.apache.org/jira/browse/YARN-5587 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Labels: oct16-hard > Fix For: 3.1.0 > > Attachments: YARN-5587-YARN-3926.001.patch, > YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch, > YARN-5587-YARN-3926.004.patch, YARN-5587-YARN-3926.005.patch, > YARN-5587-YARN-3926.006.patch, YARN-5587-YARN-3926.007.patch, > YARN-5587-YARN-3926.008.patch, YARN-5587-YARN-3926.009.patch, > YARN-5587-YARN-3926.010.patch, YARN-5587-YARN-3926.011.patch, > YARN-5587-YARN-3926.012.patch, YARN-5587-YARN-3926.013.patch, > YARN-5587-YARN-3926.014.patch, YARN-5587-YARN-3926.015.patch, > YARN-5587.branch-3.0.001.patch > > > Add support for resource profiles on the RM side to allow users to use > shorthands to specify resource requirements. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5587) Add support for resource profiles
[ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250955#comment-16250955 ] Daniel Templeton commented on YARN-5587: Adding branch-3.0 patch from resource-types branch prior to merge. > Add support for resource profiles > - > > Key: YARN-5587 > URL: https://issues.apache.org/jira/browse/YARN-5587 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Labels: oct16-hard > Fix For: 3.1.0 > > Attachments: YARN-5587-YARN-3926.001.patch, > YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch, > YARN-5587-YARN-3926.004.patch, YARN-5587-YARN-3926.005.patch, > YARN-5587-YARN-3926.006.patch, YARN-5587-YARN-3926.007.patch, > YARN-5587-YARN-3926.008.patch, YARN-5587-YARN-3926.009.patch, > YARN-5587-YARN-3926.010.patch, YARN-5587-YARN-3926.011.patch, > YARN-5587-YARN-3926.012.patch, YARN-5587-YARN-3926.013.patch, > YARN-5587-YARN-3926.014.patch, YARN-5587-YARN-3926.015.patch, > YARN-5587.branch-3.0.001.patch > > > Add support for resource profiles on the RM side to allow users to use > shorthands to specify resource requirements. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5586) Update the Resources class to consider all resource types
[ https://issues.apache.org/jira/browse/YARN-5586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-5586: --- Fix Version/s: 3.0.0 > Update the Resources class to consider all resource types > - > > Key: YARN-5586 > URL: https://issues.apache.org/jira/browse/YARN-5586 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-5586-YARN-3926.001.patch, > YARN-5586-YARN-3926.002.patch, YARN-5586.branch-3.0.001.patch > > > The Resources class provides a bunch of useful functions like clone, addTo, > etc. These need to be updated to consider all resource types instead of just > memory and cpu. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5586) Update the Resources class to consider all resource types
[ https://issues.apache.org/jira/browse/YARN-5586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-5586: --- Attachment: YARN-5586.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > Update the Resources class to consider all resource types > - > > Key: YARN-5586 > URL: https://issues.apache.org/jira/browse/YARN-5586 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-5586-YARN-3926.001.patch, > YARN-5586-YARN-3926.002.patch, YARN-5586.branch-3.0.001.patch > > > The Resources class provides a bunch of useful functions like clone, addTo, > etc. These need to be updated to consider all resource types instead of just > memory and cpu. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4830) Add support for resource types in the nodemanager
[ https://issues.apache.org/jira/browse/YARN-4830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-4830: --- Fix Version/s: 3.0.0 > Add support for resource types in the nodemanager > - > > Key: YARN-4830 > URL: https://issues.apache.org/jira/browse/YARN-4830 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-4830-YARN-3926.001.patch, > YARN-4830-YARN-3926.002.patch, YARN-4830-YARN-3926.003.patch, > YARN-4830-YARN-3926.004.patch, YARN-4830.branch-3.0.001.patch > > > The RM has support for multiple resource types. The same should be added for > the NMs. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4830) Add support for resource types in the nodemanager
[ https://issues.apache.org/jira/browse/YARN-4830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-4830: --- Attachment: YARN-4830.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > Add support for resource types in the nodemanager > - > > Key: YARN-4830 > URL: https://issues.apache.org/jira/browse/YARN-4830 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-4830-YARN-3926.001.patch, > YARN-4830-YARN-3926.002.patch, YARN-4830-YARN-3926.003.patch, > YARN-4830-YARN-3926.004.patch, YARN-4830.branch-3.0.001.patch > > > The RM has support for multiple resource types. The same should be added for > the NMs. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5242) Update DominantResourceCalculator to consider all resource types in calculations
[ https://issues.apache.org/jira/browse/YARN-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-5242: --- Attachment: YARN-5242.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > Update DominantResourceCalculator to consider all resource types in > calculations > > > Key: YARN-5242 > URL: https://issues.apache.org/jira/browse/YARN-5242 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-5242-YARN-3926.001.patch, > YARN-5242-YARN-3926.002.patch, YARN-5242-YARN-3926.003.patch, > YARN-5242-YARN-3926.004.patch, YARN-5242-YARN-3926.005.patch, > YARN-5242-YARN-3926.006.patch, YARN-5242.branch-3.0.001.patch > > > The fitsIn function in the DominantResourceCalculator only looks at memory > and cpu. It should be modified to use all available resource types. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5242) Update DominantResourceCalculator to consider all resource types in calculations
[ https://issues.apache.org/jira/browse/YARN-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-5242: --- Fix Version/s: 3.0.0 > Update DominantResourceCalculator to consider all resource types in > calculations > > > Key: YARN-5242 > URL: https://issues.apache.org/jira/browse/YARN-5242 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-5242-YARN-3926.001.patch, > YARN-5242-YARN-3926.002.patch, YARN-5242-YARN-3926.003.patch, > YARN-5242-YARN-3926.004.patch, YARN-5242-YARN-3926.005.patch, > YARN-5242-YARN-3926.006.patch, YARN-5242.branch-3.0.001.patch > > > The fitsIn function in the DominantResourceCalculator only looks at memory > and cpu. It should be modified to use all available resource types. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4829) Add support for binary units
[ https://issues.apache.org/jira/browse/YARN-4829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-4829: --- Fix Version/s: 3.0.0 > Add support for binary units > > > Key: YARN-4829 > URL: https://issues.apache.org/jira/browse/YARN-4829 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-4829-YARN-3926.001.patch, > YARN-4829-YARN-3926.002.patch, YARN-4829-YARN-3926.003.patch, > YARN-4829-YARN-3926.004.patch, YARN-4829.branch-3.0.001.patch > > > The units conversion util should have support for binary units. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4715) Add support to read resource types from a config file
[ https://issues.apache.org/jira/browse/YARN-4715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-4715: --- Attachment: YARN-4715.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > Add support to read resource types from a config file > - > > Key: YARN-4715 > URL: https://issues.apache.org/jira/browse/YARN-4715 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-4715-YARN-3926.001.patch, > YARN-4715-YARN-3926.002.patch, YARN-4715-YARN-3926.003.patch, > YARN-4715-YARN-3926.004.patch, YARN-4715-YARN-3926.005.patch, > YARN-4715.branch-3.0.001.patch > > > This ticket is to add support to allow the RM to read the resource types to > be used for scheduling from a config file. I'll file follow up tickets to add > similar support in the NM as well as to handle the RM-NM handshake protocol > issues. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4829) Add support for binary units
[ https://issues.apache.org/jira/browse/YARN-4829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-4829: --- Attachment: YARN-4829.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > Add support for binary units > > > Key: YARN-4829 > URL: https://issues.apache.org/jira/browse/YARN-4829 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-4829-YARN-3926.001.patch, > YARN-4829-YARN-3926.002.patch, YARN-4829-YARN-3926.003.patch, > YARN-4829-YARN-3926.004.patch, YARN-4829.branch-3.0.001.patch > > > The units conversion util should have support for binary units. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4715) Add support to read resource types from a config file
[ https://issues.apache.org/jira/browse/YARN-4715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-4715: --- Fix Version/s: 3.0.0 > Add support to read resource types from a config file > - > > Key: YARN-4715 > URL: https://issues.apache.org/jira/browse/YARN-4715 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-4715-YARN-3926.001.patch, > YARN-4715-YARN-3926.002.patch, YARN-4715-YARN-3926.003.patch, > YARN-4715-YARN-3926.004.patch, YARN-4715-YARN-3926.005.patch, > YARN-4715.branch-3.0.001.patch > > > This ticket is to add support to allow the RM to read the resource types to > be used for scheduling from a config file. I'll file follow up tickets to add > similar support in the NM as well as to handle the RM-NM handshake protocol > issues. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4172) Extend DominantResourceCalculator to account for all resources
[ https://issues.apache.org/jira/browse/YARN-4172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-4172: --- Attachment: YARN-4172.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > Extend DominantResourceCalculator to account for all resources > -- > > Key: YARN-4172 > URL: https://issues.apache.org/jira/browse/YARN-4172 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-4172-YARN-3926.001.patch, > YARN-4172-YARN-3926.002.patch, YARN-4172-YARN-3926.003.patch, > YARN-4172.branch-3.0.001.patch > > > Now that support for multiple resources is present in the resource class, we > need to modify DominantResourceCalculator to account for the new resources. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4172) Extend DominantResourceCalculator to account for all resources
[ https://issues.apache.org/jira/browse/YARN-4172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-4172: --- Fix Version/s: 3.0.0 > Extend DominantResourceCalculator to account for all resources > -- > > Key: YARN-4172 > URL: https://issues.apache.org/jira/browse/YARN-4172 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-4172-YARN-3926.001.patch, > YARN-4172-YARN-3926.002.patch, YARN-4172-YARN-3926.003.patch, > YARN-4172.branch-3.0.001.patch > > > Now that support for multiple resources is present in the resource class, we > need to modify DominantResourceCalculator to account for the new resources. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4081) Add support for multiple resource types in the Resource class
[ https://issues.apache.org/jira/browse/YARN-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-4081: --- Attachment: YARN-4081.branch-3.0.001.patch Adding branch-3.0 patch from resource-types branch prior to merge. > Add support for multiple resource types in the Resource class > - > > Key: YARN-4081 > URL: https://issues.apache.org/jira/browse/YARN-4081 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Fix For: 3.0.0, 3.1.0 > > Attachments: YARN-4081-YARN-3926.001.patch, > YARN-4081-YARN-3926.002.patch, YARN-4081-YARN-3926.003.patch, > YARN-4081-YARN-3926.004.patch, YARN-4081-YARN-3926.005.patch, > YARN-4081-YARN-3926.006.patch, YARN-4081-YARN-3926.007.patch, > YARN-4081-YARN-3926.008.patch, YARN-4081.branch-3.0.001.patch > > > For adding support for multiple resource types, we need to add support for > this in the Resource class. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6507) Add support in NodeManager to isolate FPGA devices with CGroups
[ https://issues.apache.org/jira/browse/YARN-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated YARN-6507: --- Attachment: YARN-6507-trunk.007.patch fix the white space > Add support in NodeManager to isolate FPGA devices with CGroups > --- > > Key: YARN-6507 > URL: https://issues.apache.org/jira/browse/YARN-6507 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Zhankun Tang >Assignee: Zhankun Tang > Attachments: YARN-6507-branch-YARN-3926.001.patch, > YARN-6507-branch-YARN-3926.002.patch, YARN-6507-trunk.001.patch, > YARN-6507-trunk.002.patch, YARN-6507-trunk.003.patch, > YARN-6507-trunk.004.patch, YARN-6507-trunk.005.patch, > YARN-6507-trunk.006.patch, YARN-6507-trunk.007.patch > > > Support local FPGA resource scheduler to assign/isolate N FPGA slots to a > container. > At the beginning, support one vendor plugin with basic features to serve > OpenCL applications -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7462) Render outstanding resource requests on application details page
[ https://issues.apache.org/jira/browse/YARN-7462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vasudevan Skm updated YARN-7462: Attachment: YARN-7462.003.patch [~sunil.gov...@gmail.com] > Render outstanding resource requests on application details page > > > Key: YARN-7462 > URL: https://issues.apache.org/jira/browse/YARN-7462 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm > Attachments: Screen Shot 2017-11-08 at 3.24.30 PM.png, Screen Shot > 2017-11-08 at 3.38.48 PM.png, YARN-7462.001.patch, YARN-7462.002.patch, > YARN-7462.003.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6507) Add support in NodeManager to isolate FPGA devices with CGroups
[
https://issues.apache.org/jira/browse/YARN-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250922#comment-16250922
]
Hadoop QA commented on YARN-6507:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 15s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
20s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 4s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 180 new + 215 unchanged - 0 fixed = 395 total (was 215) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 50s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
38s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
45s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 17m
3s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 93m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-6507 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12897442/YARN-6507-trunk.006.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit
[jira] [Updated] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
[ https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikandan R updated YARN-7159: --- Attachment: YARN-7159.017.patch > Normalize unit of resource objects in RM and avoid to do unit conversion in > critical path > - > > Key: YARN-7159 > URL: https://issues.apache.org/jira/browse/YARN-7159 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Manikandan R >Priority: Critical > Attachments: YARN-7159.001.patch, YARN-7159.002.patch, > YARN-7159.003.patch, YARN-7159.004.patch, YARN-7159.005.patch, > YARN-7159.006.patch, YARN-7159.007.patch, YARN-7159.008.patch, > YARN-7159.009.patch, YARN-7159.010.patch, YARN-7159.011.patch, > YARN-7159.012.patch, YARN-7159.013.patch, YARN-7159.015.patch, > YARN-7159.016.patch, YARN-7159.017.patch > > > Currently resource conversion could happen in critical code path when > different unit is specified by client. This could impact performance and > throughput of RM a lot. We should do unit normalization when resource passed > to RM and avoid expensive unit conversion every time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
[ https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250916#comment-16250916 ] Manikandan R commented on YARN-7159: Rebasing my patch. > Normalize unit of resource objects in RM and avoid to do unit conversion in > critical path > - > > Key: YARN-7159 > URL: https://issues.apache.org/jira/browse/YARN-7159 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Manikandan R >Priority: Critical > Attachments: YARN-7159.001.patch, YARN-7159.002.patch, > YARN-7159.003.patch, YARN-7159.004.patch, YARN-7159.005.patch, > YARN-7159.006.patch, YARN-7159.007.patch, YARN-7159.008.patch, > YARN-7159.009.patch, YARN-7159.010.patch, YARN-7159.011.patch, > YARN-7159.012.patch, YARN-7159.013.patch, YARN-7159.015.patch, > YARN-7159.016.patch, YARN-7159.017.patch > > > Currently resource conversion could happen in critical code path when > different unit is specified by client. This could impact performance and > throughput of RM a lot. We should do unit normalization when resource passed > to RM and avoid expensive unit conversion every time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7486) Race condition in service AM that can cause NPE
[
https://issues.apache.org/jira/browse/YARN-7486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250907#comment-16250907
]
Hadoop QA commented on YARN-7486:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 15s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications: The patch generated
15 new + 88 unchanged - 5 fixed = 103 total (was 93) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 18s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
31s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
19s{color} | {color:green} hadoop-yarn-services-api in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7486 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12897445/YARN-7486.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 4ed12f2f22f4 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 4f40cd3 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (YARN-7462) Render outstanding resource requests on application details page
[ https://issues.apache.org/jira/browse/YARN-7462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250902#comment-16250902 ] Vasudevan Skm commented on YARN-7462: - [~sunil.gov...@gmail.com]Thank you for the comments. Sure, let's handle the API changes in the next patch About the other comments, 1. `.history` is created by `vscode`. This will be helpful for people using vscode as their primary editor 2-5. Sure. Will update the patch 6. Hiding the table is bad UX. This will confuse the users. We render no data available in these cases, so the users don't get confused. > Render outstanding resource requests on application details page > > > Key: YARN-7462 > URL: https://issues.apache.org/jira/browse/YARN-7462 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm > Attachments: Screen Shot 2017-11-08 at 3.24.30 PM.png, Screen Shot > 2017-11-08 at 3.38.48 PM.png, YARN-7462.001.patch, YARN-7462.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7430) User and Group mapping are incorrect in docker container
[ https://issues.apache.org/jira/browse/YARN-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250879#comment-16250879 ] Varun Vasudev commented on YARN-7430: - I'm fine with it. [~ebadger] - does it sounds ok to you? > User and Group mapping are incorrect in docker container > > > Key: YARN-7430 > URL: https://issues.apache.org/jira/browse/YARN-7430 > Project: Hadoop YARN > Issue Type: Sub-task > Components: security, yarn >Affects Versions: 2.9.0, 3.0.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Blocker > Attachments: YARN-7430.001.patch, YARN-7430.png > > > In YARN-4266, the recommendation was to use -u [uid]:[gid] numeric values to > enforce user and group for the running user. In YARN-6623, this translated > to --user=test --group-add=group1. The code no longer enforce group > correctly for launched process. > In addition, the implementation in YARN-6623 requires the user and group > information to exist in container to translate username and group to uid/gid. > For users on LDAP, there is no good way to populate container with user and > group information. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7491) Make sure AM is not scheduled on an opportunistic container
Haibo Chen created YARN-7491: Summary: Make sure AM is not scheduled on an opportunistic container Key: YARN-7491 URL: https://issues.apache.org/jira/browse/YARN-7491 Project: Hadoop YARN Issue Type: Bug Components: scheduler Reporter: Haibo Chen Assignee: Haibo Chen -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7462) Render outstanding resource requests on application details page
[
https://issues.apache.org/jira/browse/YARN-7462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250868#comment-16250868
]
Sunil G commented on YARN-7462:
---
[~skmvasu] Thanks for the patch.
I have one issue in this patch. Post YARN-3926, we support more resource types
other than memory and vcores.
But REST endpoint
org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.ResourceInfo is not
having multiple resource support in a clean way. So post this patch, we need to
raise a common ticket in UI to show resources other than memory and cpu also.
Some minor comments:
# {{.history/}} is added to git ignore list. Why this is needed?
# Rename {{Priority}} to {{Scheduler Key}}
# {{ResourceName}} to {{Resource Name}}. A space to be added for better
readability
# {{NumContainers}}. Could we say like {{# Containers}} as it indicates a count.
# {{Outstanding resource requests}} to {{Outstanding Resource Requests}}
# If {{model.app.resourceRequests}} is empty, could we hide the whole table?
> Render outstanding resource requests on application details page
>
>
> Key: YARN-7462
> URL: https://issues.apache.org/jira/browse/YARN-7462
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-ui-v2
>Reporter: Vasudevan Skm
>Assignee: Vasudevan Skm
> Attachments: Screen Shot 2017-11-08 at 3.24.30 PM.png, Screen Shot
> 2017-11-08 at 3.38.48 PM.png, YARN-7462.001.patch, YARN-7462.002.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7361) Improve the docker container runtime documentation
[ https://issues.apache.org/jira/browse/YARN-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250865#comment-16250865 ] Varun Vasudev commented on YARN-7361: - +1 - I'll commit this tomorrow if no one objects. > Improve the docker container runtime documentation > -- > > Key: YARN-7361 > URL: https://issues.apache.org/jira/browse/YARN-7361 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Attachments: YARN-7361.001.patch > > > During review of YARN-7230, it was found that > yarn.nodemanager.runtime.linux.docker.capabilities is missing from the docker > containers documentation in most of the active branches. We can also improve > the warning that was introduced in YARN-6622. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7490) ConcurrentModificationException in AMRMClient
[
https://issues.apache.org/jira/browse/YARN-7490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250866#comment-16250866
]
Arun Suresh commented on YARN-7490:
---
Hmmm... I agree, it can maybe be handled in the AMRMClient itself. Looking at
the code:
{code}
public Collection getMatchingRequests(long allocationRequestId) {
RemoteRequestsTable remoteRequestsTable = getTable(allocationRequestId);
LinkedHashSet list = new LinkedHashSet<>();
if (remoteRequestsTable != null) {
Iterator reqIter =
remoteRequestsTable.iterator();
while (reqIter.hasNext()) {
ResourceRequestInfo resReqInfo = reqIter.next();
list.addAll(resReqInfo.containerRequests);
}
}
return list;
}
{code}
Maybe we just create a LinkedList instead of a LinkedHashSet and then before
returning, convert it to a Set ?
> ConcurrentModificationException in AMRMClient
> -
>
> Key: YARN-7490
> URL: https://issues.apache.org/jira/browse/YARN-7490
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jian He
>
> It is observed that below exception occurred in service AM, looks like the
> AMRMClientImpl -> ResourceRequestInfo -> containerRequests can be
> concurrently accessed without protection
> {code}
> 2017-11-13 19:48:10,487 [AMRM Callback Handler Thread] ERROR
> service.ServiceScheduler - Error in AMRMClient callback handler
> java.util.ConcurrentModificationException
> at
> java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:711)
> at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:734)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:343)
> at
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.getMatchingRequests(AMRMClientImpl.java:680)
> at
> org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.getMatchingRequests(AMRMClientAsync.java:239)
> at
> org.apache.hadoop.yarn.service.ServiceScheduler$AMRMClientCallback.onContainersAllocated(ServiceScheduler.java:486)
> at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:359)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7486) Race condition in service AM that can cause NPE
[
https://issues.apache.org/jira/browse/YARN-7486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jian He updated YARN-7486:
--
Attachment: YARN-7486.01.patch
> Race condition in service AM that can cause NPE
> ---
>
> Key: YARN-7486
> URL: https://issues.apache.org/jira/browse/YARN-7486
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-7486.01.patch
>
>
> 1. container1 completed for instance1
> 2. instance1 is added to pending list
> 3. container2 allocated, and assigned to instance1, it records the container2
> inside instance1
> 4. in the meantime, instance1 ContainerStoppedTransition is called and that
> set the container back to null.
> This cause the recorded container lost.
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.service.provider.ProviderUtils.initCompTokensForSubstitute(ProviderUtils.java:402)
> at
> org.apache.hadoop.yarn.service.provider.AbstractProviderService.buildContainerLaunchContext(AbstractProviderService.java:70)
> at
> org.apache.hadoop.yarn.service.containerlaunch.ContainerLaunchService$ContainerLauncher.run(ContainerLaunchService.java:89)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7486) Race condition in service AM that can cause NPE
[
https://issues.apache.org/jira/browse/YARN-7486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jian He updated YARN-7486:
--
Attachment: (was: YARN-7486.01.patch)
> Race condition in service AM that can cause NPE
> ---
>
> Key: YARN-7486
> URL: https://issues.apache.org/jira/browse/YARN-7486
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
>
> 1. container1 completed for instance1
> 2. instance1 is added to pending list
> 3. container2 allocated, and assigned to instance1, it records the container2
> inside instance1
> 4. in the meantime, instance1 ContainerStoppedTransition is called and that
> set the container back to null.
> This cause the recorded container lost.
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.service.provider.ProviderUtils.initCompTokensForSubstitute(ProviderUtils.java:402)
> at
> org.apache.hadoop.yarn.service.provider.AbstractProviderService.buildContainerLaunchContext(AbstractProviderService.java:70)
> at
> org.apache.hadoop.yarn.service.containerlaunch.ContainerLaunchService$ContainerLauncher.run(ContainerLaunchService.java:89)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7486) Race condition in service AM that can cause NPE
[
https://issues.apache.org/jira/browse/YARN-7486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jian He updated YARN-7486:
--
Attachment: YARN-7486.01.patch
> Race condition in service AM that can cause NPE
> ---
>
> Key: YARN-7486
> URL: https://issues.apache.org/jira/browse/YARN-7486
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-7486.01.patch
>
>
> 1. container1 completed for instance1
> 2. instance1 is added to pending list
> 3. container2 allocated, and assigned to instance1, it records the container2
> inside instance1
> 4. in the meantime, instance1 ContainerStoppedTransition is called and that
> set the container back to null.
> This cause the recorded container lost.
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.service.provider.ProviderUtils.initCompTokensForSubstitute(ProviderUtils.java:402)
> at
> org.apache.hadoop.yarn.service.provider.AbstractProviderService.buildContainerLaunchContext(AbstractProviderService.java:70)
> at
> org.apache.hadoop.yarn.service.containerlaunch.ContainerLaunchService$ContainerLauncher.run(ContainerLaunchService.java:89)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7481) Gpu locality support for Better AI scheduling
[
https://issues.apache.org/jira/browse/YARN-7481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250823#comment-16250823
]
Wangda Tan commented on YARN-7481:
--
Thanks [~qinc...@microsoft.com]/Myeongjae for filing the JIRA and uploading
design docs.
As discussed offline, even though I think the ideal case is to let YARN
understand GPU hierarchy and assign GPUs with best interconnect. I'm still
looking to bring this effort forward since this enables some use cases of GPU
and I can see the common functionalities of local resource affinity can be
reused by other features as well (for example network/SSDs/CPU, etc.).
Regarding to the design / prototype:
The proposed approach added one more field to the resource class. I'm not sure
what's the best solution of this. Probably we can add a new type to
{{org.apache.hadoop.yarn.api.protocolrecords.ResourceTypes}} (Such as
vector/bitmap, etc.), and an inherited ResourceInformation class to return the
vector/bitmap. Bitmap/vector type will be excluded from scheduling / DRF
calculation, etc.
+ folks who might be interested in this effort:
[~subru]/[~sunilg]/[~vvasudev]/[~templedf]/[~asuresh]/[~jlowe].
> Gpu locality support for Better AI scheduling
> -
>
> Key: YARN-7481
> URL: https://issues.apache.org/jira/browse/YARN-7481
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: api, RM, yarn
>Affects Versions: 2.7.2
>Reporter: Chen Qingcha
> Fix For: 2.7.2
>
> Attachments: GPU locality support for Job scheduling.pdf,
> hadoop-2.7.2-gpu.patch
>
> Original Estimate: 1,344h
> Remaining Estimate: 1,344h
>
> We enhance Hadoop with GPU support for better AI job scheduling.
> Currently, YARN-3926 also supports GPU scheduling, which treats GPU as
> countable resource.
> However, GPU placement is also very important to deep learning job for better
> efficiency.
> For example, a 2-GPU job runs on gpu {0,1} could be faster than run on gpu
> {0, 7}, if GPU 0 and 1 are under the same PCI-E switch while 0 and 7 are not.
> We add the support to Hadoop 2.7.2 to enable GPU locality scheduling, which
> support fine-grained GPU placement.
> A 64-bits bitmap is added to yarn Resource, which indicates both GPU usage
> and locality information in a node (up to 64 GPUs per node). '1' means
> available and '0' otherwise in the corresponding position of the bit.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6507) Add support in NodeManager to isolate FPGA devices with CGroups
[ https://issues.apache.org/jira/browse/YARN-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated YARN-6507: --- Attachment: YARN-6507-trunk.006.patch Add configuration and abstraction for vendor FPGA plugin > Add support in NodeManager to isolate FPGA devices with CGroups > --- > > Key: YARN-6507 > URL: https://issues.apache.org/jira/browse/YARN-6507 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Zhankun Tang >Assignee: Zhankun Tang > Attachments: YARN-6507-branch-YARN-3926.001.patch, > YARN-6507-branch-YARN-3926.002.patch, YARN-6507-trunk.001.patch, > YARN-6507-trunk.002.patch, YARN-6507-trunk.003.patch, > YARN-6507-trunk.004.patch, YARN-6507-trunk.005.patch, > YARN-6507-trunk.006.patch > > > Support local FPGA resource scheduler to assign/isolate N FPGA slots to a > container. > At the beginning, support one vendor plugin with basic features to serve > OpenCL applications -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7490) ConcurrentModificationException in AMRMClient
[
https://issues.apache.org/jira/browse/YARN-7490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250808#comment-16250808
]
Jian He commented on YARN-7490:
---
[~leftnoteasy], [~subru], [~asuresh].
should AMRMClient handle this internally ?
> ConcurrentModificationException in AMRMClient
> -
>
> Key: YARN-7490
> URL: https://issues.apache.org/jira/browse/YARN-7490
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jian He
>
> It is observed that below exception occurred in service AM, looks like the
> AMRMClientImpl -> ResourceRequestInfo -> containerRequests can be
> concurrently accessed without protection
> {code}
> 2017-11-13 19:48:10,487 [AMRM Callback Handler Thread] ERROR
> service.ServiceScheduler - Error in AMRMClient callback handler
> java.util.ConcurrentModificationException
> at
> java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:711)
> at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:734)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:343)
> at
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.getMatchingRequests(AMRMClientImpl.java:680)
> at
> org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.getMatchingRequests(AMRMClientAsync.java:239)
> at
> org.apache.hadoop.yarn.service.ServiceScheduler$AMRMClientCallback.onContainersAllocated(ServiceScheduler.java:486)
> at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:359)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7490) ConcurrentModificationException in AMRMClient
Jian He created YARN-7490:
-
Summary: ConcurrentModificationException in AMRMClient
Key: YARN-7490
URL: https://issues.apache.org/jira/browse/YARN-7490
Project: Hadoop YARN
Issue Type: Bug
Reporter: Jian He
It is observed that below exception occurred in service AM, looks like the
AMRMClientImpl -> ResourceRequestInfo -> containerRequests can be concurrently
accessed without protection
{code}
2017-11-13 19:48:10,487 [AMRM Callback Handler Thread] ERROR
service.ServiceScheduler - Error in AMRMClient callback handler
java.util.ConcurrentModificationException
at java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:711)
at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:734)
at java.util.AbstractCollection.addAll(AbstractCollection.java:343)
at
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.getMatchingRequests(AMRMClientImpl.java:680)
at
org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.getMatchingRequests(AMRMClientAsync.java:239)
at
org.apache.hadoop.yarn.service.ServiceScheduler$AMRMClientCallback.onContainersAllocated(ServiceScheduler.java:486)
at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:359)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7218) ApiServer REST API naming convention /ws/v1 is already used in Hadoop v2
[
https://issues.apache.org/jira/browse/YARN-7218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250797#comment-16250797
]
Hadoop QA commented on YARN-7218:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
1s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 52s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
28s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}134m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
| |
hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisherForV2 |
| | hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher |
| Timed out junit tests |
org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands |
| |
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
| | org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
|
\\
\\
|| Subsystem ||
[jira] [Commented] (YARN-7488) Make ServiceClient.getAppId method public to return ApplicationId for a service name
[
https://issues.apache.org/jira/browse/YARN-7488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250767#comment-16250767
]
Hudson commented on YARN-7488:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13235 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13235/])
YARN-7488. Make ServiceClient.getAppId method public to return (jianhe: rev
4f40cd314ab14f735a465fb9dff2dc1bf118e703)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/client/ServiceClient.java
> Make ServiceClient.getAppId method public to return ApplicationId for a
> service name
>
>
> Key: YARN-7488
> URL: https://issues.apache.org/jira/browse/YARN-7488
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Gour Saha
>Assignee: Gour Saha
> Fix For: 3.1.0
>
> Attachments: YARN-7488.001.patch
>
>
> There are several APIs in YarnClient which can be used when the ApplicationId
> is available. For a YARN service getting the ApplicationId corresponding to a
> service name will open up all these APIs for the service. Hence making the
> below API public will be very helpful.
> {code}
> private synchronized ApplicationId getAppId(String serviceName)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7488) Make ServiceClient.getAppId method public to return ApplicationId for a service name
[
https://issues.apache.org/jira/browse/YARN-7488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250713#comment-16250713
]
Hadoop QA commented on YARN-7488:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 30s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
27s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7488 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12897422/YARN-7488.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux db39bb3b5d5c 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 5323b00 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/18472/testReport/ |
| Max. process+thread count | 625 (vs. ulimit of 5000) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/18472/console |
| Powered by |
[jira] [Commented] (YARN-7487) Make sure volume includes GPU base libraries exists after created by plugin
[
https://issues.apache.org/jira/browse/YARN-7487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250710#comment-16250710
]
Hadoop QA commented on YARN-7487:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 15s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 2 new + 11 unchanged - 1 fixed = 13 total (was 12) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 59s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 36s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7487 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12897417/YARN-7487.wip.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle cc |
| uname | Linux 7caf8a2b0519 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 5323b00 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (YARN-6078) Containers stuck in Localizing state
[
https://issues.apache.org/jira/browse/YARN-6078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250704#comment-16250704
]
Hadoop QA commented on YARN-6078:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 25m
1s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
16s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m
3s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:17213a0 |
| JIRA Issue | YARN-6078 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12897419/YARN-6078-branch-2.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 07c4c29dc783 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / f894eef |
| maven | version: Apache Maven 3.3.9
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| Default Java | 1.7.0_151 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/18471/testReport/ |
| Max. process+thread count | 156 (vs. ulimit of 5000) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/18471/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Containers stuck in Localizing state
>
>
> Key: YARN-6078
> URL: https://issues.apache.org/jira/browse/YARN-6078
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jagadish
>Assignee: Billie Rinaldi
> Attachments:
[jira] [Updated] (YARN-7489) ConcurrentModificationException in RMAppImpl#getRMAppMetrics
[
https://issues.apache.org/jira/browse/YARN-7489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Yang updated YARN-7489:
---
Description:
The REST clients have sometimes failed to query applications through apps REST
API in RMWebService and it happened when iterating
attempts(RMWebServices#getApps --> AppInfo# -->
RMAppImpl#getRMAppMetrics) and meanwhile these attempts
changed(AttemptFailedTransition#transition -->
RMAppImpl#createAndStartNewAttempt --> RMAppImpl#createNewAttempt).
Application state changed within the lockup period of writeLock in RMAppImpl,
so that we can add readLock before iterating attempts to fix this problem.
Exception stack:
{noformat}
java.util.ConcurrentModificationException
at
java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
at
java.util.LinkedHashMap$LinkedValueIterator.next(LinkedHashMap.java:747)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.getRMAppMetrics(RMAppImpl.java:1487)
at
org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.AppInfo.(AppInfo.java:199)
at
org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices.getApps(RMWebServices.java:597)
at sun.reflect.GeneratedMethodAccessor81.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
at
org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:178)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
{noformat}
was:
The REST clients have sometimes failed to query applications through apps REST
API in RMWebService and it happened when iterating
attempts(RMWebServices#getApps --> AppInfo# -->
RMAppImpl#getRMAppMetrics) and meanwhile these attempts
changed(AttemptFailedTransition#transition -->
RMAppImpl#createAndStartNewAttempt --> RMAppImpl#createNewAttempt).
Application state changed within the lockup period of writeLock in RMAppImpl,
so that we can add readLock before iterating attempts to fix this problem.
Error logs:
{noformat}
java.util.ConcurrentModificationException
at
java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
at
java.util.LinkedHashMap$LinkedValueIterator.next(LinkedHashMap.java:747)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.getRMAppMetrics(RMAppImpl.java:1487)
at
org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.AppInfo.(AppInfo.java:199)
at
org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices.getApps(RMWebServices.java:597)
at sun.reflect.GeneratedMethodAccessor81.invoke(Unknown Source)
at
