[jira] [Commented] (YARN-6509) Add a size threshold beyond which yarn logs will require a force option
[
https://issues.apache.org/jira/browse/YARN-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978227#comment-15978227
]
Hadoop QA commented on YARN-6509:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
30s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client in
trunk has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The patch generated 6 new +
101 unchanged - 2 fixed = 107 total (was 103) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m
4s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | YARN-6509 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12864440/YARN-6509.1.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 0defd523e5b7 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / b080338 |
| Default Java | 1.8.0_121 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/15708/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/15708/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15708/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15708/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add a size threshold beyond which yarn logs will require
[jira] [Commented] (YARN-6445) Improve YARN-3926 performance with respect to SLS
[ https://issues.apache.org/jira/browse/YARN-6445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978395#comment-15978395 ] Sunil G commented on YARN-6445: --- Latest patch looks good for me. Jenkins failure is due to UI and we can skip that for now. Once this branch is rebased, it will go away. I could commit this later today is there are no other comments. [~templedf], could you please confirm. > Improve YARN-3926 performance with respect to SLS > - > > Key: YARN-6445 > URL: https://issues.apache.org/jira/browse/YARN-6445 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-6445-YARN-3926.001.patch, > YARN-6445-YARN-3926.002.patch, YARN-6445-YARN-3926.003.patch, > YARN-6445-YARN-3926.004.patch > > > As part of the SLS runs on YARN-3926, we discovered a bunch of bottlenecks > around object creation and garbage collection. This JIRA is to apply a fix > for those bottlenecks. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-2113) Add cross-user preemption within CapacityScheduler's leaf-queue
[ https://issues.apache.org/jira/browse/YARN-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2113: -- Attachment: YARN-2113.0009.patch Missed adding a test case for the discussed scenario in earlier patch. Uploading new one with a test case. findbugs warnings are not related to this patch. > Add cross-user preemption within CapacityScheduler's leaf-queue > --- > > Key: YARN-2113 > URL: https://issues.apache.org/jira/browse/YARN-2113 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Vinod Kumar Vavilapalli >Assignee: Sunil G > Attachments: > TestNoIntraQueuePreemptionIfBelowUserLimitAndDifferentPrioritiesWithExtraUsers.txt, > YARN-2113.0001.patch, YARN-2113.0002.patch, YARN-2113.0003.patch, > YARN-2113.0004.patch, YARN-2113.0005.patch, YARN-2113.0006.patch, > YARN-2113.0007.patch, YARN-2113.0008.patch, YARN-2113.0009.patch, > YARN-2113.v0.patch > > > Preemption today only works across queues and moves around resources across > queues per demand and usage. We should also have user-level preemption within > a queue, to balance capacity across users in a predictable manner. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6419) Support to launch native-service deployment from new YARN UI
[ https://issues.apache.org/jira/browse/YARN-6419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-6419: -- Parent Issue: YARN-5079 (was: YARN-3368) > Support to launch native-service deployment from new YARN UI > > > Key: YARN-6419 > URL: https://issues.apache.org/jira/browse/YARN-6419 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-ui-v2 >Reporter: Akhil PB >Assignee: Akhil PB > Attachments: YARN-6419.001.patch, YARN-6419.002.patch, > YARN-6419.003.patch, YARN-6419.004.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2113) Add cross-user preemption within CapacityScheduler's leaf-queue
[
https://issues.apache.org/jira/browse/YARN-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978643#comment-15978643
]
Hadoop QA commented on YARN-2113:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
59s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in trunk has 8 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 25s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 27 new + 168 unchanged - 3 fixed = 195 total (was 171)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 38m
41s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | YARN-2113 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12864481/YARN-2113.0009.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 67ff0c81d672 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / b080338 |
| Default Java | 1.8.0_121 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/15709/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/15709/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15709/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://
[jira] [Commented] (YARN-6428) Queue AM limit is not honored in CS always
[ https://issues.apache.org/jira/browse/YARN-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978691#comment-15978691 ] Bibin A Chundatt commented on YARN-6428: [~sunilg] / [~leftnoteasy] Any comments? > Queue AM limit is not honored in CS always > --- > > Key: YARN-6428 > URL: https://issues.apache.org/jira/browse/YARN-6428 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: YARN-6428.0001.patch > > > Steps to reproduce > > Setup cluster with 40 GB and 40 vcores with 4 Node managers with 10 GB each. > Configure 100% to default queue as capacity and max am limit as 10 % > Minimum scheduler memory and vcore as 512,1 > *Expected* > AM limit 4096 and 4 vores > *Actual* > AM limit 4096+512 and 4+1 vcore -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3839) Quit throwing NMNotYetReadyException
[
https://issues.apache.org/jira/browse/YARN-3839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978705#comment-15978705
]
Jason Lowe commented on YARN-3839:
--
Please see my [earlier
comment|https://issues.apache.org/jira/browse/YARN-3839?focusedCommentId=15975315&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15975315].
The patch is malformed for the {{patch}} command:
{noformat}
$ patch -p1 < YARN-3839.003.patch
patching file
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ContainerManagementProtocol.java
patching file
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/ServerProxy.java
patching file
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
patching file
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
patching file
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManager.java
patching file
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
patching file
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/DummyContainerManager.java
patching file
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java
patching file
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/BaseContainerManagerTest.java
patch: malformed patch at line 369: @@ -75,6 +76,10 @@
{noformat}
The first malformed patch hunk is this one:
{noformat}
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-serv
er-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/container
manager/BaseContainerManagerTest.java b/hadoop-yarn-project/hadoop-yarn/hadoop-y
arn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/s
erver/nodemanager/containermanager/BaseContainerManagerTest.java
index ad0a831..8de5678 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-node
manager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager
/BaseContainerManagerTest.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-node
manager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager
/BaseContainerManagerTest.java
@@ -65,6 +65,7 @@
import org.apache.hadoop.yarn.server.nodemanager.Context;
import org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor;
import org.apache.hadoop.yarn.server.nodemanager.DeletionService;
import org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService;
import org.apache.hadoop.yarn.server.nodemanager.LocalRMInterface;
import org.apache.hadoop.yarn.server.nodemanager.NodeHealthCheckerService;
{noformat}
Note how there are not any lines added/changed/deleted in the hunk. The patch
will need to be regenerated, and you can test it with the {{patch}} command on
a clean view of trunk to know whether the QA bot is going to be able to apply
it before posting.
> Quit throwing NMNotYetReadyException
>
>
> Key: YARN-3839
> URL: https://issues.apache.org/jira/browse/YARN-3839
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: Karthik Kambatla
>Assignee: Manikandan R
> Attachments: YARN-3839.001.patch, YARN-3839.002.patch,
> YARN-3839.003.patch
>
>
> Quit throwing NMNotYetReadyException when NM has not yet registered with the
> RM.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6405) Improve configuring services through REST API
[ https://issues.apache.org/jira/browse/YARN-6405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978743#comment-15978743 ] Billie Rinaldi commented on YARN-6405: -- Yes, that's exactly what we want. Classic Slider apps (HBase, Accumulo, Solr, etc.) write their data to HDFS and don't care where their containers are running. Also, a major part of the usefulness of unique names is that containers will have well-known hostnames, meaning they can assume that a hostname like solr1.solr-app.user.domain will exist. For ZooKeeper, which is a new type of app that writes its data locally, it is even more important to have fixed hostnames because they are used in the zoo.cfg file. But for ZooKeeper it seems like it will be okay if we can configure infinite auto-restart for its containers, since that mimics the old "strict placement" behavior. Is being able to configure that through the API on the todo list? > Improve configuring services through REST API > - > > Key: YARN-6405 > URL: https://issues.apache.org/jira/browse/YARN-6405 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: unique-names-test.patch, > YARN-6405.yarn-native-services.01.patch, > YARN-6405.yarn-native-services.02.patch, > YARN-6405.yarn-native-services.03.patch, > YARN-6405.yarn-native-services.04.patch, > YARN-6405.yarn-native-services.05.patch, > YARN-6405.yarn-native-services.06.patch > > > YARN-4793 defined various ways that user can config services through the > configuration section of the REST API. But, some semantics are not yet > supported in the back-end server (AM). YARN-6255 has done some work, this > jira is to complete this task - support configuring services through REST API -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3839) Quit throwing NMNotYetReadyException
[ https://issues.apache.org/jira/browse/YARN-3839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikandan R updated YARN-3839: --- Attachment: YARN-3839.004.patch > Quit throwing NMNotYetReadyException > > > Key: YARN-3839 > URL: https://issues.apache.org/jira/browse/YARN-3839 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Karthik Kambatla >Assignee: Manikandan R > Attachments: YARN-3839.001.patch, YARN-3839.002.patch, > YARN-3839.003.patch, YARN-3839.004.patch > > > Quit throwing NMNotYetReadyException when NM has not yet registered with the > RM. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3839) Quit throwing NMNotYetReadyException
[ https://issues.apache.org/jira/browse/YARN-3839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978775#comment-15978775 ] Manikandan R commented on YARN-3839: Ah ! got it [~jlowe]. Thanks for your quick support. I did some manual edits in earlier patch to do clean up which has caused this problem. Regenerated the patch, verified using "patch" command in clean working copy and attaching the same. > Quit throwing NMNotYetReadyException > > > Key: YARN-3839 > URL: https://issues.apache.org/jira/browse/YARN-3839 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Karthik Kambatla >Assignee: Manikandan R > Attachments: YARN-3839.001.patch, YARN-3839.002.patch, > YARN-3839.003.patch > > > Quit throwing NMNotYetReadyException when NM has not yet registered with the > RM. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6509) Add a size threshold beyond which yarn logs will require a force option
[ https://issues.apache.org/jira/browse/YARN-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978783#comment-15978783 ] Sean Busbey commented on YARN-6509: --- this should get flagged as an incompatible change since it alters default behavior of the CLI tools. > Add a size threshold beyond which yarn logs will require a force option > --- > > Key: YARN-6509 > URL: https://issues.apache.org/jira/browse/YARN-6509 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Xuan Gong > Attachments: YARN-6509.1.patch > > > An accidental fetch for a long running application can lead to scenario which > the large size of log can fill up a disk. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6335) Port slider's groovy unit tests to yarn native services
[ https://issues.apache.org/jira/browse/YARN-6335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978838#comment-15978838 ] Gour Saha commented on YARN-6335: - Resource normalization is a historic code in Slider. SLIDER-1201 only makes normalization configurable. Note, in its current state it is not just validation, but actual normalization that is happening. > Port slider's groovy unit tests to yarn native services > --- > > Key: YARN-6335 > URL: https://issues.apache.org/jira/browse/YARN-6335 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Fix For: yarn-native-services > > Attachments: YARN-6335-yarn-native-services.001.patch, > YARN-6335-yarn-native-services.002.patch, > YARN-6335-yarn-native-services.003.patch, > YARN-6335-yarn-native-services.004.patch, > YARN-6335-yarn-native-services.005.patch, > YARN-6335-yarn-native-services.006.patch, > YARN-6335-yarn-native-services.007.patch > > > Slider has a lot of useful unit tests implemented in groovy. We could convert > these to Java for YARN native services. This scope of this ticket will > include unit / minicluster tests only and will not include Slider's funtests > which require a running cluster. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3839) Quit throwing NMNotYetReadyException
[
https://issues.apache.org/jira/browse/YARN-3839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978908#comment-15978908
]
Hadoop QA commented on YARN-3839:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
1s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in
trunk has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
48s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 5 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
43s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 43s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 47s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 2 new + 287 unchanged - 16 fixed = 289 total (was 303) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
26s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
21s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
47s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 67m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | YARN-3839 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12864512/YARN-3839.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 55ab0f568095 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.
[jira] [Updated] (YARN-6494) add mounting of HDFS Short-Circuit path for docker containers
[
https://issues.apache.org/jira/browse/YARN-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jaeboo Jeong updated YARN-6494:
---
Attachment: YARN-6494.002.patch
> add mounting of HDFS Short-Circuit path for docker containers
> -
>
> Key: YARN-6494
> URL: https://issues.apache.org/jira/browse/YARN-6494
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Reporter: Jaeboo Jeong
>Assignee: Jaeboo Jeong
> Attachments: YARN-6494.001.patch, YARN-6494.002.patch
>
>
> Currently there is a error message about HDFS short-circuit when docker
> container start.
> {code}
> WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: error
> creating DomainSocket
> java.net.ConnectException: connect(2) error: No such file or directory when
> trying to connect to ‘xxx’
> at org.apache.hadoop.net.unix.DomainSocket.connect0(Native Method)
> at org.apache.hadoop.net.unix.DomainSocket.connect(DomainSocket.java:250)
> at
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory.createSocket(DomainSocketFactory.java:164)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.nextDomainPeer(BlockReaderFactory.java:752)
> ...
> {code}
> if dfs.client.read.shortcircuit is true and dfs.domain.socket.path isn't
> equal “”, we need to mount volume for short-circuit path.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6494) add mounting of HDFS Short-Circuit path for docker containers
[
https://issues.apache.org/jira/browse/YARN-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978913#comment-15978913
]
Jaeboo Jeong commented on YARN-6494:
On branch-3.0.0-alpha2, it compiled well.
But on trunk branch, there is a compile error because of HDFS-11596.
I made a new patch.
> add mounting of HDFS Short-Circuit path for docker containers
> -
>
> Key: YARN-6494
> URL: https://issues.apache.org/jira/browse/YARN-6494
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Reporter: Jaeboo Jeong
>Assignee: Jaeboo Jeong
> Attachments: YARN-6494.001.patch, YARN-6494.002.patch
>
>
> Currently there is a error message about HDFS short-circuit when docker
> container start.
> {code}
> WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: error
> creating DomainSocket
> java.net.ConnectException: connect(2) error: No such file or directory when
> trying to connect to ‘xxx’
> at org.apache.hadoop.net.unix.DomainSocket.connect0(Native Method)
> at org.apache.hadoop.net.unix.DomainSocket.connect(DomainSocket.java:250)
> at
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory.createSocket(DomainSocketFactory.java:164)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.nextDomainPeer(BlockReaderFactory.java:752)
> ...
> {code}
> if dfs.client.read.shortcircuit is true and dfs.domain.socket.path isn't
> equal “”, we need to mount volume for short-circuit path.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6398) Implement a new native-service UI
[ https://issues.apache.org/jira/browse/YARN-6398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akhil PB updated YARN-6398: --- Attachment: YARN-6398.001.patch v1 patch > Implement a new native-service UI > - > > Key: YARN-6398 > URL: https://issues.apache.org/jira/browse/YARN-6398 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-native-services >Reporter: Sunil G >Assignee: Akhil PB > Attachments: YARN-6398.001.patch > > > Create a new and advanced native service UI which can co-exist with the new > Yarn UI. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6417) TestContainerManager.testContainerLaunchAndStop disabled
[ https://issues.apache.org/jira/browse/YARN-6417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978935#comment-15978935 ] Marcell Hegedus commented on YARN-6417: --- TestContainerManager.testContainerLaunchAndStop runs successfully but running its subclass is a bit tricky. [~miklos.szeg...@cloudera.com], could you advise what is the proper way of running TestContainerManagerWithLCE? > TestContainerManager.testContainerLaunchAndStop disabled > > > Key: YARN-6417 > URL: https://issues.apache.org/jira/browse/YARN-6417 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Miklos Szegedi >Priority: Minor > Labels: newbie > > TestContainerManager.testContainerLaunchAndStop was disabled in YARN-1897 but > it is passing. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-679) add an entry point that can start any Yarn service
[ https://issues.apache.org/jira/browse/YARN-679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-679: Attachment: YARN-679-013.patch Finally had time to sit down & go through all the review. Thanks dan! I have taken on *all* the javadoc suggestions, even the one about Signalled being misspelt, when that was clearly just an EN_US locale error. Easier to switch rather than to argue the details. Code changes done, plus some other tweaks * moved some public static methods to protected. * fixed a test which was failing with the different exception text from the new Stax config parser. * hit the "optimise imports" button in the IDE. This patch only really makes sense when you implement a service which uses it. This is essentially what we've been using in Slider; once it is in I will use it for some of the S3A work as well as making it a standalone registry. This patch makes it trivial to use any Service either as a service or a standalone entry point, see > add an entry point that can start any Yarn service > -- > > Key: YARN-679 > URL: https://issues.apache.org/jira/browse/YARN-679 > Project: Hadoop YARN > Issue Type: New Feature > Components: api >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: org.apache.hadoop.servic...mon 3.0.0-SNAPSHOT API).pdf, > YARN-679-001.patch, YARN-679-002.patch, YARN-679-002.patch, > YARN-679-003.patch, YARN-679-004.patch, YARN-679-005.patch, > YARN-679-006.patch, YARN-679-007.patch, YARN-679-008.patch, > YARN-679-009.patch, YARN-679-010.patch, YARN-679-011.patch, YARN-679-013.patch > > Time Spent: 72h > Remaining Estimate: 0h > > There's no need to write separate .main classes for every Yarn service, given > that the startup mechanism should be identical: create, init, start, wait for > stopped -with an interrupt handler to trigger a clean shutdown on a control-c > interrupt. > Provide one that takes any classname, and a list of config files/options -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5949) Add pluggable configuration policy interface as a component of MutableCSConfigurationProvider
[ https://issues.apache.org/jira/browse/YARN-5949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978972#comment-15978972 ] Xuan Gong commented on YARN-5949: - Thanks for the patch. [~jhung] Overall looks good. In the same patch, we could use this new QueueAdminConfigurationMutationPolicy in the REST API for changing YARN scheduler configurations ? So, we could have the end-to-end story for the MutableCSConfigurationProvider ? > Add pluggable configuration policy interface as a component of > MutableCSConfigurationProvider > - > > Key: YARN-5949 > URL: https://issues.apache.org/jira/browse/YARN-5949 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-5949-YARN-5734.001.patch > > > This will allow different policies to customize how/if configuration changes > should be applied (for example, a policy might restrict whether a > configuration change by a certain user is allowed). This will be enforced by > the MutableCSConfigurationProvider. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6457) Allow custom SSL configuration to be supplied in WebApps
[ https://issues.apache.org/jira/browse/YARN-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978998#comment-15978998 ] Haibo Chen commented on YARN-6457: -- The master branch is 'trunk', so you can just create a PR against it. The traditional workflow can be found at https://wiki.apache.org/hadoop/HowToContribute > Allow custom SSL configuration to be supplied in WebApps > > > Key: YARN-6457 > URL: https://issues.apache.org/jira/browse/YARN-6457 > Project: Hadoop YARN > Issue Type: Improvement > Components: webapp, yarn >Reporter: Sanjay M Pujare > Original Estimate: 96h > Remaining Estimate: 96h > > Currently a custom SSL store cannot be passed on to WebApps which forces the > embedded web-server to use the default keystore set up in ssl-server.xml for > the whole Hadoop cluster. There are cases where the Hadoop app needs to use > its own/custom keystore. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6509) Add a size threshold beyond which yarn logs will require a force option
[ https://issues.apache.org/jira/browse/YARN-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-6509: Fix Version/s: 2.9.0 > Add a size threshold beyond which yarn logs will require a force option > --- > > Key: YARN-6509 > URL: https://issues.apache.org/jira/browse/YARN-6509 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Xuan Gong > Fix For: 2.9.0 > > Attachments: YARN-6509.1.patch > > > An accidental fetch for a long running application can lead to scenario which > the large size of log can fill up a disk. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6509) Add a size threshold beyond which yarn logs will require a force option
[ https://issues.apache.org/jira/browse/YARN-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979025#comment-15979025 ] Xuan Gong commented on YARN-6509: - [~busbey] bq. this should get flagged as an incompatible change since it alters default behavior of the CLI tools. Thanks for the comment. I think that it will be fine. All the related changes will be in 2.9 including the improvement which add "--size" option. > Add a size threshold beyond which yarn logs will require a force option > --- > > Key: YARN-6509 > URL: https://issues.apache.org/jira/browse/YARN-6509 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Xuan Gong > Fix For: 2.9.0 > > Attachments: YARN-6509.1.patch > > > An accidental fetch for a long running application can lead to scenario which > the large size of log can fill up a disk. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6509) Add a size threshold beyond which yarn logs will require a force option
[ https://issues.apache.org/jira/browse/YARN-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979027#comment-15979027 ] Xuan Gong commented on YARN-6509: - findbug issues are not related to this fix. And we could ignore the checkstyle issues > Add a size threshold beyond which yarn logs will require a force option > --- > > Key: YARN-6509 > URL: https://issues.apache.org/jira/browse/YARN-6509 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Xuan Gong > Fix For: 2.9.0 > > Attachments: YARN-6509.1.patch > > > An accidental fetch for a long running application can lead to scenario which > the large size of log can fill up a disk. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6509) Add a size threshold beyond which yarn logs will require a force option
[
https://issues.apache.org/jira/browse/YARN-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979037#comment-15979037
]
Sean Busbey commented on YARN-6509:
---
{quote}Thanks for the comment. I think that it will be fine. All the related
changes will be in 2.9 including the improvement which add "--size" option.
{quote}
I agree it's fine, but it should be flagged so that we can be sure it's called
out in release notes appropriately.
{quote}And we could ignore the checkstyle issues{quote}
The line length ones should be corrected. I agree that the MethodLength ones
can stay, since they're not really new problems.
> Add a size threshold beyond which yarn logs will require a force option
> ---
>
> Key: YARN-6509
> URL: https://issues.apache.org/jira/browse/YARN-6509
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Xuan Gong
> Fix For: 2.9.0
>
> Attachments: YARN-6509.1.patch
>
>
> An accidental fetch for a long running application can lead to scenario which
> the large size of log can fill up a disk.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6384) Add configuratin to set max cpu usage when strict-resource-usage is false with cgroups
[ https://issues.apache.org/jira/browse/YARN-6384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979067#comment-15979067 ] Miklos Szegedi commented on YARN-6384: -- Sure. > Add configuratin to set max cpu usage when strict-resource-usage is false > with cgroups > -- > > Key: YARN-6384 > URL: https://issues.apache.org/jira/browse/YARN-6384 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: dengkai > > When using cgroups on yarn, if > yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage is > false, user may get very more cpu time than expected based on the vcores. > There should be a upper limit even resource-usage is not strict, just like a > percentage which user can get more than promised by vcores. I think it's > important in a shared cluster. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6500) Do not mount inaccessible cgroups directories in CgroupsLCEResourcesHandler
[ https://issues.apache.org/jira/browse/YARN-6500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979084#comment-15979084 ] Haibo Chen commented on YARN-6500: -- The findbugs warnings seems unrelated. +1. Will wait until next Monday to commit it in case someone has more comments. > Do not mount inaccessible cgroups directories in CgroupsLCEResourcesHandler > --- > > Key: YARN-6500 > URL: https://issues.apache.org/jira/browse/YARN-6500 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha2 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: YARN-6500.000.patch, YARN-6500.001.patch, > YARN-6500.002.patch, YARN-6500.003.patch > > > Port YARN-6433 findControllerInMtab change from CGroupsHandlerImpl to > CgroupsLCEResourcesHandler -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6500) Do not mount inaccessible cgroups directories in CgroupsLCEResourcesHandler
[ https://issues.apache.org/jira/browse/YARN-6500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6500: - Component/s: nodemanager > Do not mount inaccessible cgroups directories in CgroupsLCEResourcesHandler > --- > > Key: YARN-6500 > URL: https://issues.apache.org/jira/browse/YARN-6500 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha2 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: YARN-6500.000.patch, YARN-6500.001.patch, > YARN-6500.002.patch, YARN-6500.003.patch > > > Port YARN-6433 findControllerInMtab change from CGroupsHandlerImpl to > CgroupsLCEResourcesHandler -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6500) Do not mount inaccessible cgroups directories in CgroupsLCEResourcesHandler
[ https://issues.apache.org/jira/browse/YARN-6500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6500: - Affects Version/s: 3.0.0-alpha2 > Do not mount inaccessible cgroups directories in CgroupsLCEResourcesHandler > --- > > Key: YARN-6500 > URL: https://issues.apache.org/jira/browse/YARN-6500 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha2 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: YARN-6500.000.patch, YARN-6500.001.patch, > YARN-6500.002.patch, YARN-6500.003.patch > > > Port YARN-6433 findControllerInMtab change from CGroupsHandlerImpl to > CgroupsLCEResourcesHandler -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-679) add an entry point that can start any Yarn service
[
https://issues.apache.org/jira/browse/YARN-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979092#comment-15979092
]
Hadoop QA commented on YARN-679:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 22 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
29s{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 17
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 14m 26s{color}
| {color:red} root generated 8 new + 777 unchanged - 0 fixed = 785 total (was
777) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 45 new + 106 unchanged - 30 fixed = 151 total (was 136) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 67 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
22s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 67m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | YARN-679 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12864532/YARN-679-013.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ed761c9305ea 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / b080338 |
| Default Java | 1.8.0_121 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/15711/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
|
| javac |
https://builds.apache.org/job/PreCommit-YARN-Build/15711/artifact/patchprocess/diff-compile-javac-root.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/15711/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/15711/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15711/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-c
[jira] [Commented] (YARN-6500) Do not mount inaccessible cgroups directories in CgroupsLCEResourcesHandler
[ https://issues.apache.org/jira/browse/YARN-6500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979100#comment-15979100 ] Daniel Templeton commented on YARN-6500: LGTM2 > Do not mount inaccessible cgroups directories in CgroupsLCEResourcesHandler > --- > > Key: YARN-6500 > URL: https://issues.apache.org/jira/browse/YARN-6500 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha2 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: YARN-6500.000.patch, YARN-6500.001.patch, > YARN-6500.002.patch, YARN-6500.003.patch > > > Port YARN-6433 findControllerInMtab change from CGroupsHandlerImpl to > CgroupsLCEResourcesHandler -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6500) Do not mount inaccessible cgroups directories in CgroupsLCEResourcesHandler
[ https://issues.apache.org/jira/browse/YARN-6500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979116#comment-15979116 ] Robert Kanter commented on YARN-6500: - +1 > Do not mount inaccessible cgroups directories in CgroupsLCEResourcesHandler > --- > > Key: YARN-6500 > URL: https://issues.apache.org/jira/browse/YARN-6500 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha2 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: YARN-6500.000.patch, YARN-6500.001.patch, > YARN-6500.002.patch, YARN-6500.003.patch > > > Port YARN-6433 findControllerInMtab change from CGroupsHandlerImpl to > CgroupsLCEResourcesHandler -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6510) Fix warning - procfs stat file is not in the expected format: YARN-3344 is not enough
Wilfred Spiegelenburg created YARN-6510:
---
Summary: Fix warning - procfs stat file is not in the expected
format: YARN-3344 is not enough
Key: YARN-6510
URL: https://issues.apache.org/jira/browse/YARN-6510
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 3.0.0-alpha2, 2.8.0
Reporter: Wilfred Spiegelenburg
Assignee: Wilfred Spiegelenburg
Even with the fix for YARN-3344 we still have issues with the procfs format.
This is the case that is causing issues:
{code}
[user@nm1 ~]$ cat /proc/2406/stat
2406 (ib_fmr(mlx4_0)) S 2 0 0 0 -1 2149613632 0 0 0 0 166 126908 0 0 20 0 1 0
4284 0 0 18446744073709551615 0 0 0 0 0 0 0 2147483647 0 18446744073709551615 0
0 17 6 0 0 0 0 0
{code}
We do not handle the parenthesis in the name which causes the pattern matching
to fail
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3666) Federation Intercepting and propagating AM-RM communications
[ https://issues.apache.org/jira/browse/YARN-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-3666: --- Description: In order, to support transparent "spanning" of jobs across sub-clusters, all AM-RM communications are proxied (via YARN-2884). This JIRA tracks federation-specific mechanisms that decide how to "split/broadcast" requests to the RMs and "merge" answers to the AM. This the part one jira, which sets up the basic structure, without secondary subclusters. All requests are forwarded to home subcluster. was: In order, to support transparent "spanning" of jobs across sub-clusters, all AM-RM communications are proxied (via YARN-2884). This JIRA tracks federation-specific mechanisms that decide how to "split/broadcast" requests to the RMs and "merge" answers to the AM. > Federation Intercepting and propagating AM-RM communications > > > Key: YARN-3666 > URL: https://issues.apache.org/jira/browse/YARN-3666 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Kishore Chaliparambil >Assignee: Botong Huang > Attachments: YARN-3666-YARN-2915.v1.patch, > YARN-3666-YARN-2915.v2.patch, YARN-3666-YARN-2915.v3.patch, > YARN-3666-YARN-2915.v4.patch, YARN-3666-YARN-2915.v5.patch > > > In order, to support transparent "spanning" of jobs across sub-clusters, all > AM-RM communications are proxied (via YARN-2884). > This JIRA tracks federation-specific mechanisms that decide how to > "split/broadcast" requests to the RMs and "merge" answers to > the AM. > This the part one jira, which sets up the basic structure, without secondary > subclusters. All requests are forwarded to home subcluster. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3666) Federation Intercepting and propagating AM-RM communications (part one: home RM only)
[ https://issues.apache.org/jira/browse/YARN-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-3666: --- Summary: Federation Intercepting and propagating AM-RM communications (part one: home RM only) (was: Federation Intercepting and propagating AM-RM communications) > Federation Intercepting and propagating AM-RM communications (part one: home > RM only) > - > > Key: YARN-3666 > URL: https://issues.apache.org/jira/browse/YARN-3666 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Kishore Chaliparambil >Assignee: Botong Huang > Attachments: YARN-3666-YARN-2915.v1.patch, > YARN-3666-YARN-2915.v2.patch, YARN-3666-YARN-2915.v3.patch, > YARN-3666-YARN-2915.v4.patch, YARN-3666-YARN-2915.v5.patch > > > In order, to support transparent "spanning" of jobs across sub-clusters, all > AM-RM communications are proxied (via YARN-2884). > This JIRA tracks federation-specific mechanisms that decide how to > "split/broadcast" requests to the RMs and "merge" answers to > the AM. > This the part one jira, which sets up the basic structure, without secondary > subclusters. All requests are forwarded to home subcluster. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3666) Federation Intercepting and propagating AM-RM communications (part one: home RM only)
[ https://issues.apache.org/jira/browse/YARN-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-3666: --- Description: In order, to support transparent "spanning" of jobs across sub-clusters, all AM-RM communications are proxied (via YARN-2884). This JIRA tracks federation-specific mechanisms that decide how to "split/broadcast" requests to the RMs and "merge" answers to the AM. This the part one jira, which sets up the basic structure, without secondary subclusters. All requests are forwarded to home subcluster. Part two is in Yarn-6511 was: In order, to support transparent "spanning" of jobs across sub-clusters, all AM-RM communications are proxied (via YARN-2884). This JIRA tracks federation-specific mechanisms that decide how to "split/broadcast" requests to the RMs and "merge" answers to the AM. This the part one jira, which sets up the basic structure, without secondary subclusters. All requests are forwarded to home subcluster. > Federation Intercepting and propagating AM-RM communications (part one: home > RM only) > - > > Key: YARN-3666 > URL: https://issues.apache.org/jira/browse/YARN-3666 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Kishore Chaliparambil >Assignee: Botong Huang > Attachments: YARN-3666-YARN-2915.v1.patch, > YARN-3666-YARN-2915.v2.patch, YARN-3666-YARN-2915.v3.patch, > YARN-3666-YARN-2915.v4.patch, YARN-3666-YARN-2915.v5.patch > > > In order, to support transparent "spanning" of jobs across sub-clusters, all > AM-RM communications are proxied (via YARN-2884). > This JIRA tracks federation-specific mechanisms that decide how to > "split/broadcast" requests to the RMs and "merge" answers to > the AM. > This the part one jira, which sets up the basic structure, without secondary > subclusters. All requests are forwarded to home subcluster. > Part two is in Yarn-6511 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6511) Federation Intercepting and propagating AM-RM communications (part two: secondary subclusters added)
Botong Huang created YARN-6511: -- Summary: Federation Intercepting and propagating AM-RM communications (part two: secondary subclusters added) Key: YARN-6511 URL: https://issues.apache.org/jira/browse/YARN-6511 Project: Hadoop YARN Issue Type: Task Reporter: Botong Huang Assignee: Botong Huang -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6511) Federation Intercepting and propagating AM-RM communications (part two: secondary subclusters added)
[ https://issues.apache.org/jira/browse/YARN-6511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-6511: --- Description: In order to support transparent "spanning" of jobs across sub-clusters, all AM-RM communications are proxied (via YARN-2884). This JIRA tracks federation-specific mechanisms that decide how to "split/broadcast" requests to the RMs and "merge" answers to the AM. This the part two jira, which adds secondary subclusters and do full split-merge for requests. Part one is in YARN-3666 > Federation Intercepting and propagating AM-RM communications (part two: > secondary subclusters added) > > > Key: YARN-6511 > URL: https://issues.apache.org/jira/browse/YARN-6511 > Project: Hadoop YARN > Issue Type: Task >Reporter: Botong Huang >Assignee: Botong Huang > > In order to support transparent "spanning" of jobs across sub-clusters, all > AM-RM communications are proxied (via YARN-2884). > This JIRA tracks federation-specific mechanisms that decide how to > "split/broadcast" requests to the RMs and "merge" answers to > the AM. > This the part two jira, which adds secondary subclusters and do full > split-merge for requests. Part one is in YARN-3666 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3666) Federation Intercepting and propagating AM-RM communications (part one: home RM only)
[ https://issues.apache.org/jira/browse/YARN-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-3666: --- Description: In order, to support transparent "spanning" of jobs across sub-clusters, all AM-RM communications are proxied (via YARN-2884). This JIRA tracks federation-specific mechanisms that decide how to "split/broadcast" requests to the RMs and "merge" answers to the AM. This the part one jira, which sets up the basic structure, without secondary subclusters. All requests are forwarded to home subcluster. Part two is in YARN-6511 was: In order, to support transparent "spanning" of jobs across sub-clusters, all AM-RM communications are proxied (via YARN-2884). This JIRA tracks federation-specific mechanisms that decide how to "split/broadcast" requests to the RMs and "merge" answers to the AM. This the part one jira, which sets up the basic structure, without secondary subclusters. All requests are forwarded to home subcluster. Part two is in Yarn-6511 > Federation Intercepting and propagating AM-RM communications (part one: home > RM only) > - > > Key: YARN-3666 > URL: https://issues.apache.org/jira/browse/YARN-3666 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Kishore Chaliparambil >Assignee: Botong Huang > Attachments: YARN-3666-YARN-2915.v1.patch, > YARN-3666-YARN-2915.v2.patch, YARN-3666-YARN-2915.v3.patch, > YARN-3666-YARN-2915.v4.patch, YARN-3666-YARN-2915.v5.patch > > > In order, to support transparent "spanning" of jobs across sub-clusters, all > AM-RM communications are proxied (via YARN-2884). > This JIRA tracks federation-specific mechanisms that decide how to > "split/broadcast" requests to the RMs and "merge" answers to > the AM. > This the part one jira, which sets up the basic structure, without secondary > subclusters. All requests are forwarded to home subcluster. Part two is in > YARN-6511 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6511) Federation Intercepting and propagating AM-RM communications (part two: secondary subclusters added)
[ https://issues.apache.org/jira/browse/YARN-6511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-6511: --- Issue Type: Sub-task (was: Task) Parent: YARN-2915 > Federation Intercepting and propagating AM-RM communications (part two: > secondary subclusters added) > > > Key: YARN-6511 > URL: https://issues.apache.org/jira/browse/YARN-6511 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: Botong Huang > > In order to support transparent "spanning" of jobs across sub-clusters, all > AM-RM communications are proxied (via YARN-2884). > This JIRA tracks federation-specific mechanisms that decide how to > "split/broadcast" requests to the RMs and "merge" answers to > the AM. > This the part two jira, which adds secondary subclusters and do full > split-merge for requests. Part one is in YARN-3666 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-574) PrivateLocalizer does not support parallel resource download via ContainerLocalizer
[
https://issues.apache.org/jira/browse/YARN-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979326#comment-15979326
]
Sangjin Lee commented on YARN-574:
--
Thanks for your contribution [~ajithshetty].
Regarding the use of {{AtomicInteger}}, why not use {{Semaphore}} for this? The
semantics we're demanding is really that of a semaphore. We could also
eliminate it altogether with Jason's suggestion but if we're going to use it,
using a real counting semaphore is clearer.
Could you please update the patch for Jason's feedback and mine here too?
Thanks!
> PrivateLocalizer does not support parallel resource download via
> ContainerLocalizer
> ---
>
> Key: YARN-574
> URL: https://issues.apache.org/jira/browse/YARN-574
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.6.0, 2.8.0, 2.7.1
>Reporter: Omkar Vinit Joshi
>Assignee: Ajith S
> Attachments: YARN-574.03.patch, YARN-574.04.patch, YARN-574.05.patch,
> YARN-574.1.patch, YARN-574.2.patch
>
>
> At present private resources will be downloaded in parallel only if multiple
> containers request the same resource. However otherwise it will be serial.
> The protocol between PrivateLocalizer and ContainerLocalizer supports
> multiple downloads however it is not used and only one resource is sent for
> downloading at a time.
> I think we can increase / assure parallelism (even for single container
> requesting resource) for private/application resources by making multiple
> downloads per ContainerLocalizer.
> Total Parallelism before
> = number of threads allotted for PublicLocalizer [public resource] + number
> of containers[private and application resource]
> Total Parallelism after
> = number of threads allotted for PublicLocalizer [public resource] + number
> of containers * max downloads per container [private and application resource]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-574) PrivateLocalizer does not support parallel resource download via ContainerLocalizer
[
https://issues.apache.org/jira/browse/YARN-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979330#comment-15979330
]
Hadoop QA commented on YARN-574:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} YARN-574 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-574 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12849264/YARN-574.05.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15712/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> PrivateLocalizer does not support parallel resource download via
> ContainerLocalizer
> ---
>
> Key: YARN-574
> URL: https://issues.apache.org/jira/browse/YARN-574
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.6.0, 2.8.0, 2.7.1
>Reporter: Omkar Vinit Joshi
>Assignee: Ajith S
> Attachments: YARN-574.03.patch, YARN-574.04.patch, YARN-574.05.patch,
> YARN-574.1.patch, YARN-574.2.patch
>
>
> At present private resources will be downloaded in parallel only if multiple
> containers request the same resource. However otherwise it will be serial.
> The protocol between PrivateLocalizer and ContainerLocalizer supports
> multiple downloads however it is not used and only one resource is sent for
> downloading at a time.
> I think we can increase / assure parallelism (even for single container
> requesting resource) for private/application resources by making multiple
> downloads per ContainerLocalizer.
> Total Parallelism before
> = number of threads allotted for PublicLocalizer [public resource] + number
> of containers[private and application resource]
> Total Parallelism after
> = number of threads allotted for PublicLocalizer [public resource] + number
> of containers * max downloads per container [private and application resource]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6131) FairScheduler: Lower update interval for faster tests
[ https://issues.apache.org/jira/browse/YARN-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-6131: -- Fix Version/s: 3.0.0-alpha3 > FairScheduler: Lower update interval for faster tests > - > > Key: YARN-6131 > URL: https://issues.apache.org/jira/browse/YARN-6131 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: yarn-6131-1.patch > > > FairScheduler uses an update interval of 500ms by default. Tests could run > faster if we lower this value to 10ms. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6368) Decommissioning an NM results in a -1 exit code
[
https://issues.apache.org/jira/browse/YARN-6368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated YARN-6368:
--
Fix Version/s: 3.0.0-alpha3
2.9.0
> Decommissioning an NM results in a -1 exit code
> ---
>
> Key: YARN-6368
> URL: https://issues.apache.org/jira/browse/YARN-6368
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-6368.000.patch, YARN-6368.001.patch,
> YARN-6368.002.patch, YARN-6368.003.patch, YARN-6368.004.patch
>
>
> In NodeManager.java we should exit normally in case the RM shuts down the
> node:
> {code}
> } finally {
> if (shouldExitOnShutdownEvent
> && !ShutdownHookManager.get().isShutdownInProgress()) {
> ExitUtil.terminate(-1);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6061) Add an UncaughtExceptionHandler for critical threads in RM
[ https://issues.apache.org/jira/browse/YARN-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-6061: -- Fix Version/s: 3.0.0-alpha3 > Add an UncaughtExceptionHandler for critical threads in RM > -- > > Key: YARN-6061 > URL: https://issues.apache.org/jira/browse/YARN-6061 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Yufei Gu >Assignee: Yufei Gu > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: YARN-6061.001.patch, YARN-6061.002.patch, > YARN-6061.003.patch, YARN-6061.004.patch, YARN-6061.005.patch, > YARN-6061.006.patch, YARN-6061.007.patch, YARN-6061.008.patch, > YARN-6061.009.patch, YARN-6061.010.patch > > > There are several threads in fair scheduler. The thread will quit when there > is a runtime exception inside it. We should bring down the RM when that > happens. Otherwise, there may be some weird behavior in RM. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6448) Continuous scheduling thread crashes while sorting nodes
[
https://issues.apache.org/jira/browse/YARN-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated YARN-6448:
--
Fix Version/s: 3.0.0-alpha3
> Continuous scheduling thread crashes while sorting nodes
>
>
> Key: YARN-6448
> URL: https://issues.apache.org/jira/browse/YARN-6448
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-6448.001.patch, YARN-6448.002.patch,
> YARN-6448.003.patch, YARN-6448.004.patch
>
>
> YARN-4719 remove the lock in continuous scheduling while sorting nodes. It
> breaks the order in comparison if nodes changes while sorting.
> {code}
> 2017-04-04 23:42:26,123 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.RMCriticalThreadUncaughtExceptionHandler:
> Critical thread FairSchedulerContinuousScheduling crashed!
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:175)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker.sortedNodeList(ClusterNodeTracker.java:306)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.continuousSchedulingAttempt(FairScheduler.java:884)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$ContinuousSchedulingThread.run(FairScheduler.java:316)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6448) Continuous scheduling thread crashes while sorting nodes
[
https://issues.apache.org/jira/browse/YARN-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979429#comment-15979429
]
Andrew Wang commented on YARN-6448:
---
Please remember to set a fix version when committing to trunk, thx!
> Continuous scheduling thread crashes while sorting nodes
>
>
> Key: YARN-6448
> URL: https://issues.apache.org/jira/browse/YARN-6448
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-6448.001.patch, YARN-6448.002.patch,
> YARN-6448.003.patch, YARN-6448.004.patch
>
>
> YARN-4719 remove the lock in continuous scheduling while sorting nodes. It
> breaks the order in comparison if nodes changes while sorting.
> {code}
> 2017-04-04 23:42:26,123 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.RMCriticalThreadUncaughtExceptionHandler:
> Critical thread FairSchedulerContinuousScheduling crashed!
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:175)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker.sortedNodeList(ClusterNodeTracker.java:306)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.continuousSchedulingAttempt(FairScheduler.java:884)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$ContinuousSchedulingThread.run(FairScheduler.java:316)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6436) TestSchedulingPolicy#testParseSchedulingPolicy timeout is too low
[ https://issues.apache.org/jira/browse/YARN-6436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-6436: -- Fix Version/s: 3.0.0-alpha3 > TestSchedulingPolicy#testParseSchedulingPolicy timeout is too low > - > > Key: YARN-6436 > URL: https://issues.apache.org/jira/browse/YARN-6436 > Project: Hadoop YARN > Issue Type: Bug > Components: test >Reporter: Jason Lowe >Assignee: Eric Badger > Fix For: 2.9.0, 2.8.1, 3.0.0-alpha3 > > Attachments: YARN-6436.001.patch, YARN-6436.002.patch > > > The timeout for testParseSchedulingPolicy is only one second. An I/O hiccup > on a VM can make this test fail for the wrong reasons. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6184) Introduce loading icon in each page of new YARN UI
[ https://issues.apache.org/jira/browse/YARN-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-6184: -- Fix Version/s: 3.0.0-alpha3 > Introduce loading icon in each page of new YARN UI > -- > > Key: YARN-6184 > URL: https://issues.apache.org/jira/browse/YARN-6184 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Akhil PB >Assignee: Akhil PB > Fix For: 3.0.0-alpha3 > > Attachments: YARN-6184.001.patch, YARN-6184.002.patch, > YARN-6184.003.patch, YARN-6184.004.patch, YARN-6184.005.patch, > YARN-6184.006.patch > > > Add loading icon in each page in new YARN-UI. This will help in cases where > we download large data to client side. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6183) Few missing informations in Application and Application Attempt pages for new YARN UI.
[ https://issues.apache.org/jira/browse/YARN-6183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-6183: -- Fix Version/s: 3.0.0-alpha3 > Few missing informations in Application and Application Attempt pages for new > YARN UI. > -- > > Key: YARN-6183 > URL: https://issues.apache.org/jira/browse/YARN-6183 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Akhil PB >Assignee: Akhil PB > Fix For: 3.0.0-alpha3 > > Attachments: YARN-6183.001.patch, YARN-6183.002.patch, > YARN-6183.003.patch, YARN-6183.004.patch > > > Fix missing information in application and app-attempt pages in new YARN-UI. > In Basic Info table, priority and IsUnmanagedAM fields are missing for some > applications. > In App Master panel, change label Expr to Expression. > In app-attempt page, logUrl and NM web UI needs to be opened in new tab. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6309) Fair scheduler docs should have the queue and queuePlacementPolicy elements listed in bold so that they're easier to see
[
https://issues.apache.org/jira/browse/YARN-6309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated YARN-6309:
--
Fix Version/s: 3.0.0-alpha3
2.9.0
> Fair scheduler docs should have the queue and queuePlacementPolicy elements
> listed in bold so that they're easier to see
>
>
> Key: YARN-6309
> URL: https://issues.apache.org/jira/browse/YARN-6309
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha2
>Reporter: Daniel Templeton
>Assignee: esmaeil mirzaee
>Priority: Minor
> Labels: docs, newbie
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN_6309.001.patch, YARN-6309.patch
>
>
> Under {{Allocation file format : Queue elements}}, all of the element names
> should be bold, e.g. {{minResources}}, {{maxResources}}, etc. Same for
> {{Allocation file format : A queuePlacementPolicy element}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6231) FairSchedulerTestBase helper methods should call scheduler.update to avoid flakiness
[ https://issues.apache.org/jira/browse/YARN-6231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-6231: -- Fix Version/s: 3.0.0-alpha3 > FairSchedulerTestBase helper methods should call scheduler.update to avoid > flakiness > > > Key: YARN-6231 > URL: https://issues.apache.org/jira/browse/YARN-6231 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Arun Suresh >Assignee: Karthik Kambatla > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: YARN-6231.001.patch, YARN-6231.002.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6164) Expose Queue Configurations per Node Label through YARN client api
[ https://issues.apache.org/jira/browse/YARN-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-6164: -- Fix Version/s: 3.0.0-alpha3 > Expose Queue Configurations per Node Label through YARN client api > -- > > Key: YARN-6164 > URL: https://issues.apache.org/jira/browse/YARN-6164 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.7.2 >Reporter: Benson Qiu >Assignee: Benson Qiu > Fix For: 3.0.0-alpha3 > > Attachments: YARN-6164.001.patch, YARN-6164.002.patch, > YARN-6164.003.patch, YARN-6164.004.patch, YARN-6164.005.patch, > YARN-6164.006.patch, YARN-6164.007.patch, YARN-6164.008.patch, > YARN-6164-branch-2.001.patch > > > `yarn.scheduler.capacity.maximum-am-resource-percent` is exposed through the > [Cluster Scheduler > API|http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Scheduler_API], > but not through > [YarnClient|https://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/client/api/YarnClient.html]. > Since YarnClient and RM REST APIs depend on different ports (8032 vs 8088 by > default), it would be nice to expose `maximum-am-resource-percent` in > YarnClient as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6196) Improve Resource Donut chart with better label in Node page of new YARN UI
[ https://issues.apache.org/jira/browse/YARN-6196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-6196: -- Fix Version/s: 3.0.0-alpha3 > Improve Resource Donut chart with better label in Node page of new YARN UI > -- > > Key: YARN-6196 > URL: https://issues.apache.org/jira/browse/YARN-6196 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Akhil PB >Assignee: Akhil PB > Fix For: 3.0.0-alpha3 > > Attachments: YARN-6196.001.patch > > > In nodes page: > # Change 'Nodes Table' label to 'Information' > # Show Health Report as N/A if not available > # When there are 0 nodes in cluster, nodes page breaks. > In node page: > # Node Health Report missing > # NodeManager Start Time shows Invalid Date > # Reverse colors in the 'Resouce - Memory' and 'Resource - VCores' donut > charts > # Convert Resource Memory into GB/TB > # Diagnostics is empty in Container Information -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6106) Document FairScheduler 'allowPreemptionFrom' queue property
[ https://issues.apache.org/jira/browse/YARN-6106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-6106: -- Fix Version/s: 3.0.0-alpha3 > Document FairScheduler 'allowPreemptionFrom' queue property > --- > > Key: YARN-6106 > URL: https://issues.apache.org/jira/browse/YARN-6106 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Reporter: Yufei Gu >Assignee: Yufei Gu >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: YARN-6106.001.patch, YARN-6106.002.patch > > > How 'allowPreemptionFrom' works is discussed in YARN-5831. Basically, if the > parent queue is non-preemptable, the children must be non-preemptable. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6417) TestContainerManager.testContainerLaunchAndStop disabled
[
https://issues.apache.org/jira/browse/YARN-6417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979452#comment-15979452
]
Miklos Szegedi commented on YARN-6417:
--
You can build with native profile and run the test specifying the path to the
container-executor:
{code}
mvn package -Pdist -Pnative -DskipTests -Dmaven.javadoc.skip=true
mvn test
-Dyarn.nodemanager.linux-container-executor.path=./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0-alpha3-SNAPSHOT/bin/container-executor
-Dtest=TestContainerManagerWithLCE
{code}
> TestContainerManager.testContainerLaunchAndStop disabled
>
>
> Key: YARN-6417
> URL: https://issues.apache.org/jira/browse/YARN-6417
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: Miklos Szegedi
>Priority: Minor
> Labels: newbie
>
> TestContainerManager.testContainerLaunchAndStop was disabled in YARN-1897 but
> it is passing.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6126) Obtaining app logs for Running application fails with "Unable to parse json from webservice. Error:"
[ https://issues.apache.org/jira/browse/YARN-6126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-6126: -- Fix Version/s: 3.0.0-alpha3 > Obtaining app logs for Running application fails with "Unable to parse json > from webservice. Error:" > > > Key: YARN-6126 > URL: https://issues.apache.org/jira/browse/YARN-6126 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sumana Sathish >Assignee: Xuan Gong > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: YARN-6126.branch-2.1.patch, YARN-6126.trunk.1.patch > > > After YARN-6099, we can output aggregated log files info as well as local log > files info if we enabled partial log aggregation. This breaks the original > logic in logsCLI -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6264) AM not launched when a single vcore is available on the cluster
[
https://issues.apache.org/jira/browse/YARN-6264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated YARN-6264:
--
Fix Version/s: 3.0.0-alpha3
> AM not launched when a single vcore is available on the cluster
> ---
>
> Key: YARN-6264
> URL: https://issues.apache.org/jira/browse/YARN-6264
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-6264.001.patch, YARN-6264.002.patch,
> YARN-6264.003.patch, YARN-6264.004.patch, YARN-6264.005.patch,
> YARN-6264.006.patch
>
>
> In method {{canRunAppAM()}}, we will get a zero vcore if there is only one
> vcores, and the {{maxAMShare}} is between 0 and 1 because
> {{Resources#multiply}} round down double to a integer by default. This
> potentially happens frequently when assigning a container to AM just after
> preemption happens since it is more likely there is only one vcore on the
> cluster just after preemption.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6193) FairScheduler might not trigger preemption when using DRF
[
https://issues.apache.org/jira/browse/YARN-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated YARN-6193:
--
Fix Version/s: 3.0.0-alpha3
> FairScheduler might not trigger preemption when using DRF
> -
>
> Key: YARN-6193
> URL: https://issues.apache.org/jira/browse/YARN-6193
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-6193.000.patch, YARN-6193.001.patch,
> YARN-6193.002.patch, YARN-6193.003.patch
>
>
> {{FSAppAttempt#canContainerBePreempted}} verifies preempting a container
> doesn't lead to new starvation ({{Resources.fitsIn}}). When using DRF, this
> leads to verifying both resources instead of just the dominant resource.
> Ideally, the check should be based on policy.
> Note that current implementation of
> {{DominantResourceFairnessPolicy#checkIfUsageOverFairShare}} is broken.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6172) FSLeafQueue demand update needs to be atomic
[
https://issues.apache.org/jira/browse/YARN-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated YARN-6172:
--
Fix Version/s: 3.0.0-alpha3
> FSLeafQueue demand update needs to be atomic
>
>
> Key: YARN-6172
> URL: https://issues.apache.org/jira/browse/YARN-6172
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Varun Saxena
>Assignee: Miklos Szegedi
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-6172.000.patch, YARN-6172.001.patch
>
>
> Refer to test report
> https://builds.apache.org/job/PreCommit-YARN-Build/14882/testReport/
> {noformat}
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation.verifyLeafQueueStarvation(TestFSAppStarvation.java:133)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation.testPreemptionEnabled(TestFSAppStarvation.java:106)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6215) FairScheduler preemption and update should not run concurrently
[
https://issues.apache.org/jira/browse/YARN-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated YARN-6215:
--
Fix Version/s: 3.0.0-alpha3
> FairScheduler preemption and update should not run concurrently
> ---
>
> Key: YARN-6215
> URL: https://issues.apache.org/jira/browse/YARN-6215
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler, test
>Reporter: Sunil G
>Assignee: Tao Jie
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-6215.001.patch, YARN-6215.002.patch
>
>
> *Error Message*
> Incorrect number of containers on the greedy app expected:<4> but was:<8>
> Failed test case
> [link|https://builds.apache.org/job/PreCommit-YARN-Build/15038/testReport/org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair/TestFairSchedulerPreemption/testPreemptionBetweenNonSiblingQueues_FairSharePreemptionWithDRF_/]
> *Stacktrace*
> {noformat}
> java.lang.AssertionError: Incorrect number of containers on the greedy app
> expected:<4> but was:<8>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.verifyPreemption(TestFairSchedulerPreemption.java:282)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption.testPreemptionBetweenNonSiblingQueues(TestFairSchedulerPreemption.java:323)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6472) Improve Java sandbox regex
[
https://issues.apache.org/jira/browse/YARN-6472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated YARN-6472:
Summary: Improve Java sandbox regex (was: Possible Java sandbox
improvements)
> Improve Java sandbox regex
> --
>
> Key: YARN-6472
> URL: https://issues.apache.org/jira/browse/YARN-6472
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Greg Phillips
> Attachments: YARN-6472.001.patch, YARN-6472.002.patch
>
>
> I set the sandbox to enforcing mode. Unfortunately I was able to break out of
> the sandbox running native code with the following command:
> {code}
> cmd = "$JAVA_HOME/bin/java %s -Xmx825955249
> org.apache.hadoop.yarn.applications.helloworld.HelloWorld `touch
> ../../helloworld`" + \
> " 1>/AppMaster.stdout 2>/AppMaster.stderr"
> $ ls .../nm-local-dir/usercache/root/appcache/
> helloworld
> {code}
> Also, if I am not using sandboxes, could we create the nm-sandbox-policies
> directory (empty) lazily?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-5889) Improve and refactor user-limit calculation in capacity scheduler
[ https://issues.apache.org/jira/browse/YARN-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reopened YARN-5889: --- > Improve and refactor user-limit calculation in capacity scheduler > - > > Key: YARN-5889 > URL: https://issues.apache.org/jira/browse/YARN-5889 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Reporter: Sunil G >Assignee: Sunil G > Fix For: 3.0.0-alpha3 > > Attachments: YARN-5889.0001.patch, > YARN-5889.0001.suggested.patchnotes, YARN-5889.0002.patch, > YARN-5889.0003.patch, YARN-5889.0004.patch, YARN-5889.0005.patch, > YARN-5889.0006.patch, YARN-5889.0007.patch, YARN-5889.0008.patch, > YARN-5889.0009.patch, YARN-5889.0010.patch, YARN-5889.v0.patch, > YARN-5889.v1.patch, YARN-5889.v2.patch > > > Currently user-limit is computed during every heartbeat allocation cycle with > a write lock. To improve performance, this tickets is focussing on moving > user-limit calculation out of heartbeat allocation flow. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5889) Improve and refactor user-limit calculation in capacity scheduler
[ https://issues.apache.org/jira/browse/YARN-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved YARN-5889. --- Resolution: Fixed Re-resolving as "fixed" rather than "resolved" so it shows properly in the release notes. > Improve and refactor user-limit calculation in capacity scheduler > - > > Key: YARN-5889 > URL: https://issues.apache.org/jira/browse/YARN-5889 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Reporter: Sunil G >Assignee: Sunil G > Fix For: 3.0.0-alpha3 > > Attachments: YARN-5889.0001.patch, > YARN-5889.0001.suggested.patchnotes, YARN-5889.0002.patch, > YARN-5889.0003.patch, YARN-5889.0004.patch, YARN-5889.0005.patch, > YARN-5889.0006.patch, YARN-5889.0007.patch, YARN-5889.0008.patch, > YARN-5889.0009.patch, YARN-5889.0010.patch, YARN-5889.v0.patch, > YARN-5889.v1.patch, YARN-5889.v2.patch > > > Currently user-limit is computed during every heartbeat allocation cycle with > a write lock. To improve performance, this tickets is focussing on moving > user-limit calculation out of heartbeat allocation flow. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6472) Improve Java sandbox regex
[
https://issues.apache.org/jira/browse/YARN-6472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979491#comment-15979491
]
Robert Kanter commented on YARN-6472:
-
One trivial comment:
# Can you add a code comment to {{DelegatingLinuxContainerRuntime}} indicating
that the order is important. We don't want someone later moving things around
and accidentally re-opening this hole.
> Improve Java sandbox regex
> --
>
> Key: YARN-6472
> URL: https://issues.apache.org/jira/browse/YARN-6472
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Greg Phillips
> Attachments: YARN-6472.001.patch, YARN-6472.002.patch
>
>
> I set the sandbox to enforcing mode. Unfortunately I was able to break out of
> the sandbox running native code with the following command:
> {code}
> cmd = "$JAVA_HOME/bin/java %s -Xmx825955249
> org.apache.hadoop.yarn.applications.helloworld.HelloWorld `touch
> ../../helloworld`" + \
> " 1>/AppMaster.stdout 2>/AppMaster.stderr"
> $ ls .../nm-local-dir/usercache/root/appcache/
> helloworld
> {code}
> Also, if I am not using sandboxes, could we create the nm-sandbox-policies
> directory (empty) lazily?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6447) Provide container sandbox policies for groups
[
https://issues.apache.org/jira/browse/YARN-6447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979537#comment-15979537
]
Robert Kanter commented on YARN-6447:
-
Looks good overall. A few minor things:
# We should make sure to reset all system properties being set in the unit
tests. While we're at it, we should fix the other ones already in
{{TestJavaSandboxLinuxContainerRuntime}}.
# Can you add a test for the scenario where a user belongs to multiple groups?
# If you specify
{{yarn.nodemanager.runtime.linux.sandbox-mode.policy.group.foo}} it looks like
you get both the foo group's policy and also either the default policy or the
{{yarn.nodemanager.runtime.linux.sandbox-mode.policy}} policy. Shouldn't you
only get one of the three? i.e. ONE of the group's (or groups') policies, the
default policy, or the custom policy
# Would capitalization be a problem for the groups? What if a user is in the
"FOo" group but the admin accidentally specifies
{{yarn.nodemanager.runtime.linux.sandbox-mode.policy.group.fOo}}?
> Provide container sandbox policies for groups
> --
>
> Key: YARN-6447
> URL: https://issues.apache.org/jira/browse/YARN-6447
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.0.0-alpha3
>Reporter: Greg Phillips
>Assignee: Greg Phillips
>Priority: Minor
> Attachments: YARN-6447.001.patch
>
>
> Currently the container sandbox feature
> ([YARN-5280|https://issues.apache.org/jira/browse/YARN-5280]) allows YARN
> administrators to use one Java Security Manager policy file to limit the
> permissions granted to YARN containers. It would be useful to allow for
> different policy files to be used based on groups.
> For example, an administrator may want to ensure standard users who write
> applications for the MapReduce or Tez frameworks are not allowed to open
> arbitrary network connections within their data processing code. Users who
> are designing the ETL pipelines however may need to open sockets to extract
> data from external sources. By assigning these sets of users to different
> groups and setting specific policies for each group you can assert fine
> grained control over the permissions granted to each Java based container
> across a YARN cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6457) Allow custom SSL configuration to be supplied in WebApps
[ https://issues.apache.org/jira/browse/YARN-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979538#comment-15979538 ] ASF GitHub Bot commented on YARN-6457: -- GitHub user sanjaypujare opened a pull request: https://github.com/apache/hadoop/pull/216 YARN-6457 use existing conf object as sslConf object in WebApps for the builder to use for the HttpServer2 Merged from 2.7.1 branch You can merge this pull request into a Git repository by running: $ git pull https://github.com/sanjaypujare/hadoop YARN-6457.trunk.sanjay Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/216.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #216 commit 8218dab40b96984a383aa4ad9fa6ae2b053cff4a Author: Sanjay Pujare Date: 2017-04-21T23:00:13Z YARN-6457 use existing conf object as sslConf object in WebApps for the builder to use for the HttpServer2 > Allow custom SSL configuration to be supplied in WebApps > > > Key: YARN-6457 > URL: https://issues.apache.org/jira/browse/YARN-6457 > Project: Hadoop YARN > Issue Type: Improvement > Components: webapp, yarn >Reporter: Sanjay M Pujare > Original Estimate: 96h > Remaining Estimate: 96h > > Currently a custom SSL store cannot be passed on to WebApps which forces the > embedded web-server to use the default keystore set up in ssl-server.xml for > the whole Hadoop cluster. There are cases where the Hadoop app needs to use > its own/custom keystore. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6457) Allow custom SSL configuration to be supplied in WebApps
[ https://issues.apache.org/jira/browse/YARN-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979541#comment-15979541 ] Sanjay M Pujare commented on YARN-6457: --- [~haibochen] I have the new PR https://github.com/apache/hadoop/pull/216 . Pls let me know, thanks! > Allow custom SSL configuration to be supplied in WebApps > > > Key: YARN-6457 > URL: https://issues.apache.org/jira/browse/YARN-6457 > Project: Hadoop YARN > Issue Type: Improvement > Components: webapp, yarn >Reporter: Sanjay M Pujare > Original Estimate: 96h > Remaining Estimate: 96h > > Currently a custom SSL store cannot be passed on to WebApps which forces the > embedded web-server to use the default keystore set up in ssl-server.xml for > the whole Hadoop cluster. There are cases where the Hadoop app needs to use > its own/custom keystore. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3427) Remove deprecated methods from ResourceCalculatorProcessTree
[
https://issues.apache.org/jira/browse/YARN-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated YARN-3427:
--
Release Note:
The deprecated ProcessTree methods getCumulativeVmem
and getCumulativeRssmem have been removed.
was:
The deprecated {{ProcessTree}} methods {{getCumulativeVmem}}
and {{getCumulativeRssmem}} have been removed.
> Remove deprecated methods from ResourceCalculatorProcessTree
>
>
> Key: YARN-3427
> URL: https://issues.apache.org/jira/browse/YARN-3427
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>Priority: Blocker
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-3427.000.patch, YARN-3427.001.patch
>
>
> In 2.7, we made ResourceCalculatorProcessTree Public and exposed some
> existing ill-formed methods as deprecated ones for use by Tez.
> We should remove it in 3.0.0, considering that the methods have been
> deprecated for the all 2.x.y releases that it is marked Public in.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3427) Remove deprecated methods from ResourceCalculatorProcessTree
[
https://issues.apache.org/jira/browse/YARN-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated YARN-3427:
--
Release Note:
The deprecated {{ProcessTree}} methods {{getCumulativeVmem}}
and {{getCumulativeRssmem}} have been removed.
Added a release note for this incompatible change, please update if it's
incorrect.
> Remove deprecated methods from ResourceCalculatorProcessTree
>
>
> Key: YARN-3427
> URL: https://issues.apache.org/jira/browse/YARN-3427
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Karthik Kambatla
>Assignee: Miklos Szegedi
>Priority: Blocker
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-3427.000.patch, YARN-3427.001.patch
>
>
> In 2.7, we made ResourceCalculatorProcessTree Public and exposed some
> existing ill-formed methods as deprecated ones for use by Tez.
> We should remove it in 3.0.0, considering that the methods have been
> deprecated for the all 2.x.y releases that it is marked Public in.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3742) YARN RM will shut down if ZKClient creation times out
[
https://issues.apache.org/jira/browse/YARN-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979612#comment-15979612
]
Karthik Kambatla commented on YARN-3742:
That looks much cleaner. Thanks for updating the patch.
Couple of nits:
# When creating RMFatalEvent, start the message with capital letters in
AdminService, ActiveStandbyElectorBasedElectorService and
RMCriticalThreadUncaughtExceptionHandler
# RMCriticalThread...: Would "Critical thread, %s, exited unexpectedly" be a
grammatically better message?
> YARN RM will shut down if ZKClient creation times out
> ---
>
> Key: YARN-3742
> URL: https://issues.apache.org/jira/browse/YARN-3742
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Daniel Templeton
> Attachments: YARN-3742.001.patch, YARN-3742.002.patch
>
>
> The RM goes down showing the following stacktrace if the ZK client connection
> fails to be created. We should not exit but transition to StandBy and stop
> doing things and let the other RM take over.
> {code}
> 2015-04-19 01:22:20,513 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Received a
> org.apache.hadoop.yarn.server.resourcemanager.RMFatalEvent of type
> STATE_STORE_OP_FAILED. Cause:
> java.io.IOException: Wait for ZKClient creation timed out
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithCheck(ZKRMStateStore.java:1066)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:1090)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.existsWithRetries(ZKRMStateStore.java:996)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.updateApplicationStateInternal(ZKRMStateStore.java:643)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppTransition.transition(RMStateStore.java:162)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppTransition.transition(RMStateStore.java:147)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.handleStoreEvent(RMStateStore.java:806)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:879)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:874)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6510) Fix warning - procfs stat file is not in the expected format: YARN-3344 is not enough
[
https://issues.apache.org/jira/browse/YARN-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wilfred Spiegelenburg updated YARN-6510:
Attachment: YARN-6510.01.patch
Patch including test
> Fix warning - procfs stat file is not in the expected format: YARN-3344 is
> not enough
> -
>
> Key: YARN-6510
> URL: https://issues.apache.org/jira/browse/YARN-6510
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha2
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
> Attachments: YARN-6510.01.patch
>
>
> Even with the fix for YARN-3344 we still have issues with the procfs format.
> This is the case that is causing issues:
> {code}
> [user@nm1 ~]$ cat /proc/2406/stat
> 2406 (ib_fmr(mlx4_0)) S 2 0 0 0 -1 2149613632 0 0 0 0 166 126908 0 0 20 0 1 0
> 4284 0 0 18446744073709551615 0 0 0 0 0 0 0 2147483647 0 18446744073709551615
> 0 0 17 6 0 0 0 0 0
> {code}
> We do not handle the parenthesis in the name which causes the pattern
> matching to fail
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6509) Add a size threshold beyond which yarn logs will require a force option
[ https://issues.apache.org/jira/browse/YARN-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979649#comment-15979649 ] Junping Du commented on YARN-6509: -- It sounds like current behavior get involved since YARN-5088 which hasn't been released yet and no release note available. So I think it should be fine without mark JIRA as incompatible. However, I agree that we need to add release notes here for users to understand new command line. [~busbey], if our motivation for marking incompatible flag here is to remind us to put release notes. I would be +1 on that. > Add a size threshold beyond which yarn logs will require a force option > --- > > Key: YARN-6509 > URL: https://issues.apache.org/jira/browse/YARN-6509 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Xuan Gong > Fix For: 2.9.0 > > Attachments: YARN-6509.1.patch > > > An accidental fetch for a long running application can lead to scenario which > the large size of log can fill up a disk. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6127) Add support for work preserving NM restart when AMRMProxy is enabled
[
https://issues.apache.org/jira/browse/YARN-6127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subru Krishnan reassigned YARN-6127:
Assignee: Botong Huang (was: Subru Krishnan)
> Add support for work preserving NM restart when AMRMProxy is enabled
>
>
> Key: YARN-6127
> URL: https://issues.apache.org/jira/browse/YARN-6127
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: amrmproxy, nodemanager
>Reporter: Subru Krishnan
>Assignee: Botong Huang
>
> YARN-1336 added the ability to restart NM without loosing any running
> containers. In a Federated YARN environment, there's additional state in the
> {{AMRMProxy}} to allow for spanning across multiple sub-clusters, so we need
> to enhance {{AMRMProxy}} to support work-preserving restart.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
